An offshore pasture fish school intelligent breeding system based on multi-modal data monitoring

Abstract

The present application relates to a kind of offshore pasture fish school intelligent breeding system based on multi-modal data monitoring, through multi-modal perception layer, the joint collection of visual, acoustic and water environment multi-modal data is carried out, and feature fusion is carried out, then through data and modeling layer, mapping and fish school monitoring are carried out, and through high-order intelligent agent layer, multi-modal large model is called to the mapping and fish school monitoring result is carried out knowledge reasoning and language generation to obtain optimal strategy, finally through self-optimizing decision layer, linkage related breeding equipment is carried out according to optimal strategy, realize the closed-loop control and self-adaptive optimization of breeding process.The present application can greatly improve the accuracy of multi-modal large model strategy planning after obtaining the multi-modal data of fish school, effectively improves the intelligent, unmanned and sustainable level of offshore pasture breeding by constructing fish school graph structure and obtaining fish school monitoring result based on the fish school graph structure, and then calling multi-modal large model to plan optimal strategy.

Description

Technical Field

[0001] This invention relates to the field of marine intelligent aquaculture technology, and in particular to an intelligent aquaculture system for marine ranches based on multimodal data monitoring. Background Technology

[0002] With the expansion of deep-sea aquaculture, traditional manual observation methods are no longer sufficient to meet the demand for real-time, accurate, and comprehensive monitoring of fish health, behavior, and environmental conditions. Existing technologies largely rely on single sensors (such as cameras or sonar) for data collection, which suffers from limited information dimensions, poor anti-interference capabilities, and difficulty in data fusion and analysis. Furthermore, existing systems lack the ability to model fish social structures, individual health, and behavioral predictions, and cannot achieve closed-loop control from monitoring to decision-making. Therefore, there is an urgent need for an integrated system with multimodal perception, intelligent modeling, high-order reasoning, and autonomous decision-making capabilities to improve the level of intelligence in marine ranching. Summary of the Invention

[0003] The purpose of this invention is to at least address one of the shortcomings of the prior art and provide an intelligent aquaculture system for marine ranches based on multimodal data monitoring.

[0004] To achieve the above objectives, the present invention adopts the following technical solution:

[0005] Specifically, a smart aquaculture system for marine ranches based on multimodal data monitoring is proposed, including the following: The multimodal perception layer is used to jointly collect visual, acoustic and aquatic environment multimodal data through heterogeneous sensing nodes pre-deployed in aquaculture cages and their surrounding waters, and to perform feature fusion on the multimodal data to obtain fused features. The data and modeling layer is used to model individual fish or local groups as graph structure nodes based on graph neural networks and graph attention mechanisms. Based on the fusion features provided by the multimodal perception layer, the fish school graph structure is constructed, and the fish school monitoring results are obtained by reasoning based on the fish school graph structure. The higher-order intelligent agent layer is used to perform knowledge reasoning and language generation on the fish swarm monitoring results using a multimodal large model, output structured monitoring reports, management strategy suggestions and risk warnings, and screen the optimal management strategy combination through scenario simulation; The autonomous decision-making layer is used to transform the optimal management strategy combination into executable control commands. Based on the executable control commands, feeding, oxygenation, water quality control and disease intervention equipment are linked to achieve closed-loop control and adaptive optimization of the aquaculture process.

[0006] Furthermore, specifically, the multimodal sensing layer includes, Infrared underwater cameras are used to acquire images of fish schools and achieve target recognition, quantity statistics, size estimation, and behavior analysis; multibeam sonar and fish finders are used to penetrate turbid water to detect fish density, outline, depth distribution, and net damage; and water chemistry sensors are used to monitor environmental parameters such as dissolved oxygen, temperature, salinity, pH value, and chlorophyll in real time.

[0007] Furthermore, specifically, feature fusion is performed on the multimodal data to obtain fused features, including: The feature matrices for the three input modalities—visual, acoustic, and environmental—are defined as follows: Visual modality: ; Acoustic modes: ; Environmental modalities: ; in, Let represent the number of input feature vectors in the visual, acoustic, and environmental modalities, respectively. Additionally, define... This represents the i-th feature vector in the visual modality. This represents the j-th eigenvector in the acoustic mode. This represents the k-th eigenvector in the environmental modality; To achieve feature alignment across different modalities, a linear projection is performed on each modality to map the features of each modality to a unified representation space, generating corresponding query, key, and value matrices. ; in These represent the Query, Key, and Value projection matrices for the visual modality, respectively. These represent the Query, Key, and Value projection matrices of the acoustic modes, respectively. These represent the Query, Key, and Value projection matrices of the environment modality, respectively. These are the Query, Key, and Value matrices for the visual modality. These are the Query, Key, and Value matrices for the acoustic modes, respectively. These are the Query, Key, and Value matrices for the respective environment modalities; Enhanced features are obtained by achieving information complementarity and feature enhancement among visual, acoustic and environmental modalities through a bidirectional co-attention mechanism. The enhanced features are fused using a multi-head attention mechanism to obtain fused features.

[0008] Furthermore, specifically, enhanced features are obtained by achieving information complementarity and feature enhancement among visual, acoustic, and environmental modalities through a bidirectional co-attention mechanism, including: Unnormalized attention score matrix for acoustics in computer vision: ; in, This represents the transpose of the acoustic modal key matrix. To provide a unified dimension for model representation, used for scaling dot products; Further calculation of the weighted values ​​of visual absorption from acoustics and the weighted value of acoustic absorption from visual perception. , ; in, Used for normalization calculations; Similarly, the weighted value of visual absorption from the environment is calculated in the same way as above. and the weighted value of visual absorption of the environment. : ; Calculate the weighted value of acoustic absorption from the environment and the weighted value of acoustic absorption by the environment ; .

[0009] For weighted features obtained from different modalities within the same modality, a fusion method is used to obtain the final enhanced feature. (Visual modality enhancement) Represented as: ; in, This represents the original features of the visual modality. and The weighting coefficients are used for fusion. The calculation method for the enhancement features of acoustic modes and environmental modes is similar.

[0010] Furthermore, specifically, the enhanced features are fused using a multi-head attention mechanism to obtain fused features, including... ; Where h represents the number of attention heads, This represents the result output by the i-th attention head on modality m. This indicates that the outputs of h heads are concatenated along the feature dimension. For multi-head output linear projection matrix, Finally, the fused features are represented as: ; in, This indicates a weighted summation fusion operation.

[0011] Furthermore, specifically, constructing the fish school graph structure includes, This invention abstracts a fish school into a graph structure G=(V,E), where V is the set of nodes, each node i∈V represents an individual fish or a local fish school, and E is the set of edges representing the potential interactions between nodes. Unlike traditional methods that analyze fish schools based solely on single-frame images or single sensor data, this invention constructs a fish school graph structure, abstracting individual fish or local fish schools into nodes, and characterizing the spatial proximity and behavioral interactions between fish schools through edge relationships, thereby expressing the group structure characteristics of the fish school. Based on this, a graph attention mechanism is introduced. By learning the influence weights of different neighboring nodes on the target node, the model can adaptively focus on neighboring nodes that have a greater impact on the health status or behavioral changes of the target node. Compared to traditional rule-based or statistical feature-based analysis methods, this invention can more effectively characterize the dynamic changes in fish school behavior in complex marine environments, thereby improving the accuracy of fish health status identification, anomaly detection, and behavior prediction.

[0012] Node feature vectors The feature is obtained by calculating the fusion features of the nodes, and then transformed by linear transformation. Mapping to a unified representation space yields the projected node representation. : ; Where W is a learnable linear transformation matrix; Calculate the unnormalized attention score between node i and its neighbor node j: ; Calculate the unnormalized attention score between node i and its neighbor node k: ; in, This indicates the strength of the influence of node j on node i. This indicates the strength of the influence of node k on node i. Let be the attention function. For a leaky linear rectified activation function, This represents vector concatenation. The projected feature vector of node i. The projected feature vector of node j, The projected feature vector of node k; To obtain standardized attention weights, softmax normalization is applied to the unnormalized attention scores of all neighboring nodes to obtain the final attention coefficients: ; Where α is a learnable parameter vector, Represents an exponential function. Let k be the set of neighbors of node i, and k represent node k. Based on the attention weights, the features of neighboring nodes are weighted and fused, and the updated node representation is as follows: ; in Let K be a non-linear activation function, and K be the number of heads in the multi-head attention mechanism. This represents the attention weight of node j to node i in the k-th attention head. Let be the linear transformation matrix of the k-th attention head. Let be the input feature vector of node j. This represents the updated node representation.

[0013] Furthermore, specifically, the fish school monitoring results are obtained by reasoning based on the aforementioned fish school diagram structure, including: Health status classification: embed the L-th layer embedding vector of each node i. Input classifier results in: ; in, The classifier weight matrix is... For bias vectors, Let health category probability vector be , Using average cross-entropy loss: ; in, For the number of health categories, This indicates summing over all health categories. This is a set of labeled training nodes. Let i be the true category label. To predict class probabilities, The average cross-entropy loss; Anomaly detection: The error is reconstructed using an autoencoder, and the L-th layer embedding vector of each node i is... Input self-decoder function Perform original feature reconstruction: ; in, Let i be the original feature of node i. The features are reconstructed; Reconstruction error is defined as: ; in, This represents the reconstruction error of node i. This indicates the calculation of the L2 norm; The reconstruction loss is: ; in, Let be the total number of nodes in the graph structure. To average the reconstruction loss, The final anomaly detection uses a threshold method: for any node, when its... If so, the node is determined to be an abnormal individual, where The preset threshold is determined by the distribution of the validation set; Behavior prediction: Modeling temporal embeddings using gated recurrent units (GRUs): ; Where t represents the current time step, This represents the graph embedding of node i at time t. This represents the GRU hidden state of node i at time step t-1. This is a gated loop unit function. This represents the hidden state of the GRU at time t; Predicting future location using regression head: ; in, Indicates the predicted time offset. For the regression weight matrix, For regression bias, Indicates the predicted future location; Mean squared error loss is used: ; in, Indicates the actual future location. denoted as the mean squared error of the regression task.

[0014] Furthermore, specifically, the higher-order agent utilizes the multimodal large model GPT-4o to perform knowledge reasoning and language generation on the fish swarm monitoring results.

[0015] The beneficial effects of this invention are as follows: This invention proposes an intelligent aquaculture system for marine ranching fish swarms based on multimodal data monitoring. The system employs a multimodal perception layer to jointly collect visual, acoustic, and aquatic environment multimodal data, and performs feature fusion. A data and modeling layer then constructs a map and monitors the fish swarms. A higher-order intelligent agent layer invokes a multimodal large-scale model to perform knowledge reasoning and language generation on the mapping and fish swarm monitoring results to derive the optimal strategy. Finally, an autonomous decision-making layer implements coordinated feeding, oxygenation, water quality control, and disease intervention equipment based on the optimal strategy, achieving closed-loop control and adaptive optimization of the aquaculture process. After acquiring multimodal data on the fish swarms, this invention constructs a fish swarm graph structure and infers the fish swarm monitoring results based on this graph structure. Then, it invokes a multimodal large-scale model for optimal strategy planning, significantly improving the accuracy of strategy planning and effectively enhancing the intelligence, unmanned operation, and sustainability of marine ranching. Attached Figure Description

[0016] The above and other features of this disclosure will become more apparent from the detailed description of the embodiments illustrated in conjunction with the accompanying drawings. In the accompanying drawings, the same reference numerals denote the same or similar elements. Obviously, the drawings described below are merely some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained from these drawings without any creative effort. In the drawings: Figure 1 The diagram shows a flowchart of an intelligent aquaculture system for marine ranches based on multimodal data monitoring, according to the present invention. Detailed Implementation

[0017] The following will provide a clear and complete description of the concept, specific structure, and technical effects of the present invention in conjunction with embodiments and accompanying drawings, so as to fully understand the purpose, solution, and effects of the present invention. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The same reference numerals used throughout the accompanying drawings indicate the same or similar parts.

[0018] Example 1, referring to Figure 1 This invention proposes an intelligent aquaculture system for marine ranches based on multimodal data monitoring, comprising the following: The multimodal perception layer is used to jointly collect visual, acoustic and aquatic environment multimodal data through heterogeneous sensing nodes pre-deployed in aquaculture cages and their surrounding waters, and to perform feature fusion on the multimodal data to obtain fused features. The data and modeling layer is used to model individual fish or local groups as graph structure nodes based on graph neural networks and graph attention mechanisms. Based on the fusion features provided by the multimodal perception layer, the fish school graph structure is constructed, and the fish school monitoring results are obtained by reasoning based on the fish school graph structure. The higher-order intelligent agent layer is used to perform knowledge reasoning and language generation on the fish swarm monitoring results using a multimodal large model, output structured monitoring reports, management strategy suggestions and risk warnings, and screen the optimal management strategy combination through scenario simulation; Based on the results of multimodal perception and GNN modeling, the multimodal large model GPT-4o is used for knowledge reasoning and language generation to realize the generation of structured monitoring reports, group behavior analysis, health risk prediction and management strategy recommendation. Combined with scenario simulation, the optimal combination of management strategies is selected to provide input basis for autonomous decision-making.

[0019] The autonomous decision-making layer is used to transform the optimal management strategy combination into executable control commands. Based on the executable control commands, feeding, oxygenation, water quality control and disease intervention equipment are linked to achieve closed-loop control and adaptive optimization of the aquaculture process.

[0020] Based on monitoring reports and candidate management strategies output by the higher-level intelligent agent layer, these strategies are transformed into executable control commands, linking feeders, aeration devices, water quality control systems, and disease intervention equipment to achieve closed-loop control and adaptive optimization. When abnormal fish health status or sudden changes in environmental parameters are detected, the autonomous decision-making layer can dynamically adjust feeding amounts, aeration frequency, and water quality control parameters to ensure stable operation of the aquaculture system and improve the suitability of the fish's growth environment. In a preferred embodiment of the present invention, specifically, the multimodal sensing layer includes... Infrared underwater cameras are used to acquire images of fish schools and achieve target recognition, quantity statistics, size estimation, and behavior analysis; multibeam sonar and fish finders are used to penetrate turbid water to detect fish density, outline, depth distribution, and net damage; and water chemistry sensors are used to monitor environmental parameters such as dissolved oxygen, temperature, salinity, pH value, and chlorophyll in real time.

[0021] In a preferred embodiment of the present invention, specifically, feature fusion is performed on the multimodal data to obtain fused features, including: The feature matrices for the three input modalities—visual, acoustic, and environmental—are defined as follows: Visual modality: ; Acoustic modes: ; Environmental modalities: ; in, Let represent the number of input feature vectors in the visual, acoustic, and environmental modalities, respectively. Additionally, define... This represents the i-th feature vector in the visual modality. This represents the j-th eigenvector in the acoustic mode. This represents the k-th eigenvector in the environmental modality; To achieve feature alignment across different modalities, a linear projection is performed on each modality to map the features of each modality to a unified representation space, generating corresponding query, key, and value matrices. ; in These represent the Query, Key, and Value projection matrices for the visual modality, respectively. These represent the Query, Key, and Value projection matrices of the acoustic modes, respectively. These represent the Query, Key, and Value projection matrices of the environment modality, respectively. These are the Query, Key, and Value matrices for the visual modality. These are the Query, Key, and Value matrices for the acoustic modes, respectively. These are the Query, Key, and Value matrices for the respective environment modalities; Enhanced features are obtained by achieving information complementarity and feature enhancement among visual, acoustic and environmental modalities through a bidirectional co-attention mechanism. The enhanced features are fused using a multi-head attention mechanism to obtain fused features.

[0022] In a preferred embodiment of the present invention, specifically, enhanced features are obtained by achieving information complementarity and feature enhancement among visual, acoustic, and environmental modalities through a bidirectional co-attention mechanism, including: Unnormalized attention score matrix for acoustics in computer vision: ; in, This represents the transpose of the acoustic modal key matrix. To provide a unified dimension for model representation, used for scaling dot products; As a preferred embodiment of the present invention, the weighted value of visual absorption from acoustic absorption is calculated. and the weighted value of acoustic absorption from visual perception. , ; in, Used for normalization calculations; Similarly, the weighted value of visual absorption from the environment is calculated in the same way as above. and the weighted value of visual absorption of the environment. : ; Calculate the weighted value of acoustic absorption from the environment and the weighted value of acoustic absorption by the environment ; .

[0023] For weighted features obtained from different modalities within the same modality, a fusion method is used to obtain the final enhanced feature. (Visual modality enhancement) Represented as: ; in, This represents the original features of the visual modality. and The weighting coefficients are used for fusion. The calculation method for the enhancement features of acoustic modes and environmental modes is similar.

[0024] In a preferred embodiment of the present invention, specifically, the enhanced features are fused using a multi-head attention mechanism to obtain fused features, including... ; Where h represents the number of attention heads, This represents the result output by the i-th attention head on modality m. This indicates that the outputs of h heads are concatenated along the feature dimension. For multi-head output linear projection matrix, Finally, the fused features are represented as: ; in, This indicates a weighted summation fusion operation.

[0025] In a preferred embodiment of the present invention, specifically, constructing a fish school diagram structure includes: This invention abstracts a fish school into a graph structure G=(V,E), where V is the set of nodes, each node i∈V represents an individual fish or a local fish school, and E is the set of edges representing the potential interactions between nodes. Unlike traditional methods that analyze fish schools based solely on single-frame images or single sensor data, this invention constructs a fish school graph structure, abstracting individual fish or local fish schools into nodes, and characterizing the spatial proximity and behavioral interactions between fish schools through edge relationships, thereby expressing the group structure characteristics of the fish school. Based on this, a graph attention mechanism is introduced. By learning the influence weights of different neighboring nodes on the target node, the model can adaptively focus on neighboring nodes that have a greater impact on the health status or behavioral changes of the target node. Compared to traditional rule-based or statistical feature-based analysis methods, this invention can more effectively characterize the dynamic changes in fish school behavior in complex marine environments, thereby improving the accuracy of fish health status identification, anomaly detection, and behavior prediction.

[0026] Node feature vectors The feature is obtained by calculating the fusion features of the nodes, and then transformed by linear transformation. Mapping to a unified representation space yields the projected node representation. : ; Where W is a learnable linear transformation matrix; Calculate the unnormalized attention score between node i and its neighbor node j: ; Calculate the unnormalized attention score between node i and its neighbor node k: ; in, This indicates the strength of the influence of node j on node i. This indicates the strength of the influence of node k on node i. Let be the attention function. For a leaky linear rectified activation function, This represents vector concatenation. The projected feature vector of node i. The projected feature vector of node j, The projected feature vector of node k; To obtain standardized attention weights, softmax normalization is applied to the unnormalized attention scores of all neighboring nodes to obtain the final attention coefficients: ; Where α is a learnable parameter vector, Represents an exponential function. Let k be the set of neighbors of node i, and k represent node k. Based on the attention weights, the features of neighboring nodes are weighted and fused, and the updated node representation is as follows: ; in Let K be a non-linear activation function, and K be the number of heads in the multi-head attention mechanism. This represents the attention weight of node j to node i in the k-th attention head. Let be the linear transformation matrix of the k-th attention head. Let be the input feature vector of node j. This represents the updated node representation.

[0027] In a preferred embodiment of the present invention, specifically, the fish school monitoring results are obtained by reasoning based on the fish school diagram structure, including: Health status classification: embed the L-th layer embedding vector of each node i. Input classifier results in: ; in, The classifier weight matrix is... For bias vectors, Let health category probability vector be , Using average cross-entropy loss: ; in, For the number of health categories, This indicates summing over all health categories. This is a set of labeled training nodes. Let i be the true category label. To predict class probabilities, The average cross-entropy loss; Anomaly detection: The error is reconstructed using an autoencoder, and the L-th layer embedding vector of each node i is... Input self-decoder function Perform original feature reconstruction: ; in, Let i be the original feature of node i. The features are reconstructed; Reconstruction error is defined as: ; in, This represents the reconstruction error of node i. This indicates the calculation of the L2 norm; The reconstruction loss is: ; in, Let be the total number of nodes in the graph structure. To average the reconstruction loss, The final anomaly detection uses a threshold method: for any node, when its... If so, the node is determined to be an abnormal individual, where The preset threshold is determined by the distribution of the validation set; Behavior prediction: Modeling temporal embeddings using gated recurrent units (GRUs): ; Where t represents the current time step, This represents the graph embedding of node i at time t. This represents the GRU hidden state of node i at time step t-1. This is a gated loop unit function. This represents the hidden state of the GRU at time t; Predicting future location using regression head: ; in, Indicates the predicted time offset. For the regression weight matrix, For regression bias, Indicates the predicted future location; Mean squared error loss is used: ; in, Indicates the actual future location. denoted as the mean squared error of the regression task.

[0028] In a preferred embodiment of the present invention, the higher-order agent specifically uses the multimodal large model GPT-4o to perform knowledge reasoning and language generation on the fish swarm monitoring results.

[0029] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0030] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or system capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0031] Although the description of the invention has been quite detailed and particularly of several described embodiments, it is not intended to limit it to any of these details or embodiments or any particular embodiment, but should be considered as providing a broad possible interpretation of the claims by referring to the appended claims and taking into account the prior art, thereby effectively covering the intended scope of the invention. Furthermore, the invention has been described above with respect to embodiments foreseeable by the inventors in order to provide a useful description, and non-substantial modifications to the invention that have not yet been foreseen may still represent equivalent modifications.

[0032] The above description is merely a preferred embodiment of the present invention. The present invention is not limited to the above-described embodiments. Any embodiment that achieves the technical effects of the present invention using the same means should fall within the protection scope of the present invention. Within the protection scope of the present invention, various modifications and variations can be made to the technical solutions and / or implementation methods.

Claims

1. A smart aquaculture system for marine ranches based on multimodal data monitoring, characterized in that, Including the following: The multimodal perception layer is used to jointly collect visual, acoustic and aquatic environment multimodal data through heterogeneous sensing nodes pre-deployed in aquaculture cages and their surrounding waters, and to perform feature fusion on the multimodal data to obtain fused features. The data and modeling layer is used to model individual fish or local groups as graph structure nodes based on graph neural networks and graph attention mechanisms. Based on the fusion features provided by the multimodal perception layer, the fish school graph structure is constructed, and the fish school monitoring results are obtained by reasoning based on the fish school graph structure. The higher-order intelligent agent layer is used to perform knowledge reasoning and language generation on the fish swarm monitoring results using a multimodal large model, output structured monitoring reports, management strategy suggestions and risk warnings, and screen the optimal management strategy combination through scenario simulation; The autonomous decision-making layer is used to transform the optimal management strategy combination into executable control commands. Based on the executable control commands, feeding, oxygenation, water quality control and disease intervention equipment are linked to achieve closed-loop control and adaptive optimization of the aquaculture process.

2. The intelligent aquaculture system for marine ranching fish schools based on multimodal data monitoring according to claim 1, characterized in that, Specifically, the multimodal sensing layer includes, Infrared underwater cameras are used to acquire images of fish schools and achieve target recognition, quantity statistics, size estimation, and behavior analysis; multibeam sonar and fish finders are used to penetrate turbid water to detect fish density, outline, depth distribution, and net damage; and water chemistry sensors are used to monitor environmental parameters such as dissolved oxygen, temperature, salinity, pH value, and chlorophyll in real time.

3. The intelligent aquaculture system for marine ranching fish schools based on multimodal data monitoring according to claim 2, characterized in that, Specifically, feature fusion is performed on the multimodal data to obtain fused features, including: The feature matrices for the three input modalities—visual, acoustic, and environmental—are defined as follows: Visual modality: ; Acoustic modes: ; Environmental modalities: ; in, Let represent the number of input feature vectors in the visual, acoustic, and environmental modalities, respectively. Additionally, define... This represents the i-th feature vector in the visual modality. This represents the j-th eigenvector in the acoustic mode. This represents the k-th eigenvector in the environmental modality; To achieve feature alignment across different modalities, a linear projection is performed on each modality to map the features of each modality to a unified representation space, generating corresponding query, key, and value matrices. ； in These represent the Query, Key, and Value projection matrices for the visual modality, respectively. These represent the Query, Key, and Value projection matrices of the acoustic modes, respectively. These represent the Query, Key, and Value projection matrices of the environment modality, respectively. These are the Query, Key, and Value matrices for the visual modality. These are the Query, Key, and Value matrices for the acoustic modes, respectively. These are the Query, Key, and Value matrices for the respective environment modalities; Enhanced features are obtained by achieving information complementarity and feature enhancement among visual, acoustic and environmental modalities through a bidirectional co-attention mechanism. The enhanced features are fused using a multi-head attention mechanism to obtain fused features.

4. The intelligent aquaculture system for marine ranching fish schools based on multimodal data monitoring according to claim 3, characterized in that, Specifically, enhanced features are obtained by achieving information complementarity and feature enhancement among visual, acoustic, and environmental modalities through a bidirectional co-attention mechanism, including: Unnormalized attention score matrix for acoustics in computer vision: ； in, This represents the transpose of the acoustic modal key matrix. To provide a unified dimension for model representation, used for scaling dot products; Further calculation of the weighted values ​​of visual absorption from acoustics and the weighted value of acoustic absorption from visual perception , ； in, Used for normalization calculations; Similarly, the weighted value of visual absorption from the environment is calculated in the same way as above. and the weighted value of visual absorption of the environment. : ； Calculate the weighted value of acoustic absorption from the environment and the weighted value of acoustic absorption from the environment ; ； For weighted features obtained from different modalities within the same modality, the final enhanced features are obtained through fusion, resulting in visual modality enhancement. Represented as: ； in, This represents the original features of the visual modality. and The calculation method for the enhancement features of acoustic and environmental modes is similar to that for the fusion of weighting coefficients.

5. The intelligent aquaculture system for marine ranching fish schools based on multimodal data monitoring according to claim 4, characterized in that, Specifically, enhanced features are fused using a multi-head attention mechanism to obtain fused features, including: ； Where h represents the number of attention heads, This represents the result output by the i-th attention head on mode m. This indicates that the outputs of h heads are concatenated along the feature dimension. For multi-head output linear projection matrix, Finally, the fused features are represented as: ； in, This indicates a weighted summation fusion operation.

6. The intelligent aquaculture system for marine ranching fish schools based on multimodal data monitoring according to claim 1, characterized in that, Specifically, constructing the fish school graph structure includes, The fish school is abstracted as a graph structure G=(V,E), where V is the set of nodes, each node i∈V represents an individual fish or a local fish school, and E is the set of edges, representing the potential interaction relationships between nodes; Node feature vectors The feature is obtained by calculating the fusion features of the nodes, and then transformed by linear transformation. Mapping to a unified representation space yields the projected node representation. : ； Where W is a learnable linear transformation matrix; Calculate the unnormalized attention score between node i and its neighbor node j: ； Calculate the unnormalized attention score between node i and its neighbor node k: ； in, This indicates the strength of the influence of node j on node i. This indicates the strength of the influence of node k on node i. Let be the attention function. For a leaky linear rectified activation function, This represents vector concatenation. Let be the projected feature vector of node i. Let be the projected feature vector of node j. Let be the projected feature vector of node k; To obtain standardized attention weights, softmax normalization is applied to the unnormalized attention scores of all neighboring nodes to obtain the final attention coefficients: ； Where α is a learnable parameter vector, Represents an exponential function. Let k be the set of neighbors of node i, and k represent node k. Based on the attention weights, the features of neighboring nodes are weighted and fused, and the updated node representation is as follows: ； in Let K be a non-linear activation function, and K be the number of heads in the multi-head attention mechanism. This represents the attention weight of node j to node i in the k-th attention head. Let be the linear transformation matrix of the k-th attention head. Let be the input feature vector of node j. This represents the updated node representation.

7. The intelligent aquaculture system for marine ranching fish schools based on multimodal data monitoring according to claim 6, characterized in that, Specifically, the fish school monitoring results are obtained by reasoning based on the fish school diagram structure, including: Health status classification: embed the L-th layer embedding vector of each node i. Input classifier results in: ； in, The classifier weight matrix is... For bias vectors, Let health category probability vector be , Using average cross-entropy loss: ； in, For the number of health categories, This indicates summing over all health categories. This is a set of labeled training nodes. Let i be the true category label. To predict class probabilities, The average cross-entropy loss; Anomaly detection: The error is reconstructed using an autoencoder, and the L-th layer embedding vector of each node i is... Input self-decoder function Perform original feature reconstruction: ； Reconstruction error is defined as: ； in, Let i be the original feature of node i. For the reconstructed features, This represents the reconstruction error of node i. This indicates the calculation of the L2 norm; The reconstruction loss is: ； in, Let be the total number of nodes in the graph structure. To average the reconstruction loss, The final anomaly detection uses a threshold method: for any node, when its... If so, the node is determined to be an abnormal individual, where The preset threshold is determined by the distribution of the validation set; Behavior prediction: Modeling temporal embeddings using gated recurrent units (GRUs): ； Where t represents the current time step, This represents the graph embedding of node i at time t. This represents the GRU hidden state of node i at time step t-1. This is a gated loop unit function. This represents the hidden state of the GRU at time t; Predicting future location using regression head: ； in, Indicates the predicted time offset. For the regression weight matrix, For regression bias, Indicates the predicted future location; Mean squared error loss is used: ； in, Indicates the actual future location. denoted as the mean squared error of the regression task.

8. The intelligent aquaculture system for marine ranching fish schools based on multimodal data monitoring according to claim 1, characterized in that, Specifically, the higher-order agent uses the multimodal large model GPT-4o to perform knowledge reasoning and language generation on the fish swarm monitoring results.

Images