Reservoir ecology intelligent visual real-time monitoring system and method

Abstract

The invention provides an intelligent visual real-time monitoring system and method for reservoir ecology. The system covers multi-modal data collection, graph structure data construction, deep learning model application and fusion feature vector output. The system comprises a collection module, a graph data module, a neural network module (including a Transform encoder and a decoder), an execution module, an output module and a working module. The reservoir ecological multi-modal data is collected, graph structure data based on a sensor network and an ecological relationship network is constructed, a Transform model is applied to process and output a fusion feature vector, and the vector reflects multi-modal feature relevance and a data time-space relationship and is used for early warning or decision-making tasks. According to the invention, comprehensive, real-time and intelligent monitoring of reservoir ecology is realized, the accuracy and timeliness of early warning and decision making are improved, and powerful technical support is provided for reservoir ecology management.

Description

Technical Field

[0001] The present application relates to the field of ecological monitoring, and in particular to a reservoir ecological intelligent visual real-time monitoring system and method. Background Art

[0002] In reservoir ecological management, traditional monitoring systems mainly rely on single-mode data collection and analysis, such as collecting water quality data only through water quality monitors, or recording video images of the reservoir area through video monitoring equipment. However, this single-mode data often cannot fully reflect the complexity and dynamics of the reservoir ecological environment, resulting in limited monitoring effects and difficulty in achieving comprehensive, real-time and intelligent monitoring of reservoir ecology.

[0003] In addition, traditional data processing and analysis methods are usually based on simple statistics and rule judgments, and cannot deeply explore the potential associations and spatiotemporal relationships between data, thus limiting the accuracy and timeliness of early warning and decision-making. For example, changes in water quality data may be related to multiple factors such as weather, topography, and biological activities in the reservoir area, but traditional monitoring systems often cannot comprehensively consider these factors, resulting in insufficient basis for early warning and decision-making. Summary of the invention

[0004] The purpose of the present application is to overcome the above-mentioned prior art and provide a monitoring method and system for cardiology nursing.

[0005] The present application provides a reservoir ecological intelligent visual real-time monitoring system, comprising:

[0006] The collection module collects multimodal data related to reservoir ecology;

[0007] A graph data module, which constructs graph structure data based on sensor networks and ecological relationship networks according to the multimodal data;

[0008] A neural network module is provided to construct a Transformer encoder, which includes a plurality of identical encoding layers stacked together, each encoding layer including a multi-head self-attention sublayer and a fully connected feedforward network sublayer; position encoding is introduced into the Transformer encoder; a Transformer decoder is constructed, which includes a plurality of identical decoding layers stacked together, each decoding layer including a multi-head self-attention sublayer, an encoder-decoder attention sublayer and a fully connected feedforward network sublayer;

[0009] An execution module inputs the graph structure data into the Transformer encoder, and the output of the Transformer encoder serves as the initial input of the Transformer decoder;

[0010] Output module, the Transformer decoder outputs a fused feature vector, which contains the correlation between multimodal features and the spatial and temporal relationship between data;

[0011] The working module is used in early warning or decision-making tasks according to the feature vector.

[0012] Optionally, collect multimodal data related to reservoir ecology, including:

[0013] Collect water quality data including temperature, pH, dissolved oxygen content and turbidity through water quality sensors;

[0014] Collect biological species data through biometric technology, including the types and numbers of fish, aquatic plants and microorganisms;

[0015] Collect image data through high-definition cameras to monitor ecological changes in the reservoir area;

[0016] Acoustic data is collected through acoustic sensors to monitor hydroacoustic activity and biological sounds in the reservoir area.

[0017] Optionally, constructing graph structure data based on sensor networks and ecological relationship networks according to the multimodal data includes:

[0018] The sensors are used as nodes of graph structure data, and the communication or spatial relationship between sensors is used as the edge to construct the sensor network graph;

[0019] The biological species are used as nodes of the graph structure data, and the food chains and symbiotic relationships between organisms are used as edges to construct an ecological relationship network diagram.

[0020] Optionally, build a Transformer encoder, including:

[0021] In each encoding layer, multiple self-attention operations are performed in parallel using different weight matrices to form a multi-head self-attention mechanism;

[0022] The representation of each position is transformed through a fully connected feed-forward network to generate a new representation.

[0023] Optionally, the introducing position coding in the encoder includes:

[0024] The position encoding is calculated using a combination of sine and cosine functions;

[0025] The position code is added to each position of the input data, so that each input data contains the content and position information of the original data.

[0026] Optionally, build a Transformer decoder, including:

[0027] In each decoding layer, a multi-head self-attention mechanism is used to process the encoder output to learn and understand the intrinsic connection between intermediate representations;

[0028] The encoder-decoder attention mechanism is used to calculate the attention weight between the current state of the decoder and all output states of the encoder to achieve effective transmission and integration of information.

[0029] Optionally, inputting the graph structure data into the Transformer encoder includes:

[0030] The nodes and edges in the graph structure data are converted into feature vectors as input of the Transformer encoder.

[0031] Optionally, the feature vector is used in warning or decision-making tasks, including:

[0032] The fused feature vector is input into the early warning model, and an early warning is issued according to the preset thresholds and rules;

[0033] Alternatively, the fused feature vectors can be input into a decision support system to assist decision makers in making ecological protection and management decisions.

[0034] Optionally, also include:

[0035] The collected multimodal data are preprocessed, including data cleaning, data format conversion, and data normalization, to improve data quality and model training effects.

[0036] The present application also provides a reservoir ecological intelligent visual real-time monitoring method, comprising:

[0037] Collect multimodal data related to reservoir ecology;

[0038] According to the multimodal data, constructing graph structure data based on sensor networks and ecological relationship networks;

[0039] Constructing a Transformer encoder, including a plurality of identical encoding layers stacked together, each encoding layer including a multi-head self-attention sublayer and a fully connected feed-forward network sublayer; introducing position encoding in the Transformer encoder;

[0040] Construct a Transformer decoder, which consists of multiple identical decoding layers stacked together. Each decoding layer contains a multi-head self-attention sublayer, an encoder-decoder attention sublayer, and a fully connected feedforward network sublayer.

[0041] Input the graph structure data into the Transformer encoder, and use the output of the Transformer encoder as the initial input of the Transformer decoder;

[0042] The Transformer decoder outputs a fused feature vector that contains the correlation between multimodal features and the spatial and temporal relationship between data;

[0043] The feature vector is used in early warning or decision-making tasks.

[0044] The beneficial effects of this application are:

[0045] Invention point 1: multimodal data integration.

[0046] Invention point 2: innovative application of deep learning models.

[0047] The present application provides a reservoir ecological intelligent visual real-time monitoring system, including: a collection module, which collects multimodal data related to the reservoir ecology; a graph data module, which constructs graph structure data based on a sensor network and an ecological relationship network according to the multimodal data; a neural network module, which constructs a Transformer encoder, which includes a plurality of identical encoding layers stacked together, each encoding layer including a multi-head self-attention sublayer and a fully connected feedforward network sublayer; introduces position coding in the Transformer encoder; constructs a Transformer decoder, which includes a plurality of identical decoding layers stacked together, each decoding layer including a multi-head self-attention sublayer, an encoder-decoder attention sublayer and a fully connected feedforward network sublayer; an execution module, which inputs the graph structure data into the Transformer encoder, and the output of the Transformer encoder is used as the initial input of the Transformer decoder; an output module, in which the Transformer decoder outputs a fused feature vector, which includes the correlation between multimodal features and the spatial and temporal relationship between data; and a working module, which is used in early warning or decision-making tasks according to the feature vector. This application achieves comprehensive, real-time and intelligent monitoring of reservoir ecology through multimodal data collection, graph structure data construction, deep learning model application and fusion feature vector output, improves the accuracy and timeliness of early warning and decision-making, and provides strong technical support for reservoir ecological management. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a schematic diagram of the reservoir ecological intelligent visual real-time monitoring system in this application;

[0049] Figure 2 It is a schematic diagram of the reservoir ecological intelligent visual real-time monitoring method in this application. DETAILED DESCRIPTION

[0050] The present application is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present application and implement it.

[0051] The following contents are all examples of specific implementation processes provided for describing in detail the technical solutions to be protected by this application. However, this application is also implemented in other ways different from the descriptions herein. Under the guidance of the concept of this application, those skilled in the art may implement this application using different technical means. Therefore, this application is not limited to the specific embodiments below.

[0052] Please refer to Figure 1 As shown, the present application provides a reservoir ecological intelligent visual real-time monitoring system, which includes: a collection module, a graph data module, a neural network module, an execution module, an output module, and a working module.

[0053] The collection module 101 collects multimodal data related to reservoir ecology.

[0054] In this application, the multimodal data refers to a series of diverse data sets collected from different information channels and sensor devices in the process of monitoring and managing reservoir ecosystems, reflecting the state, changes and ecological processes of reservoirs. The multimodal data includes:

[0055] The water temperature, pH value, dissolved oxygen content and turbidity of the reservoir are continuously monitored by water quality sensors.

[0056] By setting up underwater cameras or sonar equipment in the reservoir and combining biometric technology, different types of fish can be identified and counted, and the distribution and number changes of fish can be obtained.

[0057] Use drones or ground cameras to capture images of the reservoir area, combine image processing technology to identify and classify aquatic plants, and obtain the growth conditions and distribution of aquatic plants.

[0058] By collecting reservoir water samples, using microscopy and molecular biology techniques, the types and numbers of microorganisms can be identified and counted, and the diversity and stability of the microbial community can be obtained.

[0059] High-definition cameras are used to capture ecological changes in the reservoir area, such as water level changes, vegetation growth, and animal activities; they also include abnormal events in the reservoir area, such as pollution incidents and illegal fishing.

[0060] The water acoustic activity and biological sounds in the reservoir area are monitored through sound sensors.

[0061] The graph data module 102 constructs graph structure data based on the sensor network and the ecological relationship network according to the multimodal data.

[0062] Graph Neural Network (GNN) is a neural network specially designed to process graph structured data. Graph structured data consists of nodes (representing entities) and edges (representing the relationship between nodes). GNN can learn the representation of graph nodes and the connection relationship between nodes. By iteratively updating the representation of nodes, GNN can capture the complex information in the graph structure and generate richer feature representations.

[0063] In reservoir ecological monitoring, multimodal data presents complex graph structures, including sensor networks (where sensors are nodes and the communication or spatial relationships between sensors are edges), and ecological relationship networks (where biological species are nodes and the food chains and symbiotic relationships between them are edges).

[0064] In the system, feature vectors of different modes (such as water quality data and biological species data) are used as input nodes of GNN. GNN learns the connection relationship and representation between nodes by iteratively updating the representation of nodes. In this way, each node will obtain a representation that contains the information of its surrounding nodes, thereby capturing the complex information in the graph structure.

[0065] The neural network module 103 constructs a Transformer encoder, which includes a plurality of identical encoding layers stacked together, each encoding layer including a multi-head self-attention sublayer and a fully connected feedforward network sublayer; introduces position encoding in the Transformer encoder; and constructs a Transformer decoder, which includes a plurality of identical decoding layers stacked together, each decoding layer including a multi-head self-attention sublayer, an encoder-decoder attention sublayer and a fully connected feedforward network sublayer.

[0066] Specifically, the output of the GNN is used as one of the inputs of the Transformer encoder. In this way, the Transformer encoder can receive feature representations containing graph structure information and further learn the correlation between multimodal features. The output of the graph neural network is formatted before being input into the Transformer encoder.

[0067] The core of the Transformer architecture lies in its self-attention mechanism and positional encoding. The self-attention mechanism allows the model to focus on other positions in the sequence when processing each position of the input sequence, thereby capturing the dependencies within the sequence. Positional encoding is used to provide the model with information about the position of each element in the sequence.

[0068] The Transformer encoder performs multiple self-attention operations in parallel in the multi-head self-attention sublayer, each using a different weight matrix, allowing the model to focus on different parts of the input sequence at the same time. The Transformer encoder's multi-head self-attention mechanism divides the input data into multiple "heads" (i.e., multiple subsequences), and independently calculates the attention weights on each head, and then multiplies these weights with the corresponding input values ​​to obtain the weighted output. Finally, the outputs of each head are spliced ​​together and the final output is obtained through a linear transformation.

[0069] The fully connected feed-forward network of the Transformer encoder transforms the representation of each position to generate a new representation.

[0070] In order to introduce the correlation between multimodal features, this application uses feature vectors of different modalities (such as image features, sound features) as the input of the encoder, and learns the association between them through a multi-head self-attention mechanism. Feature vectors are numerical vectors converted from raw data (such as images and sounds) through feature extraction algorithms. These vectors can represent the key characteristics of the raw data. The multi-head self-attention mechanism divides the input data into multiple "heads" (i.e., multiple subsequences), and independently calculates the attention weights on each head, and then multiplies these weights with the corresponding input values ​​to obtain a weighted output. Finally, the outputs of each head are spliced ​​together, and the final output is obtained through a linear transformation.

[0071] In addition, the present application adds position encoding to the encoder to capture the spatial and temporal relationship between data.

[0072] In Transformer, the input sequence is converted into a series of vectors, which interact with each other through the self-attention mechanism to generate new representations. However, without additional position information, the model will not be able to distinguish elements at different positions in the input sequence, because the self-attention mechanism is symmetric, that is, it does not consider the order relationship between elements. Therefore, in order to maintain the order information of the sequence, this application adds position encoding to the input data.

[0073] Position encoding is done through a combination of sine and cosine functions. The benefit of this approach is that it provides a continuous representation of position information, which is particularly useful for processing variable-length sequences.

[0074] Specifically, for each position pos and dimension d in the input sequence, the position encoding is calculated as follows:

[0075]

[0076] in, k and c are constants. Where a and b are constants. i is the index of the dimension, starting from 0.

[0077] The f(i) function introduces a nonlinear transformation based on the dimension index, making the power of 2 on each dimension different, thereby increasing the diversity of position encoding. The g(i) function adjusts the frequency of the sine and cosine functions so that they change linearly with the increase of the dimension index. This change can further increase the discrimination of the position encoding.

[0078] In the Transformer encoder, positional encoding is added to each position of the input data. In this way, each input data contains both the content and position information of the original data. When these vectors interact through the self-attention mechanism, the model can take into account both the content and position relationship of the data, thereby generating a more accurate representation.

[0079] In the multi-head self-attention sublayer of the Transformer decoder, the decoder pays attention to its previous output positions to generate a coherent sequence output. First, a masking matrix with the same length as the decoder input sequence is constructed, in which each element indicates whether a position in the decoder should be paid attention to by other positions. For the position i currently being generated, all positions after i in the masking matrix are set to be invisible. For example, set to negative infinity or a very small value to ensure that the weights of these positions are close to zero when calculating the softmax function.

[0080] The expression of the shielding matrix is ​​as follows:

[0081]

[0082] Among them, M ij Represents the element in the i-th row and j-th column in the shielding matrix.

[0083] When calculating self-attention, the masking matrix is ​​added to the attention score matrix. Since the positions after i in the masking matrix are set to be invisible, the weights obtained by these positions when calculating the softmax function will be very small, thereby shielding the subsequent positions. Through the masking strategy, the decoder can only rely on the previous output position when generating the current output, thereby ensuring the coherence of the sequence. The specific calculation steps are as follows:

[0084] Assuming the attention score matrix is ​​A, the masked score matrix A′ can be expressed as:

[0085] A ij ′=A ij +M ij

[0086] Apply the softmax function to the masked score matrix A′ to obtain the attention weight matrix S:

[0087]

[0088] Apply the softmax function to each row of the score matrix A′ to obtain the attention weight matrix W:

[0089]

[0090] Since the positions after i in the masking matrix are set to large negative numbers, the weights of these positions will be close to zero when performing the softmax calculation.

[0091] Finally, the attention weight matrix W and the value matrix V are used to calculate the weighted sum to get the final attention output:

[0092]

[0093] Where Vj represents the element in the jth column of the value matrix.

[0094] The encoder-decoder attention sublayer allows the decoder to focus on the encoder's output, thereby capturing information about the input sequence. Specifically, after the encoder processes the input data, it generates a series of encoded states. When the decoder generates each output, it receives the input data of the current modality, which is converted into a representation compatible with the encoder output.

[0095] When processing the current output, the decoder generates a current state vector, which represents the information of all the outputs generated by the decoder so far. For each state of the encoder output and the current state of the decoder, an attention score is calculated. This score reflects the correlation between the encoder state and the current state of the decoder.

[0096] When calculating the attention score, an additional modality fusion module is introduced, which is able to process the input data of the current state and combine it with the encoder state and the decoder current state to calculate the final attention score.

[0097] For each state of the encoder output and the current state of the decoder, an attention score is calculated. The attention score reflects the correlation between the encoder state and the current state of the decoder. The calculation steps of the attention score are:

[0098] First, the modality fusion module is used to process the input data of the current state and combine it with the encoder state and the decoder current state, expressed as:

[0099] c i =fmodal-fusion(xcurrent ,h enc,i ,s dec )

[0100] Among them, x current is the input data of the current modality, fmodal-fusion is the function of the modal fusion module, c i is the combined representation, h enc,i is each state of the encoder output, s dec is the current state vector generated by the decoder when processing the current output.

[0101] Then, a compatibility function (such as the softmax function) is used to calculate the attention score:

[0102] e i =score(c i )=v T tanh(W c c i +b c )

[0103]

[0104] Among them, W c and b c are the weights and biases of the linear layer, and v is the weight vector of the compatibility function. i is the normalized attention score, which indicates the importance of the i-th encoder state to the current state of the decoder.

[0105] Finally, the attention score is used to weight the encoder state to obtain the context vector c context :

[0106]

[0107] Context vector c context Contains information about the encoder state that is most relevant to the decoder's current state, and is used by the decoder to generate the next output.

[0108] The context vector is combined with the input data of the current state to generate a multimodal representation that combines features of different modalities. This multimodal representation will be passed to the next sub-layer of the decoder to generate the final output.

[0109] By introducing the modal fusion module and attention mechanism, effective integration of data of different modalities is achieved, and the efficiency of information processing is improved.

[0110] In this application, the decoder is mainly used to generate fused feature representations. It receives the output of the encoder as input, and learns the fusion method between different modal features through multi-head self-attention and encoder-decoder attention mechanisms. Finally, the decoder outputs a fused feature vector, which contains the correlation between multimodal features and the spatial and temporal relationship between data. Specifically, the decoder first receives the output of the encoder, which contains the encoded representation of the input data. The decoder also needs to initialize a state vector, which will serve as the starting point of the decoding process.

[0111] The multi-head self-attention mechanism is first applied inside the decoder, allowing the decoder to pay attention to all previously generated outputs when processing the current output. This helps capture the dependencies within the output sequence and improves the accuracy of decoding.

[0112] Based on the multi-head self-attention and encoder-decoder attention mechanisms, the decoder fuses the features of different modalities. By iteratively applying the attention mechanism and feedforward neural network, the feature representation is gradually optimized and refined.

[0113] In each iteration, the decoder recalculates the attention weights and updates the feature representation based on the current feature representation and the encoder output. In each iteration, the decoder first applies the attention mechanism to calculate the attention weights between the current feature representation and the encoder output. Then, the encoder output is weighted summed according to these weights to obtain a new context vector. The decoder then combines the context vector with the feature representation of the previous iteration and performs a nonlinear transformation through a feedforward neural network.

[0114] This process is repeated multiple times until a predetermined number of iterations is reached or the feature representation reaches a preset convergence criterion.

[0115] The decoder outputs a final fused feature vector, which not only contains the correlation between multimodal features, but also contains the spatial and temporal relationship between data.

[0116] In the decoder, a multi-head self-attention mechanism is used to process the output of the encoder, that is, the intermediate representations that have undergone preliminary transformation. Through this mechanism, the decoder can learn and understand the intrinsic connections between these intermediate representations, laying the foundation for subsequent feature fusion.

[0117] The encoder-decoder attention mechanism is a bridge between the encoder and the decoder, which allows the decoder to refer to the encoder's output when generating output. This mechanism achieves effective transmission and integration of information by calculating the attention weight between the decoder's current state and all the encoder's output states.

[0118] Specifically, in the context of feature fusion, the encoder-decoder attention mechanism is used to guide the decoder to learn how to fuse features from different modalities. Through this mechanism, the decoder can selectively focus on the key information in the encoder output, thereby achieving effective integration of multimodal features.

[0119] The final output of the decoder is a fused feature vector, which contains not only the original feature information from different modalities, but also the correlation between these features and the spatial and temporal relationship between the data.

[0120] In the Transformer architecture, the Attention mechanism is a core component. It allows the model to dynamically adjust its attention when processing the input sequence, thereby capturing the key information in the sequence. In our system, we introduced the Attention Score to measure the correlation between different modal features. By calculating the attention score, the model can learn which modal features are more important in a specific task and adjust its fusion strategy accordingly.

[0121] The attention score is a numerical measure of the relevance between different elements. In Transformer, the attention score is usually obtained by calculating the dot product or cosine similarity between the query and the key. These scores are then usually scaled (for example, by dividing by the square root of the dimension of the key) and normalized by the softmax function to ensure that they sum to 1.

[0122] Attention score plays a crucial role in Transformer. It not only determines which elements are more important in a specific task, but also guides how the model integrates this information to generate the final output. By adjusting the attention score, the model can flexibly adapt to different input and task requirements, thereby improving its generalization ability and performance.

[0123] In this application, the Attention mechanism is used to process multimodal features. By calculating the attention scores between different modal features, the model can learn which modal features are more important in a specific task. This mechanism allows the model to dynamically adjust its fusion strategy to better integrate information from different modalities.

[0124] Since the Attention mechanism allows the model to dynamically adjust its attention when processing the input sequence, it can play a key role in multimodal feature fusion. By focusing on those modal features that are most relevant to the current task, the model can more effectively extract and utilize this information, thereby improving the accuracy and reliability of the fused features.

[0125] The execution module 104 inputs the graph structure data into the Transformer encoder, and the output of the Transformer encoder serves as the initial input of the Transformer decoder.

[0126] After constructing the encoder and decoder and introducing the Attention mechanism and GNN, the fused feature representation is generated. Specifically, the feature vectors of different modalities are used as the input of the encoder, and the association between them is learned through the multi-head self-attention mechanism. Then, the output of the encoder is passed to the decoder, and the fusion method between the features of different modalities is learned through the multi-head self-attention and encoder-decoder attention mechanisms.

[0127] Output module 105, the Transformer decoder outputs a fused feature vector, which contains the correlation between multimodal features and the spatial and temporal relationship between data.

[0128] The decoder outputs a fused feature vector, which is a crucial link in the reservoir ecological intelligent visual real-time monitoring system. This feature vector not only condenses rich information from multiple modalities, but also deeply reveals the correlation between these information and the complex relationship between data in spatial and temporal dimensions.

[0129] The working module 106 uses the feature vector in early warning or decision-making tasks.

[0130] In the early warning task, the working module 108 uses the information in the feature vector to identify possible anomalies or risks in the reservoir ecosystem. For example, if the feature vector shows that a certain indicator in the water quality data (such as dissolved oxygen content) has suddenly dropped, and at the same time, a large number of aquatic plants are observed to wither in the image data, and abnormal hydroacoustic activity is captured in the sound data, then the working module may trigger an early warning of water quality deterioration or ecological imbalance. This early warning can help managers take timely measures to prevent the problem from further deteriorating.

[0131] In the decision-making task, the working module 108 will formulate or adjust the reservoir management strategy based on the information in the feature vector. For example, if the feature vector shows that the number of fish in the reservoir has dropped sharply, and some indicators in the water quality data are also abnormal, then the working module may recommend increasing the frequency of water quality monitoring, restricting fishing activities, or taking other protective measures to restore the fish population. In addition, the working module can also predict the changing trend of the reservoir ecology in the future based on the comparison of historical data and current feature vectors, thereby providing a scientific basis for long-term management planning.

[0132] like Figure 2 As shown, the present application also provides a reservoir ecological intelligent visual real-time monitoring method, including:

[0133] Collect multimodal data related to reservoir ecology;

[0134] According to the multimodal data, constructing graph structure data based on sensor networks and ecological relationship networks;

[0135] Constructing a Transformer encoder, including a plurality of identical encoding layers stacked together, each encoding layer including a multi-head self-attention sublayer and a fully connected feed-forward network sublayer; introducing position encoding in the Transformer encoder;

[0136] Construct a Transformer decoder, which consists of multiple identical decoding layers stacked together. Each decoding layer contains a multi-head self-attention sublayer, an encoder-decoder attention sublayer, and a fully connected feedforward network sublayer.

[0137] Input the graph structure data into the Transformer encoder, and use the output of the Transformer encoder as the initial input of the Transformer decoder;

[0138] The Transformer decoder outputs a fused feature vector that contains the correlation between multimodal features and the spatial and temporal relationship between data;

[0139] The feature vector is used in early warning or decision-making tasks.

Claims

1. A reservoir ecological intelligent visual real-time monitoring system, characterized in that: include: The collection module collects multimodal data related to reservoir ecology; A graph data module, which constructs graph structure data based on sensor networks and ecological relationship networks according to the multimodal data; A neural network module is used to construct a Transformer encoder, which includes a plurality of identical encoding layers stacked together, each encoding layer including a multi-head self-attention sublayer and a fully connected feed-forward network sublayer; position encoding is introduced into the Transformer encoder; Construct a Transformer decoder, which consists of multiple identical decoding layers stacked together. Each decoding layer contains a multi-head self-attention sublayer, an encoder-decoder attention sublayer, and a fully connected feedforward network sublayer. An execution module inputs the graph structure data into the Transformer encoder, and the output of the Transformer encoder serves as the initial input of the Transformer decoder; Output module, the Transformer decoder outputs a fused feature vector, which contains the correlation between multimodal features and the spatial and temporal relationship between data; The working module is used in early warning or decision-making tasks according to the feature vector.

2. According to claim 1, a reservoir ecological intelligent visual real-time monitoring system is characterized in that: Collect multimodal data related to reservoir ecology, including: Collect water quality data including temperature, pH, dissolved oxygen content and turbidity through water quality sensors; Collect biological species data through biometric technology, including the types and numbers of fish, aquatic plants and microorganisms; Collect image data through high-definition cameras to monitor ecological changes in the reservoir area; Acoustic data is collected through acoustic sensors to monitor hydroacoustic activity and biological sounds in the reservoir area.

3. According to claim 1, a reservoir ecological intelligent visual real-time monitoring system is characterized in that: According to the multimodal data, a graph structure data based on a sensor network and an ecological relationship network is constructed, including: The sensors are used as nodes of graph structure data, and the communication or spatial relationship between sensors is used as the edge to construct the sensor network graph; The biological species are used as nodes of the graph structure data, and the food chains and symbiotic relationships between organisms are used as edges to construct an ecological relationship network diagram.

4. According to claim 1, a reservoir ecological intelligent visual real-time monitoring system is characterized in that: Build the Transformer encoder, including: In each encoding layer, multiple self-attention operations are performed in parallel using different weight matrices to form a multi-head self-attention mechanism; The representation of each position is transformed through a fully connected feed-forward network to generate a new representation.

5. According to claim 1, a reservoir ecological intelligent visual real-time monitoring system is characterized in that: The introducing of position coding in the encoder comprises: The position encoding is calculated using a combination of sine and cosine functions; The position code is added to each position of the input data, so that each input data contains the content and position information of the original data.

6. According to claim 1, a reservoir ecological intelligent visual real-time monitoring system is characterized in that: Build the Transformer decoder, including: In each decoding layer, a multi-head self-attention mechanism is used to process the encoder output to learn and understand the intrinsic connection between intermediate representations; The encoder-decoder attention mechanism is used to calculate the attention weight between the current state of the decoder and all output states of the encoder to achieve effective transmission and integration of information.

7. According to claim 1, a reservoir ecological intelligent visual real-time monitoring system is characterized in that: Inputting the graph structure data into the Transformer encoder includes: The nodes and edges in the graph structure data are converted into feature vectors as input of the Transformer encoder.

8. According to claim 1, a reservoir ecological intelligent visual real-time monitoring system is characterized in that: Based on the feature vector, it is used in warning or decision-making tasks, including: The fused feature vector is input into the early warning model, and an early warning is issued according to the preset thresholds and rules; Alternatively, the fused feature vectors can be input into a decision support system to assist decision makers in making ecological protection and management decisions.

9. According to claim 1, a reservoir ecological intelligent visual real-time monitoring system is characterized in that: Also includes: The collected multimodal data are preprocessed, including data cleaning, data format conversion, and data normalization, to improve data quality and model training effects.

10. A reservoir ecological intelligent visual real-time monitoring method, characterized in that: include: Collect multimodal data related to reservoir ecology; According to the multimodal data, constructing graph structure data based on sensor networks and ecological relationship networks; Constructing a Transformer encoder, including a plurality of identical encoding layers stacked together, each encoding layer including a multi-head self-attention sublayer and a fully connected feed-forward network sublayer; introducing position encoding in the Transformer encoder; Construct a Transformer decoder, which consists of multiple identical decoding layers stacked together. Each decoding layer contains a multi-head self-attention sublayer, an encoder-decoder attention sublayer, and a fully connected feedforward network sublayer. Input the graph structure data into the Transformer encoder, and use the output of the Transformer encoder as the initial input of the Transformer decoder; The Transformer decoder outputs a fused feature vector that contains the correlation between multimodal features and the spatial and temporal relationship between data; The feature vector is used in early warning or decision-making tasks.

Images