River pollution detection method based on multi-modal data fusion

Abstract

The invention discloses a river pollution detection method based on multi-modal data fusion, and belongs to the technical field of environment monitoring. The method comprises the following steps: firstly, collecting water quality sensor data, remote sensing image data and meteorological data, and preprocessing the data, including data cleaning, normalization and feature extraction; feature fusion is carried out on the preprocessed data, comprehensive features capable of representing the river pollution condition are extracted, and dimension reduction is carried out on the fused comprehensive features; in the model training and optimizing stage, a river pollution detection model based on fusion features is constructed, and the feature extraction capability and the calculation efficiency of the model are enhanced; training and optimizing the model by utilizing an optimization algorithm, and adjusting parameters and a structure of the model; finally, the optimized model is applied to actual river pollution detection, collected data are analyzed and processed in real time, river pollution events are detected, early warning is given out in time, and high-precision river pollution detection is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of environmental monitoring, and in particular to a river pollution detection method based on multimodal data fusion. Background Art

[0002] With the rapid advancement of industrialization and urbanization, river pollution is becoming increasingly serious, posing a significant threat to the ecological environment and human health. Traditional methods for detecting river pollution typically involve manual sampling combined with laboratory analysis. This model is not only labor-intensive and time-consuming, but also fails to meet the practical needs of real-time monitoring and rapid response.

[0003] In recent years, monitoring methods based on sensors and remote sensing technologies have gradually emerged and gained some application. However, in practical applications, the detection accuracy and reliability of these methods still need to be improved. To address this problem, this paper proposes a river pollution detection method based on multimodal data fusion, aiming to improve detection accuracy and reliability, and more efficiently address the challenges of river pollution detection. Summary of the Invention

[0004] The technical problem to be solved by the present invention is how to achieve high-precision river pollution detection, and provides a river pollution detection method based on multimodal data fusion.

[0005] The present invention solves the above technical problems through the following technical solutions, which include the following steps:

[0006] S1: Multimodal Data Acquisition

[0007] Collect multimodal data related to river pollution;

[0008] S2: Data Preprocessing

[0009] Preprocess the collected multimodal data;

[0010] S3: Multimodal Data Feature Fusion

[0011] The preprocessed multimodal data is fused at the feature level to obtain fused features;

[0012] S4: Fusion feature dimensionality reduction

[0013] After obtaining the fusion features, we design a contrast loss function and use the structural information of the data itself to learn the features and reduce the dimension of the fusion features.

[0014] S5: Feature Mapping Model Optimization and Training

[0015] The original ResNet12 network was selected as the baseline model, and the baseline model was improved to obtain an improved feature mapping model. The fusion features after dimension reduction were input into the improved feature mapping model for training to obtain the trained model, namely the river pollution detection model.

[0016] S6: Pollution detection and identification

[0017] The multimodal data actually collected are processed by S2, S3 and S4 and then input into the river pollution detection model to obtain the output index value. Based on the comparison between the output index value and the pollution index range, it is judged whether the river is polluted; when river pollution is detected, an early warning signal is issued in time.

[0018] Furthermore, in step S1, the multimodal data includes water quality sensor data, remote sensing image data, and meteorological data.

[0019] Furthermore, in step S2, the preprocessing includes data cleaning, normalization, and feature extraction operations.

[0020] Furthermore, in step S3, the specific process of feature-level fusion is as follows:

[0021] S31: The data features of different modes in the preprocessed multimodal data are used as nodes in the graph convolutional network. Each node represents the feature vector of a data sample, and the feature vectors of the nodes must be standardized. The edges between the nodes are constructed based on the correlation and spatiotemporal relationship between the data. The weights of the edges are used to reflect the similarity or correlation between the nodes. The cosine similarity is used for calculation. Then, the nodes and edges are combined into a graph structure to form a feature fusion graph g = (ζ, ψ), where ξ is the node set and ψ is the edge set.

[0022] S32: Initialize the initial states of the nodes covered in the feature fusion graph using linear transformation. For the weights of the edges, use the weight values determined when constructing the feature fusion graph as initial values for initialization.

[0023] S33: Feature propagation is carried out in the constructed feature fusion graph. Nodes receive information from neighboring nodes. Information is transmitted along edges, and edge weights influence the transmission strength. Nodes combine their current state with the received information and process it according to a dynamic neuron pulse emission mechanism. During feature propagation, dynamic neurons continuously update based on the strength and timing of the input signal, adjusting the node's own state and the weights of the edges between nodes. The dynamic neuron pulse emission mechanism can effectively capture the dynamic relationships and timing information between nodes.

[0024] S34: Adopt the Adam optimization algorithm combined with the set loss function for iterative update, learn the complex relationship and fusion pattern between multimodal data, and generate fusion features.

[0025] Furthermore, in step S31, assuming that the node feature vector is v, after normalization, a normalized feature vector v′ is obtained, and its calculation formula is:

[0026]

[0027] Where u is the mean of the node feature vector v, and σ is the standard deviation of the node feature vector v.

[0028] Furthermore, in step S32, the node's own state is adjusted as follows:

[0029] Assume that the state of node i at time t is h i (t) , after receiving information from neighboring nodes, the state of node i at time t+1 is updated to:

[0030]

[0031] Among them, N(i) is the set of neighbor nodes of node i, w i,j is the edge weight between nodes i and j, f is the nonlinear activation function, is the state of node i at time t+1.

[0032] Furthermore, in step S32, the weights of the edges between nodes are adjusted as follows:

[0033]

[0034] Where η is the learning rate, Δw i,j =α·s, α is the adjustment coefficient, s is the signal strength, They represent the weight of the edge from node i to node j at time t and time t+1 respectively.

[0035] Furthermore, in step S4, the specific processing process is as follows:

[0036] S41: Take the previously extracted fusion features as input, and treat each data sample as a node in the graph neural network. The node feature vector is the extracted fusion feature. Based on the correlation between the data samples, the edges between the nodes are constructed. The weight of the edge is used to represent the similarity between the nodes. The nodes and edges are combined into a graph structure g′=(γ,ε), which is used as the graph data for self-supervised learning.

[0037] S42: In the graph structure constructed in step S41, for each node γ 1 i ∈γ, select its neighbor nodes as positive sample pairs, select the nodes directly connected to the node as neighbor nodes, and assume that node γ 1 i The set of neighbor nodes is N(γ 1 i ), then the positive sample pair is (γ 1 i ,γ 1 j ), where γ 1 j ∈N(γ 1 i ); Randomly select non-neighbor nodes from the graph structure as negative sample pairs, and the negative sample pairs are (γ 1 i ,γ 1 k ), where γ 1 k ∈N(γ 1 i );

[0038] S43: Then, a contrast loss function is designed to measure the similarity of feature representation between positive sample pairs and negative sample pairs. The InfoNCE loss function is selected as the contrast loss function.

[0039] S44: Construct an encoder based on a multi-layer graph neural network structure to extract the feature representation of the node. The propagation rule of the encoder is:

[0040]

[0041] in, is node γ 1 i In the feature representation of layer l, w′ ij is node γ 1 i and γ 1 j The edge weight between ij is the normalization coefficient, d i is node γ 1 i Degree, W (l) is the trainable parameter matrix of layer l, σ(·) is the ReLU activation function, is a node In the feature representation of the l+1th layer, N(i) represents the set of neighbor nodes of node i;

[0042] S45: Add a dimensionality reduction layer after the output layer of the encoder to map the extracted feature representation to a low-dimensional space. The dimensionality reduction layer uses linear transformation for processing.

[0043] S46: Through self-supervised learning, the contrast loss function designed in step S43 is used for training to learn feature representations that can distinguish positive sample pairs from negative sample pairs, thereby achieving feature dimensionality reduction and ultimately obtaining dimensionality-reduced data.

[0044] Furthermore, in step S5, the convolutional layer of each block in the original ResNet12 network is improved to obtain an improved feature mapping model. The improved convolutional layer is processed as follows:

[0045] First, standard convolution with a kernel size of 3*3 is applied to 1 / 2 channels of the input feature map to obtain the first feature map. Then, depthwise separable convolution with a kernel size of 5*5 is applied to the remaining 1 / 2 channels to obtain the second feature map. The first and second feature maps are then concatenated in the channel dimension to generate a concatenated feature map. The channel dimension of the concatenated feature map is then shuffled.

[0046] Furthermore, the specific process of shuffling the channel dimension of the concatenated feature map is as follows:

[0047] S501: Convert the shape of the input feature map from [batch size, number of channels, height, width] to [batch size, number of groups, number of channels per group, height, width];

[0048] S502: permute the two dimensions of the number of groups and the number of channels per group to obtain the shape [batch size, number of channels per group, number of groups, height, width];

[0049] S503: Flatten the permuted feature map to the shape of [batch size, number of channels, height, width].

[0050] Compared with the prior art, the present invention has the following advantages:

[0051] 1. It integrates multimodal data related to river pollution, such as water quality sensor data, remote sensing image data, meteorological data, etc. Compared with the application of a single data type, it can more comprehensively reflect the river pollution status and provide a richer information basis for pollution detection and early warning.

[0052] 2. The features of different modal data are used as nodes to construct a feature fusion graph, and the node feature vectors are standardized. This solves the problem of differences in numerical range and scale between different modal data, makes the data features comparable and compatible, improves the accuracy and reliability of subsequent fusion operations, and helps to better explore the associations between data.

[0053] 3. The pulse emission mechanism of dynamic neurons is used to realize feature propagation and fusion, which can effectively capture the dynamic relationship and timing information between nodes, such as the changes in water quality data at different times in the river and the changes in water flow data at different locations. Compared with traditional static feature extraction methods, it can better reflect the actual dynamic changes of river pollution, making the feature representation more timely and representative.

[0054] 4. After obtaining the comprehensive features, dimensionality reduction processing is performed to reduce the data dimension and computational complexity. At the same time, by designing a contrast loss function and self-supervised learning, the structural information of the data itself is used for feature learning, thereby improving the efficiency and quality of feature representation.

[0055] 5. Optimize the feature mapping model structure, adopt operations such as deep separable convolution and perform channel shuffling, so that feature information of different groups can interact better, improving the feature expression ability. Compared with traditional convolutional neural networks, it can more effectively utilize limited labeled data, improve the accuracy and generalization ability of the model, and provide more powerful model support for pollution detection and early warning. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 1 is a flow chart of a river pollution detection method based on multimodal data fusion in an embodiment of the present invention;

[0057] Figure 2 This is a schematic diagram of the original ResNet12 deep network structure in an embodiment of the present invention;

[0058] Figure 3 This is a schematic diagram of the improved ResNet12 deep network structure in an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The following is a detailed description of an embodiment of the present invention. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process. However, the protection scope of the present invention is not limited to the following embodiment.

[0060] like Figure 1 As shown, this embodiment provides a technical solution: a river pollution detection method based on multimodal data fusion, comprising the following steps:

[0061] (1) Data collection and preprocessing

[0062] Collect multimodal data related to river pollution, including but not limited to water quality sensor data, remote sensing image data, meteorological data, etc. Preprocess the collected data, including data cleaning, normalization, feature extraction, etc., to improve data quality and usability.

[0063] (2) Data fusion and feature extraction

[0064] The preprocessed multimodal data is fused at the feature level. The specific process is as follows:

[0065] Step 1: Construct a feature fusion graph: First, treat the data features of different modalities in the preprocessed multimodal data as nodes in the graph convolutional network. Each node represents the feature vector of a data sample, and the feature vectors of the nodes must be standardized to ensure that the features of different modal data are comparable and compatible in terms of numerical range and scale, which facilitates subsequent fusion operations. Assuming that the node feature vector is v, after standardization, the standardized feature vector v′ can be obtained, which is calculated as follows:

[0066]

[0067] Where u is the mean of the eigenvector v, and σ is the standard deviation of the eigenvector v.

[0068] Next, based on the correlation between the data (such as the statistical correlation between different water quality chemical indicators) and the spatiotemporal relationship (such as the spatiotemporal connection between monitoring data at different locations in the river), the edges between the nodes are constructed. The weight of the edge is used to reflect the similarity or degree of correlation between the nodes. The cosine similarity (the similarity is measured by calculating the cosine value of the angle between the feature vectors of two nodes, with a value range of -1 to 1, and the closer to 1, the higher the similarity) can be used for calculation, thereby combining the nodes and edges into a graph structure to form a feature fusion graph g = (ζ, ψ), where ξ is the node set and ψ is the edge set, which can intuitively represent the complex relationships and interactions between multimodal data.

[0069] Step 2: Initialize the initial state of the nodes covered in the feature fusion graph, specifically using linear transformation for initialization:

[0070]

[0071] in, is the initial state of node i, γ i is the feature vector of the node, W is the weight matrix, and b is the bias vector. For the edge weight, the weight value determined when constructing the feature fusion graph is used as the initial value for initialization.

[0072] Subsequently, feature propagation is carried out in the constructed feature fusion graph. This process is mainly achieved by the pulse emission mechanism of dynamic neurons. The nodes receive information from neighboring nodes, and the information is transmitted through the edges. The edge weight affects the transmission intensity. The nodes combine their current state and the received information and process it according to the dynamic neuron pulse emission rules. The pulse emission mechanism of dynamic neurons can effectively capture the dynamic relationship and timing information between nodes. For example, in river pollution monitoring, it can capture dynamic information such as changes in water quality data at different time points and changes in water flow data at different locations. During fusion, the information from different modal data feature nodes is integrated through the weighted summation of dynamic neurons, and then transformed by a nonlinear activation function to generate a new node feature representation. It integrates multiple modal data information, has stronger expression capabilities, and can more comprehensively reflect the factors related to the river pollution status. Assume that the state of node i at the tth moment is After receiving information from neighboring nodes, the state of node i at time t+1 is updated to:

[0073]

[0074] Among them, N(i) is the set of neighbor nodes of node i, w i,j is the edge weight between nodes i and j, f is the nonlinear activation function, is the state of node i at time t+1.

[0075] Step 3: During the feature propagation and fusion process, dynamic neurons continuously update based on the strength of the input signal (reflecting the importance of information transmission between nodes) and time information (reflecting the timing characteristics of data), adjusting the node's own state and the weight of the edge between nodes. For example, when the signal strength received by a node from a certain modal data feature node suddenly increases, the dynamic neuron will adjust the weight of the edge between the node and the modal data feature node accordingly, highlighting the importance of this modal data in the current fusion process. The formula for adjusting the edge weight is:

[0076]

[0077] Where η is the learning rate, Δw i,j =α·s, α is an adjustment coefficient, s is the signal strength, They represent the weight of the edge from node i to node j at time t and time t+1 respectively.

[0078] At the same time, optimization algorithms (such as Adam) are combined with loss functions (the error between the predicted pollution level and the actual pollution level) for iterative updates to better learn the complex relationships and fusion patterns between multimodal data, and generate more accurate and representative comprehensive features. The final output comprehensive features can effectively characterize the river pollution status and provide strong support and basis for subsequent pollution monitoring, assessment and governance.

[0079] Step 4: After obtaining the comprehensive features, perform dimensionality reduction to reduce the data dimension and computational complexity. By designing a contrast loss function, the structural information of the data itself is used for feature learning. The specific process is as follows:

[0080] First, the previously extracted fused features (comprehensive features) are used as input, and each data sample is treated as a node in the graph neural network. The node's feature vector is the extracted fused feature. Next, edges are constructed between nodes based on the correlation between the data samples. Edge weights represent the degree of similarity between nodes, and cosine similarity can be used to calculate edge weights. In this way, the nodes and edges are combined into a graph structure, forming graph data for self-supervised learning. Through the above method, the nodes and edges are combined into a graph structure g′ = (γ, ε), where γ is the set of nodes and ε is the set of edges, forming the graph data for self-supervised learning.

[0081] Then, in the constructed graph, for each node γ 1 i ∈γ, select its neighbor nodes as positive sample pairs. The neighbor nodes can be determined by the topological structure of the graph, for example, select the nodes directly connected to the node as neighbor nodes. Assume that node γ 1 i The set of neighbor nodes is N(γ 1 i ), then the positive sample pair is (γ 1 i ,γ 1 j ), where γ 1 j ∈N(γ 1 i ).

[0082] At the same time, non-neighbor nodes are randomly selected from the graph as negative sample pairs, and the selection of negative sample pairs must ensure that they are sufficiently different from the positive sample pairs. The negative sample pairs are (γ 1 i ,γ 1 k ), where γ 1 k ∈N(γ 1 i ).

[0083] Then, a contrast loss function is designed to measure the similarity of feature representation between positive sample pairs and negative sample pairs. Common contrast loss functions such as the InfoNCE loss function are calculated as follows:

[0084]

[0085] Among them, z i and z j They are node γ i and γ j The feature representation of sim(z i ,z j ) represents the similarity between feature representations, usually using cosine similarity, τ is a temperature parameter used to control the smoothness of the similarity distribution. N is the number of negative samples, 1 [k≠i] It is an indicator function, which takes the value 1 when k≠i and 0 otherwise.

[0086] Next, we construct an encoder to extract the feature representation of the nodes. The encoder can use a multi-layer graph neural network structure, such as the convolutional network. The propagation rule of the encoder is:

[0087]

[0088] in, is node γ i In the feature representation of layer l, w′ ij is node γ i and γ j The edge weight between ij is the normalization coefficient, usually Among them, d i is node γ i Degree, W (l) is the trainable parameter matrix of layer l, σ(·) is the ReLU activation function, is node γ 1 i In the feature representation of the l+1th layer, N(i) represents the set of neighbor nodes of node i.

[0089] A dimensionality reduction layer is added after the output layer of the encoder to map the extracted feature representation to a low-dimensional space. The dimensionality reduction layer can use a linear transformation, for example:

[0090]

[0091] in, is the feature representation of the encoder output layer, W (dec) and b (dec) are the trainable parameters of the dimensionality reduction layer.

[0092] Finally, through self-supervised learning, training is performed using the previously designed contrastive loss function. During training, the model automatically adjusts its parameters to bring the feature representations of positive pairs closer together and those of negative pairs further apart, with the goal of minimizing the contrastive loss function. In this way, feature representations that distinguish between positive and negative pairs are learned, thus achieving feature dimensionality reduction and ultimately obtaining reduced-dimensionality data, reducing both data dimensionality and computational complexity.

[0093] (3) Feature mapping model optimization and training

[0094] Optimize the feature mapping model structure (the original ResNet12 deep network see Figure 2 For the sake of convenience, only one of the block structures is drawn. The network structure is composed of four identical block structures stacked together). It can effectively utilize limited labeled data to improve the accuracy of fault diagnosis. The improved network structure is shown in Figure 3 For the input of the convolution layer, a standard convolution operation is first performed to generate a partial feature map, then a depthwise separable convolution operation is performed to generate another partial feature map, then the two partial feature maps are spliced together, and finally the spliced feature maps are channel-shuffled so that the information generated by the standard convolution is fully mixed with the information generated by the depthwise separable convolution. The specific operation process is as follows:

[0095] First, use a standard convolution with a kernel size of 3*3 to obtain the features of 1 / 2 channels of the network input feature map Figure 1 , then use the depth-separable convolution with a kernel size of 5*5 to obtain the features for the remaining 1 / 2 channels Figure 2 , then the feature maps generated by the two parts are spliced in the channel dimension to generate a spliced feature map; then the channel dimension of the spliced feature map is shuffled. The specific process is:

[0096] First, convert the shape of the input feature map from [batch size, number of channels, height, width] to [batch size, number of groups, number of channels per group, height, width];

[0097] Then, the two dimensions of the number of groups and the number of channels per group are permuted to obtain the shape [batch size, number of channels per group, number of groups, height, width];

[0098] Finally, the permuted feature map is successfully flattened to the shape of [batch size, number of channels, height, width].

[0099] By rearranging the feature channels, feature information of different groups can interact better, thereby improving the feature expression capability.

[0100] The features after dimensionality reduction by multimodal fusion (1) and (2) are input into the improved feature mapping model, and the loss function (the error between the actual predicted value and the expected value) is defined. The loss function is minimized and iterative updates are performed to update the feature mapping model parameters, and finally the updated feature mapping model (optimized model) is obtained.

[0101] (4) Pollution detection and early warning

[0102] The collected multimodal input is processed through (1), (2), and (3) above to obtain the output index value. Based on the comparison between the output index value and the pollution index range, it is judged whether the river is polluted. When river pollution is detected, an early warning signal is issued in a timely manner to notify relevant departments to take appropriate measures to reduce the impact of pollution on the environment and human health.

[0103] In summary, the river pollution detection method based on multimodal data fusion in the above embodiment, during the data processing phase, first collects water quality sensor data, remote sensing image data, and meteorological data, and preprocesses this data, including data cleaning, normalization, and feature extraction. Feature fusion is then performed on the preprocessed data to extract comprehensive features that can characterize the river pollution status, and the fused comprehensive features are then subjected to dimensionality reduction. During the model training and optimization phase, a river pollution detection model based on fused features is constructed to enhance the model's feature extraction capabilities and computational efficiency. The model is trained and optimized using an optimization algorithm, adjusting its parameters and structure. Finally, the optimized model is applied to actual river pollution detection, performing real-time analysis and processing on the collected data, detecting river pollution events and issuing timely warnings, thereby achieving high-precision river pollution detection.

[0104] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A river pollution detection method based on multimodal data fusion, characterized in that: The following steps are involved: S1: Multimodal Data Acquisition Collect multimodal data related to river pollution; S2: Data Preprocessing Preprocess the collected multimodal data; S3: Multimodal Data Feature Fusion The preprocessed multimodal data is fused at the feature level to obtain fused features; S4: Fusion feature dimensionality reduction After obtaining the fusion features, we design a contrast loss function and use the structural information of the data itself to learn the features and reduce the dimension of the fusion features. S5: Feature Mapping Model Optimization and Training The original ResNet12 network was selected as the baseline model, and the baseline model was improved to obtain an improved feature mapping model. The fusion features after dimension reduction were input into the improved feature mapping model for training to obtain the trained model, namely the river pollution detection model. S6: Pollution detection and identification The multimodal data actually collected are processed by S2, S3 and S4 and then input into the river pollution detection model to obtain the output index value. Based on the comparison between the output index value and the pollution index range, it is judged whether the river is polluted; when river pollution is detected, an early warning signal is issued in time.

2. A river pollution detection method based on multimodal data fusion according to claim 1, characterized in that: In step S1, the multimodal data includes water quality sensor data, remote sensing image data, and meteorological data.

3. The river pollution detection method based on multimodal data fusion according to claim 1 is characterized in that: In step S2, preprocessing includes data cleaning, normalization, and feature extraction operations.

4. The river pollution detection method based on multimodal data fusion according to claim 1 is characterized in that: In step S3, the specific process of feature-level fusion is as follows: S31: The data features of different modes in the preprocessed multimodal data are used as nodes in the graph convolutional network. Each node represents the feature vector of a data sample, and the feature vectors of the nodes must be standardized. The edges between the nodes are constructed based on the correlation and spatiotemporal relationship between the data. The weights of the edges are used to reflect the similarity or correlation between the nodes. The cosine similarity is used for calculation. Then, the nodes and edges are combined into a graph structure to form a feature fusion graph g = (ζ, ψ), where ξ is the node set and ψ is the edge set. S32: Initialize the initial states of the nodes covered in the feature fusion graph using linear transformation. For the weights of the edges, use the weight values determined when constructing the feature fusion graph as initial values for initialization. S33: Feature propagation is carried out in the constructed feature fusion graph. Nodes receive information from neighboring nodes. Information is transmitted along edges, and edge weights influence the transmission strength. Nodes combine their current state with the received information and process it according to a dynamic neuron pulse emission mechanism. During feature propagation, dynamic neurons continuously update based on the strength and timing of the input signal, adjusting the node's own state and the weights of the edges between nodes. The dynamic neuron pulse emission mechanism can effectively capture the dynamic relationships and timing information between nodes. S34: Adopt the Adam optimization algorithm combined with the set loss function for iterative update, learn the complex relationship and fusion pattern between multimodal data, and generate fusion features.

5. The river pollution detection method based on multimodal data fusion according to claim 4 is characterized in that: In step S31, assuming that the node feature vector is v, after normalization, a normalized feature vector v′ is obtained, and its calculation formula is: Where u is the mean of the node feature vector v, and σ is the standard deviation of the node feature vector v.

6. The river pollution detection method based on multimodal data fusion according to claim 5 is characterized in that: In step S32, the node's own state is adjusted as follows: Assume that the state of node i at time t is h i (t) , after receiving information from neighboring nodes, the state of node i at time t+1 is updated to: Among them, N(i) is the set of neighbor nodes of node i, w i,j is the edge weight between nodes i and j, f is the nonlinear activation function, is the state of node i at time t+1.

7. The river pollution detection method based on multimodal data fusion according to claim 6 is characterized in that: In step S32, the weights of the edges between nodes are adjusted as follows: Where η is the learning rate, Δw i,j =α·s, α is the adjustment coefficient, s is the signal strength, They represent the weight of the edge from node i to node j at time t and time t+1 respectively.

8. A river pollution detection method based on multimodal data fusion according to claim 1 or 7, characterized in that: In step S4, the specific processing process is as follows: S41: Take the previously extracted fusion features as input, and treat each data sample as a node in the graph neural network. The node feature vector is the extracted fusion feature. Based on the correlation between the data samples, the edges between the nodes are constructed. The weight of the edge is used to represent the similarity between the nodes. The nodes and edges are combined into a graph structure g′=(γ,ε), which is used as the graph data for self-supervised learning. S42: In the graph structure constructed in step S41, for each node γ 1 i ∈γ, select its neighbor nodes as positive sample pairs, select the nodes directly connected to the node as neighbor nodes, and assume that node γ 1 i The set of neighbor nodes is N(γ 1 i ), then the positive sample pair is (γ 1 i ,γ 1 j ), where γ 1 j ∈N(γ 1 i ); Randomly select non-neighbor nodes from the graph structure as negative sample pairs, and the negative sample pairs are (γ 1 i ,γ 1 k ), where γ 1 k ∈N(γ 1 i ); S43: Then, a contrast loss function is designed to measure the similarity of feature representation between positive sample pairs and negative sample pairs. The InfoNCE loss function is selected as the contrast loss function. S44: Construct an encoder based on a multi-layer graph neural network structure to extract the feature representation of the node. The propagation rule of the encoder is: in, is node γ 1 i In the feature representation of layer l, w′ ij is node γ 1 i and γ 1 j The edge weight between ij is the normalization coefficient, d i is node γ 1 i Degree, W (l) is the trainable parameter matrix of layer l, σ(·) is the ReLU activation function, is node γ 1 i In the feature representation of the l+1th layer, N(i) represents the set of neighbor nodes of node i; S45: Add a dimensionality reduction layer after the output layer of the encoder to map the extracted feature representation to a low-dimensional space. The dimensionality reduction layer uses linear transformation for processing. S46: Through self-supervised learning, the contrast loss function designed in step S43 is used for training to learn feature representations that can distinguish positive sample pairs from negative sample pairs, thereby achieving feature dimensionality reduction and ultimately obtaining dimensionality-reduced data.

9. The river pollution detection method based on multimodal data fusion according to claim 8 is characterized in that: In step S5, the convolutional layer of each block in the original ResNet12 network is improved to obtain an improved feature mapping model. The improved convolutional layer processing method is as follows: First, standard convolution with a kernel size of 3*3 is applied to 1 / 2 channels of the input feature map to obtain the first feature map. Then, depthwise separable convolution with a kernel size of 5*5 is applied to the remaining 1 / 2 channels to obtain the second feature map. The first and second feature maps are then concatenated in the channel dimension to generate a concatenated feature map. The channel dimension of the concatenated feature map is then shuffled.

10. The river pollution detection method based on multimodal data fusion according to claim 9 is characterized in that: The specific process of shuffling the channel dimension of the concatenated feature map is as follows: S501: Convert the shape of the input feature map from [batch size, number of channels, height, width] to [batch size, number of groups, number of channels per group, height, width]; S502: permute the two dimensions of the number of groups and the number of channels per group to obtain the shape [batch size, number of channels per group, number of groups, height, width]; S503: Flatten the permuted feature map to the shape of [batch size, number of channels, height, width].

Images