River pollution detection method based on unmanned aerial vehicle inspection

Abstract

The embodiment of the invention discloses a riverway pollution detection method based on unmanned aerial vehicle inspection, and the method comprises the steps: firstly obtaining a label data set of a target riverway region, training an initial riverway pollution detection network based on the label data set after building the initial riverway pollution detection network, and obtaining a trained riverway pollution detection network; and inputting the obtained current multi-modal image of the river channel region into a river channel pollution detection network for pollution detection to obtain a river channel pollution detection result. The river pollution detection network optimizes a multi-modal feature fusion mechanism, and can avoid information interference caused by insufficient utilization of feature scale sensitivity, thereby effectively improving pollution detection precision. Meanwhile, the network is adaptive to edge equipment such as an unmanned aerial vehicle, the current multi-mode image of the river channel can be quickly processed, real-time pollution detection and early warning are realized, and the timeliness and practicability of river channel pollution monitoring are enhanced.

Description

Technical Field

[0001] The embodiments of the present application relate to, but are not limited to, the fields of river pollution detection and artificial intelligence, and in particular to a river pollution detection method based on drone inspection. Background Art

[0002] In the field of river pollution detection, semantic segmentation methods based on visible light (RGB) images are widely used because they can detect pollution by identifying anomalies in water color and texture. However, this method is susceptible to interference from environmental factors such as lighting changes, specular reflections, and haze obscuration, resulting in a high false detection rate. To improve detection robustness, some studies have introduced thermal infrared (TIR) ​​modal data, utilizing the temperature difference between the polluted area and the background water to assist in detection.

[0003] However, existing multimodal fusion technologies suffer from numerous drawbacks. Traditional fusion methods (such as feature splicing and early fusion) fail to fully exploit the unique characteristics of RGB and TIR data at different scales. Direct superposition or concatenation can easily interfere with local details and global semantic information, making it difficult to effectively distinguish between real pollution and reflective artifacts. Furthermore, existing fusion strategies employ fixed weight distribution and are unable to dynamically adjust the contributions of each modality based on the scene. Furthermore, existing methods exhibit high computational complexity, making real-time inference difficult on edge devices such as drones. Furthermore, they lack a systematic preprocessing process for multimodal data, resulting in insufficient model generalization and an inability to adapt to monitoring needs across different seasons and weather conditions.

[0004] In summary, current technologies have obvious bottlenecks in multimodal feature fusion, scale-adaptive modeling, and real-time processing capabilities, which seriously restrict the accuracy and practical application effects of river pollution detection. Summary of the Invention

[0005] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0006] The embodiments of the present application provide a river pollution detection method based on drone inspection, which can effectively improve the accuracy of pollution detection and enhance the timeliness and practicality of river pollution monitoring.

[0007] An embodiment of the present application provides a river pollution detection method based on drone inspection, including: obtaining a label data set of a target river area; building an initial river pollution detection network, and training the initial river pollution detection network based on the label data set to obtain a trained river pollution detection network; obtaining a current multimodal image of the river area, inputting the current multimodal image into the river pollution detection network to perform pollution detection, and obtaining a river pollution detection result.

[0008] In one embodiment of the present application, the label data set of the target river area is obtained according to the following steps: acquiring training multimodal images, the training multimodal images including optical images and thermal infrared images, the training multimodal images being collected by a drone when patrolling along a preset route of the target river area; annotating the optical images pixel by pixel according to preset category labels to obtain label images; performing an affine transformation operation on the thermal infrared images according to the label images to obtain a corrected image; screening and verifying the label images and the corrected images to obtain a label data set.

[0009] In one embodiment of the present application, the label dataset includes the label image and the correction image of the same scene of the target river channel area; after obtaining the label dataset of the target river channel area, the method further includes: performing pixel-level spatial registration processing on the label image and the correction image respectively to obtain a registered label image and a registered correction image; performing brightness normalization and anti-interference processing on the registered label image to obtain the optimized registered label image; performing temperature radiation correction and denoising processing on the registered correction image to obtain the optimized registered correction image; performing size standardization and key area extraction processing on the optimized registered label image and the registered correction image to obtain a standard dataset.

[0010] In one embodiment of the present application, the initial river pollution detection network is trained based on the labeled data set to obtain the trained river pollution detection network, including: dividing the labeled data set into a training set, a validation set and a test set; using the training set to perform supervised training on the initial river pollution detection network, according to a preset learning rate rule, calculating a loss value based on the training results of the supervised training and the overall loss function, updating the parameters of the initial river pollution detection network according to the loss value to obtain a trained detection network, and the overall loss function is a weighted sum of a first cross entropy loss function, a second cross entropy loss function and a modal consistency loss function; verifying the trained detection network according to the validation set to obtain a verification result, adjusting the parameters of the detection network according to the verification result to obtain the optimized detection network; using the test set to evaluate the optimized detection network to obtain a performance evaluation result; when the performance evaluation result shows that it meets the standard, the optimized detection network is determined as the river pollution detection network.

[0011] In one embodiment of the present application, the river pollution detection network includes an image pre-segmentation module, a multimodal feature encoding module, a feature fusion module, a dual-branch context aggregation module and a multi-scale decoding module; the current multimodal image includes a current optical image and a current thermal infrared image of the river area; the current multimodal image is input into the river pollution detection network for pollution detection to obtain a river pollution detection result, including: inputting the current optical image and the current thermal infrared image into the river pollution detection network, using the image pre-segmentation module to segment the current optical image to obtain a river mask; using the river mask as a constraint condition, using the multimodal feature encoding module to extract features from the current optical image and the current thermal infrared image respectively to obtain a first feature and a second feature. two features; using the feature fusion module to perform feature fusion processing on the first feature and the second feature to obtain a multimodal feature; using the dual-branch context aggregation module to perform local and global feature extraction on the multimodal feature respectively to obtain local feature information and global feature information; splicing the local feature information and the global feature information to obtain a context enhancement feature; using the multi-scale decoding module to decode the context enhancement feature to obtain multi-scale feature information and a global context decoding feature; splicing and feature compression processing the global context decoding feature, the context enhancement feature and the multi-scale feature information to obtain a prediction feature; performing multiple upsampling operations on the prediction feature to obtain a pollution prediction map; obtaining a river pollution detection result based on the pollution prediction map.

[0012] In one embodiment of the present application, before inputting the current multimodal image of the river area into the river pollution detection network for pollution detection, the method includes: performing image registration processing on the current optical image and the current thermal infrared image based on feature point matching and affine transformation algorithm to obtain the registered current optical image and the current thermal infrared image; performing illumination normalization processing on the registered current optical image, and performing radiation correction on the registered current thermal infrared image to obtain a standard detection optical image and a standard detection thermal infrared image.

[0013] In one embodiment of the present application, the feature fusion module includes a multi-scale feature enhancement submodule and a spatial attention fusion submodule; the use of the feature fusion module to perform feature fusion processing on the first feature and the second feature to obtain a multimodal feature includes: using the multi-scale feature enhancement submodule to perform feature enhancement processing on the first feature and the second feature respectively to obtain a first enhanced feature and a second enhanced feature; using the spatial attention fusion submodule to perform feature extraction processing on the first enhanced feature to obtain a first attention feature, and performing feature extraction processing on the second enhanced feature to obtain a second attention feature; fusing the first attention feature and the second attention feature to obtain a multimodal feature.

[0014] In one embodiment of the present application, after obtaining the multimodal features, the method further includes: performing a convolution operation on the multimodal features according to a preset void ratio to obtain convolution features; and performing splicing and channel compression processing on the convolution features to obtain the optimized multimodal features.

[0015] In one embodiment of the present application, the multi-scale decoding module is used to decode the context enhancement feature to obtain multi-scale feature information and a global context decoding feature, including: performing a convolution operation of different step lengths on the context enhancement feature to obtain multi-scale feature information, where the multi-scale feature information includes a first scale feature and a second scale feature, and the spatial resolution of the first scale feature and the second scale feature are different; downsampling the context enhancement feature and the first scale feature respectively to obtain the downsampled context enhancement feature and the first scale feature; splicing the second scale feature, the downsampled context enhancement feature, and the first scale feature in the channel dimension to obtain a spliced ​​feature; and obtaining a global context decoding feature based on the spliced ​​feature.

[0016] In one embodiment of the present application, after obtaining the river pollution detection results, the method further includes: issuing an early warning for the river area where the pollution data in the river pollution detection results exceeds a preset threshold, and generating a trend analysis report based on the pollution source location displayed by the river pollution detection results.

[0017] The present invention provides a method for detecting river pollution based on drone inspections, which has the following beneficial effects: (1) The river pollution detection network optimizes the multimodal feature fusion mechanism, which can avoid information interference caused by insufficient utilization of feature scale sensitivity, accurately distinguish between real pollution and reflective artifacts, and effectively improve pollution detection accuracy. (2) The method is compatible with edge devices such as drones and can quickly process the current multimodal images of the river, achieving real-time pollution detection and early warning, thereby enhancing the timeliness and practicality of river pollution monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a flow chart of a river pollution detection method based on drone inspection provided in an embodiment of the present application;

[0019] Figure 2 This is a specific flow chart for obtaining a label data set provided by an embodiment of the present application;

[0020] Figure 3 This is a specific flow chart of label dataset preprocessing provided by one embodiment of the present application;

[0021] Figure 4 This is an architectural diagram of a river pollution detection network provided by one embodiment of the present application;

[0022] Figure 5 This is a structural diagram of a multimodal feature encoding module provided by one embodiment of the present application;

[0023] Figure 6 This is a structural diagram of a feature fusion module provided by an embodiment of the present application;

[0024] Figure 7 This is a structural diagram of a dual-branch context aggregation module provided by an embodiment of the present application;

[0025] Figure 8 This is a structural diagram of a multi-scale decoding module provided by an embodiment of the present application;

[0026] Figure 9 is a structural diagram of a multi-scale decoding module provided by another embodiment of the present application;

[0027] Figure 10 This is a flow chart of the training process of a river pollution detection network provided by one embodiment of the present application;

[0028] Figure 11 This is an embodiment of the present application. Figure 1 Specific flow chart of step 130;

[0029] Figure 12 This is an embodiment of the present application. Figure 11 Detailed flowchart of step 1130;

[0030] Figure 13 This is an overall flow chart of river pollution detection provided by an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0032] It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that in the flowchart. The terms "first" and "second" in the specification, claims, and the above-mentioned figures are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be noted that the structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the content disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the limitations of the implementation of this application. Therefore, they have no technical significance. Any modification of the structure, change in the proportional relationship, or adjustment of the size should still fall within the scope of the technical content disclosed in this application without affecting the efficacy and purpose of this application. At the same time, the terms such as "upper", "lower", "left", "right", "middle", and "one" cited in this specification are only for the convenience of description and are not used to limit the scope of the implementation of this application. Changes or adjustments in their relative relationships should also be considered as the scope of the implementation of this application without substantially changing the technical content.

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0034] With the rapid development of drones and remote sensing technology, river pollution monitoring has gradually shifted from traditional manual inspections to intelligent and automated methods. Among existing technologies, semantic segmentation methods based on RGB images are widely used to identify water color and texture anomalies (such as oil film reflections and algae aggregations) to determine pollution. However, single-modality RGB data is susceptible to interference from changes in ambient lighting, mirror reflections on the water surface, and obstructions from fog and haze, resulting in an increased false detection rate. For example, under strong lighting conditions, reflections from clean water may be mistaken for oil pollution; and in rainy weather, pollutants are easily missed due to reduced contrast.

[0035] To improve robustness, some studies have incorporated TIR modal data, leveraging temperature differences between polluted areas and background water bodies (e.g., industrial thermal drainage and oil film thermal radiation) to aid detection. However, existing multimodal fusion methods (e.g., feature splicing and early fusion) suffer from significant drawbacks: they underutilize feature scale sensitivity. RGB data can capture pollutant details (e.g., garbage texture) at local, high-resolution scales, while TIR data, due to its physical properties of thermal radiation detection, can effectively detect temperature anomalies (e.g., thermal drainage diffusion). However, due to the typically low spatial resolution of its sensors, it is more suitable for capturing large-scale temperature variations at medium- to low-resolution scales. Existing methods often directly superimpose or concatenate the two modal features, resulting in interference between local details and global semantic information, making it difficult to distinguish between real pollution and reflective artifacts. Furthermore, existing fusion strategies (e.g., average weighting) typically employ fixed weighting and lack the adaptability to dynamically adjust the contributions of each modality based on the scenario. For example, at night or in hazy conditions, the TIR modality should be given a higher weight to improve detection reliability. However, traditional methods struggle to achieve this adaptive adjustment, leading to confusion between the different modal features. The ability to detect boundaries and small targets is insufficient. Mainstream semantic segmentation networks (such as U-Net and DeepLab) rely on single-scale context aggregation and have limited ability to capture the edges of pollution diffusion (such as oil film boundaries) and scattered small targets (such as floating garbage), resulting in blurred boundaries and frequent holes in the segmentation results.

[0036] At the engineering application level, existing methods suffer from high computational complexity, making real-time inference difficult to achieve on edge devices such as drones. For example, while Transformer-based models can improve segmentation accuracy, the number of parameters and computational overhead limit their deployment feasibility. Furthermore, the lack of systematic preprocessing procedures for multimodal data (such as illumination normalization and cross-modal pixel alignment) limits the model's generalization capabilities, making it difficult to adapt to monitoring tasks in different seasons and weather conditions.

[0037] In summary, existing technologies have significant bottlenecks in multimodal feature fusion, scale-adaptive modeling, and real-time processing capabilities, which restrict the accuracy and practicality of river pollution detection.

[0038] In view of this, an embodiment of the present application provides a river pollution detection method based on drone inspection, and the specific steps are as follows: first, obtain a label data set of the target river area, and after building an initial river pollution detection network, train the initial river pollution detection network based on the label data set to obtain a trained river pollution detection network, and then input the current multimodal image of the acquired river area into the river pollution detection network for pollution detection to obtain the river pollution detection result. The river pollution detection network of the embodiment of the present application optimizes the multimodal feature fusion mechanism, which can avoid information interference caused by insufficient utilization of feature scale sensitivity, accurately distinguish between real pollution and reflective artifacts, and effectively improve the accuracy of pollution detection. At the same time, the network is compatible with edge devices such as drones, and can quickly process the current multimodal images of the river, realize real-time pollution detection and early warning, and enhance the timeliness and practicality of river pollution monitoring.

[0039] The embodiments of the present application are further described below with reference to the accompanying drawings.

[0040] Reference Figure 1 , Figure 1 This is a flow chart of a river pollution detection method based on drone inspection provided by an embodiment of the present application. The flow chart may specifically include but is not limited to steps 110 to 130.

[0041] Step 110: Obtain a label dataset of the target river area;

[0042] Step 120: Building an initial river pollution detection network, and training the initial river pollution detection network based on the labeled data set to obtain a trained river pollution detection network;

[0043] Step 130: Acquire a current multimodal image of the river area, and input the current multimodal image into a river pollution detection network to perform pollution detection, thereby obtaining a river pollution detection result.

[0044] Steps 110 to 130 are described in detail below.

[0045] The target river area is the specific river area selected for monitoring and analysis during a river pollution detection task. This can be a section of a river, a specific lake area, or a specific area of ​​water associated with a river. It is the specific spatial area that the entire detection plan focuses on, aiming to detect pollution within that area and obtain information on its pollution status.

[0046] In a possible embodiment, if Figure 2 As shown, the process of acquiring the label dataset of the target river area may include but is not limited to steps 210 to 240.

[0047] Step 210: Acquire training multimodal images, wherein the training multimodal images include optical images and thermal infrared images, and the training multimodal images are collected by the UAV when patrolling along a preset route of the target river area;

[0048] Step 220: labeling the optical image pixel by pixel according to a preset category label to obtain a labeled image;

[0049] Step 230: performing an affine transformation operation on the thermal infrared image according to the label image to obtain a corrected image;

[0050] Step 240: Screen and verify the labeled image and the corrected image to obtain a labeled dataset.

[0051] In a feasible embodiment, in step 210, in the river pollution monitoring scenario, a drone equipped with an optical (RGB) sensor and a thermal infrared (TIR) ​​sensor can be used to carry out multimodal data collection operations based on a pre-planned inspection route of the target river area. Specifically, the optical sensor captures the electromagnetic radiation information in the visible light band in the target river area and generates an optical image in the form of RGB or multispectral imaging, which can clearly present spatial features such as the surface texture of the water body, the shape of floating objects, and the details of riverbank features. The thermal infrared sensor detects the radiation energy distribution of the target river area in the long-wave infrared band based on the thermal radiation characteristics of the object itself, forming a thermal infrared image, which can effectively identify thermal characteristics such as the temperature difference between polluted water bodies and normal water bodies, the diffusion range of industrial hot drainage, and the abnormal thermal radiation of oil films. The image data collected synchronously by the two types of sensors constitute a multimodal image, which provides complementary feature information for subsequent river pollution detection.

[0052] In one feasible embodiment, a multispectral drone platform equipped with optical and thermal infrared sensors can periodically patrol and photograph a target river section, capturing multimodal images of the same location at each time point. Specifically, the drone can hover over key locations according to a pre-set route, capturing images at a fixed angle and constant altitude, effectively minimizing interference caused by angular deviation and scale variations.

[0053] In one feasible embodiment, in step 220, a professional remote sensing annotation tool (e.g., LabelMe) can be used to perform detailed pixel-by-pixel annotation on the optical image. During this process, category labels such as "polluted water," "clean water," and "river structure" can be pre-set. Each pixel in the optical image is then accurately classified based on these categories, with information about the different categories assigned to the corresponding pixels. This ultimately creates a labeled image with detailed category information. This labeled image clearly reflects the attributes of each region in the optical image.

[0054] In a feasible embodiment, when annotating the optical image pixel by pixel, not only can the pixels be classified according to the preset category labels (such as "polluted water body", "clean water body", "river structure", etc.), but more detailed information recording can also be carried out simultaneously. For example, for the pixel area marked as "polluted water body", the specific type of pollution is further clarified, such as oil, garbage or algae. At the same time, the distribution pattern of these pollutions in the optical image is observed and recorded to infer its diffusion trend, such as the change of the diffusion boundary of oil pollution, the extension direction of garbage accumulation, the increase or decrease trend of algae coverage area, etc. These pollution types and diffusion trend information can be used as additional annotation attributes and recorded in the label image together with the category label of the pixel point, so that the label image contains richer pollution detail information.

[0055] In a feasible embodiment, since there may be a spatial position deviation when the optical image and the thermal infrared image are collected, image registration is required to make the two modal images accurately correspond. In step 230, the annotated label image can be used as a reference to perform an affine transformation operation on the thermal infrared image. By adjusting the parameters such as rotation, translation, and scaling of the image, it is ensured that the thermal infrared image and the optical image are strictly aligned in space, and the registration error is controlled within the range of ≤n pixels (such as n=2). After the affine transformation processing, the corrected image obtained is consistent with the optical image and the label image in spatial position, which facilitates the subsequent joint analysis of the information of the two modal images and enhances the semantic association between the modalities.

[0056] It is understandable that before the affine transformation operation is performed on the thermal infrared image, the recorded pollution type, diffusion trend and additional meta-information (such as lighting conditions and weather conditions) play an important role. Meta-information such as lighting conditions and weather conditions can help determine the environmental differences between the optical image and the thermal infrared image when they are collected, so that the parameters can be adjusted more reasonably during the affine transformation process to ensure that the two modal images can be accurately aligned in different environments. After the thermal infrared image is aligned with the optical image after affine transformation, combined with information such as pollution type and diffusion trend information, the feature differences and associations of different pollution types in optical and thermal infrared modes can be analyzed in more depth, such as the correspondence between the color and texture features of oil pollution in optical images and the temperature anomaly features in thermal infrared images, as well as the consistency of the algae diffusion trend in the two modal images, thereby providing a more comprehensive and accurate semantic association basis for subsequent river pollution detection and analysis based on multimodal images.

[0057] In a feasible embodiment, in step 240, after completing the optical image annotation and thermal infrared image correction, the generated label image and correction image (i.e., multimodal image) can be further quality managed. Specifically, the label quality review work is first carried out to manually check whether there is a spatial misalignment problem between the label image and the correction image, for example, whether the contaminated area in the label image accurately matches the corresponding area of ​​the corrected thermal infrared image, and at the same time check whether there is ambiguity in the label boundary. If such samples are found, they are removed from the dataset to avoid low-quality data affecting the subsequent model training effect. Furthermore, after the quality review is completed, a certain proportion of image samples can be randomly sampled, and the annotation results can be checked again by the annotator or other reviewers, focusing on verifying the consistency of the annotation results of different annotators on the same image, as well as the consistency of the annotations of the same annotator at different times, to ensure the reliability and stability of the annotation results. After the above screening and verification process, the label images and correction images that meet the quality requirements are integrated with the pollution attribute metadata such as pollution type, diffusion trend, lighting conditions, weather conditions, etc. recorded during the annotation process to construct an image-mask pair dataset, i.e., the label dataset.

[0058] In a feasible embodiment, through steps 210 to 240, a multispectral UAV platform equipped with RGB and TIR sensors is used to periodically cruise and photograph the target river area according to a preset route to obtain multimodal images; remote sensing annotation tools such as LabelMe are used to annotate the optical images pixel by pixel, define category labels such as "polluted water body", and record pollution type, diffusion trend, and metadata such as lighting and weather. The thermal infrared image and the optical image are strictly aligned through affine transformation; after the annotation is completed, the annotation quality is reviewed, spatial misalignment and boundary fuzzy samples are eliminated, random sampling is performed to verify the annotation consistency, and finally the multimodal images, pixel-level label maps and pollution attribute metadata are integrated to construct a label dataset for the target river area, providing high-quality supervision signals for subsequent multimodal feature fusion and model training.

[0059] In a feasible embodiment, the label dataset includes a label image and a correction image of the same scene in the target river area. The former is generated by RGB image processing, and the latter is derived from TIR image processing. In order to enhance the ability of the initial river pollution detection network to extract river pollution features, the label image and correction image in the label dataset can be preprocessed to obtain a standard dataset. The preprocessing process is as follows: Figure 3 As shown, it may include but not be limited to steps 310 to 340.

[0060] Step 310: performing pixel-level spatial registration processing on the label image and the correction image respectively to obtain a registered label image and a registered correction image;

[0061] Step 320: performing brightness normalization and anti-interference processing on the registration label image to obtain an optimized registration label image;

[0062] Step 330: performing temperature radiation correction and denoising processing on the registration-corrected image to obtain an optimized registration-corrected image;

[0063] Step 340: performing size standardization and key area extraction processing on the optimized registration label image and the registration correction image to obtain a standard data set.

[0064] It is understandable that due to the difference in sensor perspective during the acquisition process, there is a spatial position deviation between the label image and the correction image. Therefore, in step 310, a feature point matching and affine transformation algorithm can be used to extract feature points from the image and establish a corresponding relationship. The image can be rotated, translated, and scaled to accurately align the two at the pixel level. The registration error is strictly controlled within a range of ≤2 pixels. This eliminates spatial offset and ensures that the multimodal data is completely consistent in spatial position. The spatially aligned registered label image and registered correction image are obtained, laying the foundation for subsequent analysis.

[0065] In one feasible embodiment, in step 320, brightness normalization can be used to adjust the image brightness to eliminate color deviations caused by illumination variations and achieve a more stable and uniform color representation. Furthermore, to address interfering factors such as haze and water reflections in the label image, an adaptive dehazing algorithm is employed to remove the effects of haze on image clarity, and specular reflection suppression techniques are employed to reduce the effects of water reflections on image quality. This effectively improves image quality and readability, resulting in an optimized, registered label image that accurately reflects the true characteristics of the river channel.

[0066] In one feasible embodiment, in step 330, considering that the dynamic range of temperature radiation values ​​in the registered and corrected images may vary, a radiation correction algorithm can be used to process them, unify the range of temperature radiation values, and enhance comparability between different images. Furthermore, to address thermal noise present in the registered and corrected images, Gaussian filtering and non-local means denoising can be applied sequentially. The former reduces noise by smoothing the image, while the latter further removes noise by leveraging image similarity, effectively improving image quality and obtaining an optimized registered and corrected image.

[0067] In a feasible embodiment, to facilitate subsequent data analysis and model training, the optimized registered label image (size H×W×3) and the registered correction image (size H×W×1) can first be adjusted to the same resolution through size standardization and cropping operations to ensure that the two images are consistent in size specifications. Then, based on the prior information of the river channel contour, the region of interest (ROI) is extracted from the image, that is, the main body of the river channel and the surrounding areas where pollution may exist are focused, and irrelevant background information is eliminated to reduce data redundancy and irrelevant information interference. Finally, the processed label image and correction image are integrated together with relevant metadata to form a standard dataset.

[0068] In a feasible embodiment, the river pollution detection network can perform multimodal semantic segmentation tasks, and its overall framework is as follows: Figure 4 As shown in Figure 1, it consists of five parts: image pre-segmentation module, multimodal feature encoding module, feature fusion module, dual-branch context aggregation module and multi-scale decoding module. Figure 4 In [1], RGB and TIR images are optical and thermal infrared images from the preprocessed labeled dataset (i.e., the standard dataset), respectively. The initial river pollution detection network performs the following tasks: First, a single-channel RGB image is used as input, and a lightweight semantic segmentation network, DeeplabV3-MobileNet, is used to generate a river mask M. Under the constraint of the river mask M, the RGB and TIR images corresponding to the region are cropped and padded to a size of H×W×3, and the image features f are extracted using the ResNet-18 backbone. rgb 、f tir Input RGB image features f into the feature fusion module rgb and TIR image features f tir , perform adaptive weighted fusion on multimodal features to obtain multimodal fusion features f aligned . Multimodal fusion feature f aligned Through the dual-branch context aggregation module, the context-enhanced feature f is obtained context , which integrates short-range (local) and long-range (global) context information. Context-enhanced feature f context After being processed by the multi-scale decoding module, the obtained f SSM The original context-enhanced feature f output by the dual-branch context aggregation module context and scale characteristics The images are spliced, and then compressed through a multi-layer perceptron and convolution layer. Finally, the original input size is restored through upsampling operation to generate a segmentation map of the river pollution area, which is recorded as

[0069] The following is a detailed introduction to each module in the river pollution detection network.

[0070] (1) Image pre-segmentation module

[0071] In this module, a single-channel RGB image is used as input and a lightweight semantic segmentation network DeeplabV3-MobileNet is used to generate a river mask M. river :

[0072] M river =f fiver (I rgb )∈{0,1} H×W (1),

[0073] Among them, M river (x,y)=1 indicates that the pixel belongs to the river channel, and subsequent pollution detection is only carried out in this area, effectively removing background interference.

[0074] (2) Multimodal feature encoding module

[0075] This module mainly extracts effective features from RGB images and TIR images. The structure is as follows Figure 5 As shown. In the river mask M river Under the constraint of river = 1 area, the RGB and TIR images corresponding to the area are cropped and padded to the size of H×W×3, respectively, and passed through the ResNet-18 encoder and 1×1 convolution to obtain their respective feature representations as shown in formula (2):

[0076]

[0077] These two branches are responsible for capturing the basic texture, color, and thermal characteristics information of their respective modalities, providing basic expression for subsequent fusion.

[0078] (3) Feature fusion module

[0079] In response to the fixed weight allocation defects and environmental interference sensitivity problems of traditional multimodal fusion methods in river pollution detection, this module proposes an adaptive fusion strategy based on the channel-space dual attention mechanism. By dynamically evaluating the contribution weights of RGB and TIR modalities (such as enhancing TIR thermal radiation response at night and strengthening RGB texture details under strong light), combining local convolution with global feature screening, it effectively suppresses water surface reflection and thermal noise interference, and realizes efficient coordination of multimodal features, such as Figure 6 shown.

[0080] The process starts with a multi-scale feature enhancement module (FE) of different scales, which first performs a 3×3 or 5×5 depthwise convolution on the input feature ReLU activation, and then compresses the number of channels through a 1×1 convolution:

[0081] f pw =Conv1×1 (DepthwiseConv k×k (ReLU(A)) (3),

[0082] Where A is represented by f rgb or f tir , DepthwiseConv k×k It is represented as k×k depth-wise separable convolution, k∈{3,5}, stride 1, padding 1, Conv 1×1 Represented as 1×1 convolution,

[0083] Generate a binary mask based on input features Only the convolution operation is performed on the area marked as 1 in the mask, and the obtained feature f partial Perform batch normalization:

[0084]

[0085] f FE =BN(f partial ) (5),

[0086] Among them, ∈ prevents division by zero error, W1 is the convolution kernel weight, BN is batch normalization,

[0087] After the multi-scale feature enhancement module, the different modal features at the same scale are further processed using the spatial attention fusion module (SA) to select important multimodal information. The following operations are performed independently for each branch: the features are input to the spatial attention fusion module, which first performs maximum pooling and average pooling and concatenates the two pooling results along the channel dimension:

[0088] f cat =Concat([AvgPool(B)],[MaxPool(B)]) (6),

[0089] Among them, B represents f1 or f2, AvgPool represents average pooling, MaxPool represents maximum pooling, Concat represents feature concatenation, and f cat ∈R 1×1×256 .

[0090] The weight map is then generated through 1×1 convolution and Sigmoid activation:

[0091] W raw =σ1(Conv 1×1 (f cat )) (7),

[0092] Among them, σ1 is the Sigmoid activation function, Wra2 ∈R 1×1×128 .

[0093] Finally, the weight map is broadcasted to the spatial dimension of the input features and multiplied element-wise:

[0094] f SA =B⊙Broadcast(W raw ) (8),

[0095] in,

[0096] The feature f output from the spatial attention fusion module h and f l Add element by element and obtain the weighted probability matrix W through 1×1 convolution and Sigmoid activation function:

[0097]

[0098] W=σ1(Conv 1×1 (f sum )) (10),

[0099] in, Expressed as element-by-element addition,

[0100] And the feature f rgb and f tir Multiply by W and (1-W) respectively to obtain the weighted feature W h and W l :

[0101] W h =W⊙f rgb (10),

[0102] W l =(1-W)⊙f tir (12),

[0103] in,

[0104] Then, splice W h and W l , obtain the fusion feature f through a standard convolution layer fused :

[0105] f fused =BN(ReLU(Conv 1×1 (Concat(W h ,W h ))))f tir (13),

[0106] in,

[0107] Finally, in order to further solve the problem of inconsistency in scale between different modalities, the fusion feature f fused Use convolution operations with different dilation rates. Set two dilation rates r1=1 and r2=2, and perform 3×3 dilation convolution respectively. The obtained features and After splicing, channel compression is performed through 1×1 convolution:

[0108]

[0109] f aligned =Conv 1×1 (f d ) (17),

[0110] in, It represents the feature obtained by convolution with a dilation rate of r.

[0111] The obtained f aligned It fuses contextual information at two different scales, retaining local details (cavity rate 1) while capturing information from a larger receptive field (cavity rate 2), providing a multi-scale, aligned feature representation for subsequent decoding tasks.

[0112] (4) Dual-branch context aggregation module

[0113] In the task of river pollution segmentation, there are significant differences in spatial scales between RGB image and TIR image data. The RGB modality can capture the color and texture of pollutants (such as oil film reflections and algae spots) at high-resolution local scales, and the TIR modality is sensitive to temperature anomalies (such as industrial thermal drainage and oil film thermal radiation) at medium and low-resolution global scales. However, existing methods often directly splice multimodal features, resulting in the complementarity between local details and global semantics not being fully utilized, especially in complex scenes (such as water surface reflection interference and real pollution coupling), the segmentation performance is limited. Therefore, the embodiment of the present application proposes a dual-branch context aggregation module that integrates local and global context information, which can effectively improve the detection robustness in complex environments (night, haze, strong light), avoid false detection or missed detection due to environmental interference, and has a lower computational cost. The module architecture is as follows: Figure 7 shown.

[0114] Figure 7 In

[15] , we first decouple the functions of obtaining local information and exploring global context into two paths; for the ternary feature map f with 256 channels, aligned , all channels are evenly divided into two groups along the channel dimension, namely In the global semantic branch, the input feature fglobal The spatial filtering is performed through 3×3 depth-separable convolution, followed by 1×1 point-by-point convolution and a second 3×3 depth-separable convolution, and finally the residual connection output feature f s It can be expressed as:

[0115]

[0116] in,

[0117] In the local detail branch, two 1×1 convolutions are used as keys and values ​​to increase the network capacity, and double normalization (Softmax normalization and L1 norm normalization) is applied between them to reduce the sensitivity of the attention map to the sample scale. The output feature f is then obtained through a residual connection. l :

[0118]

[0119] in, represents double normalization by softmax activation function and L1 norm normalization,

[0120] The outputs of the two branches are then concatenated, and after 1×1 convolution and batch normalization, global average pooling, 1×1 convolution and Sigmoid activation function are performed to obtain the weight W. After weighting, the 256 channels are divided into 4 groups (64 channels each), and the number of channels is maintained after rearrangement to obtain the context enhancement feature f. context , the process can be expressed as:

[0121]

[0122] f context =Shuffle(f sl ⊙Broadcast(W)) (22),

[0123] Among them, GAP represents global average pooling, Shuffle represents channel shuffling, and ⊙ represents dot product.

[0124] W∈R 1×1×256 ,

[0125] It is understandable that in actual application scenarios, the feature fusion module strengthens RGB details in the local high-resolution layer (3×3 convolution) and highlights the TIR temperature diffusion pattern in the global semantic layer (5×5 void convolution), which can reduce the oil film boundary segmentation error by more than 18%, effectively suppress the interference of water surface reflection and thermal noise, and improve the coherence and segmentation accuracy of the polluted area boundary. Combined with the dual-branch context aggregation module, it synchronously decouples the global pollution diffusion trend (thermal drainage path) and local high-frequency edge information (scattered garbage), which can achieve precise positioning of small targets and pollution diffusion trend modeling, assist in rapid tracing and targeted governance, shorten emergency response time by more than 30%, and significantly improve the efficiency of pollution incident response.

[0126] (5) Multi-scale decoding module

[0127] The decoding module is to enhance the context feature F context Input to the multi-scale spatial state decoder to obtain multi-scale information and global context decoding features F SSM , then feature F SSM The original context-enhanced feature f output by the dual-branch context aggregation module context and scale characteristics Small and medium scale features and The global scale features are spliced, and then compressed through a multi-layer perceptron and convolution layer, and finally restored to the original input size through upsampling operation to generate a pixel-level prediction map Modules such as Figure 8 and 9 shown.

[0128] In order to better capture semantic information at different scales, we first use two convolution operations with different strides to obtain low-resolution multi-scale features. Among them, the convolution with a stride of 2 is used to generate small and medium-scale features. A 5×5 dilated convolution with a stride of 1 and a dilation rate of 4 is used to generate global scale features. They are represented as follows:

[0129]

[0130] in,

[0131] These multi-scale features can capture coarse and fine semantic information respectively, providing multi-level expressions for subsequent global modeling.

[0132] Since features of different scales have different spatial sizes, they need to be unified to the same spatial resolution before further processing. To this end, a feature permutation operation is used to resize the original context features f context And after downsampling Lossless downsampling is performed to make their spatial resolution consistent. Since it is the lowest resolution, it can be used directly. Then apply 1×1 convolution to each branch, with a unified channel number of 128:

[0133] f′=Conv 1×1 (Unshuffle(f context )) (25),

[0134]

[0135] f′=Conv 1×1 (Unshuffle(f context )) (28),

[0136] Among them, Unshuffle means feature rearrangement.

[0137] Then, the features of the three scales are concatenated in the channel dimension to obtain a unified multi-scale feature expression:

[0138]

[0139] in, This operation allows information from different scales to be retained simultaneously and provides a unified input for subsequent state-space modeling.

[0140] Then, the concatenated features f concat The image is then fed to the 2D state-space modeling module SS2D, which uses the state-space model to scan the entire image. This process efficiently captures long-range dependencies between features and the global semantic information of the image with linear time complexity. The output features after state-space modeling are:

[0141] f SSM =SS2D(f concat ) (30).

[0142] The multi-scale decoding module effectively integrates features at different scales through dynamic weight allocation and structured state updates, complements global background information, and enhances the ability to capture the diffusion pattern of polluted areas.

[0143] Finally, the state space output f SSM The original context-enhanced feature f output by the dual-branch context aggregation module context and scale characteristics Fusion is performed to take advantage of all information:

[0144]

[0145] in,

[0146] After splicing in the channel dimension, feature compression is performed through a layer of multi-layer perceptron:

[0147] f mlp =MLP(f fusion ) (32),

[0148] in,

[0149] Finally, the original input size is restored through step-by-step upsampling operations to generate a pixel-level prediction map.

[0150] f up1 =Upsample 2× (f mlp ) (33),

[0151] f up2 =Upsample 2× (f up1 ) (34),

[0152]

[0153] Among them, MLP represents multi-layer perceptron, Upsample is the upsampling operation, H×W is consistent with the original image size, and K is the number of pollution categories (for example, K=2 represents "pollution / non-pollution", K=3 represents "no pollution / oil spill / algae" and other multi-classification scenarios).

[0154] Channel vector for each pixel position (i,j) Represents the probability distribution of the pixel belonging to each category. The final category can be obtained by softmax and argmax operations:

[0155]

[0156] The multi-scale decoding module innovatively introduces a two-dimensional state-space model. By systematically scanning the entire image with linear time complexity, it overcomes the limitations of traditional convolution operations in terms of receptive field, accurately capturing long-range dependencies, such as key information such as the spread of river pollution. By effectively fusing features at different scales, the module significantly expands the global receptive field while preserving local details in river images. This feature not only enhances the consistency of polluted area segmentation, but also improves the overall segmentation accuracy of river pollution detection, providing more reliable support for pollution monitoring and analysis.

[0157] In a feasible embodiment, after obtaining the standard data set and building the initial river pollution detection network, the initial river pollution detection network can be further trained based on the standard data set to obtain a trained river pollution detection network. The training process is as follows: Figure 10 As shown, it may include but not be limited to steps 1010 to 1050.

[0158] Step 1010: Divide the labeled dataset into a training set, a validation set, and a test set;

[0159] Step 1020: Using the training set, supervised training is performed on the initial river pollution detection network. According to a preset learning rate rule, a loss value is calculated based on the training results of the supervised training and the overall loss function. The parameters of the initial river pollution detection network are updated based on the loss value to obtain a trained detection network. The overall loss function is a weighted sum of the first cross entropy loss function, the second cross entropy loss function, and the modal consistency loss function.

[0160] Step 1030: Validate the trained detection network based on the validation set to obtain a validation result, and adjust the parameters of the detection network based on the validation result to obtain an optimized detection network.

[0161] Step 1040: Evaluate the optimized detection network using the test set to obtain a performance evaluation result;

[0162] Step 1050: When the performance evaluation result shows that the standard is met, the optimized detection network is determined as the river pollution detection network.

[0163] In a feasible embodiment, the label data set of step 1010 is a data set that has been preprocessed by steps 310 to 340. Before using the data set to conduct end-to-end supervised training on the initial river pollution detection network, the data set can be divided into a training set, a validation set, and a test set at a ratio of 70%, 15%, and 15%. The training set is used to train the initial river pollution detection network so that the network can learn the features and patterns in the data; the validation set is used to monitor the performance of the model during the training process to prevent overfitting and adjust the network parameters; the test set is used to finally evaluate the generalization ability and performance of the model after the model training and adjustment are completed, and the test set does not participate in the training during the model training process. Among them, each training sample consists of an RGB sub-image, a TIR sub-image, and a corresponding pixel-level binary mask of the pollution area. The three are paired one by one to ensure the spatial consistency and semantic relevance of the multimodal data and the label. The symbolic representation of the training sample is:

[0164] I={I rgb ,I tir ,Y} (37),

[0165] Where Y∈{0,1,...,K-1} H×W is the label mask, and K is the number of pollution categories.

[0166] In a feasible embodiment, in step 1020, the divided training set data is used to supervise the initial river pollution detection network. Supervised training means that the input of the network is the image data of the training set (including RGB sub-images and TIR sub-images), and the output is compared with the real pixel-level pollution area binary mask. The preset learning rate rule, such as the initial learning is first warmed up, and then a polynomial decay strategy is adopted to control the step size of each parameter update to balance the convergence speed and model stability. The loss value is calculated based on the training results (network prediction output) and the overall loss function (the first cross entropy loss function, the second cross entropy loss function and the modal consistency loss function are added according to certain weights). This loss value measures the difference between the network prediction and the true label. Then, based on this loss value, an optimizer (such as Adam or SGD) is used to update the parameters of the network, with the aim of gradually reducing the loss value. After multiple iterations, the trained detection network is obtained.

[0167] Specifically, in order to guide the model to continuously iteratively update parameters, a multi-loss function joint training strategy can be adopted. Among them, the core loss function is the cross entropy loss (i.e., the first cross entropy loss function), which is calculated based on formula (38).

[0168]

[0169] Furthermore, to improve the model's ability to capture boundary details and small objects, the Dice Loss can be introduced as the second cross-entropy loss function. By optimizing the overlap between the predicted area and the true area, the model's recognition accuracy for contaminated areas is enhanced. The second cross-entropy loss function is calculated according to Equation (39).

[0170]

[0171] Here, ε is a small constant used to avoid the denominator being zero.

[0172] Furthermore, in order to ensure the semantic consistency of RGB and TIR modalities in the fusion feature space and effectively eliminate the conflict between modalities, the modality consistency loss is introduced, and the specific form of its loss function is shown in formula (40).

[0173]

[0174] Finally, the overall loss function can be expressed as the weighted sum of the three, as shown in formula (41):

[0175] I=λ1L CE+λ2L Dice +λ3L Consist (41),

[0176] Among them, λ1 = 1.0, λ2 = 0.5, and λ3 = 0.2, which are determined by tuning the validation set.

[0177] During network training, either Adam or SGD can be used as the optimizer. Initial learning begins with a warm-up phase, followed by dynamic adjustments using a polynomial decay strategy to achieve a balance between convergence speed and model stability. The specific learning rate update rule follows the calculation method shown in Equation (42).

[0178]

[0179] Among them, lr imit represents the initial learning rate, iter is the current number of iterations, max_iter is the total number of iterations, and power is 0.9.

[0180] In one feasible embodiment, to further enhance network robustness, a data augmentation strategy can be implemented during the training phase. This effectively expands the diversity of training samples through diverse operations such as random cropping, horizontal flipping, angular rotation, and brightness perturbation. Furthermore, a batch training model is employed, allowing for flexible batch size adjustments based on hardware computing resources. For scenarios where computing power is limited on drone edge devices, model training can be prioritized in the cloud before deploying a lightweight network, ensuring efficient model inference in real-world applications.

[0181] In one feasible embodiment, in step 1030, the trained detection network is applied to a validation set, and the image data from the validation set is input to obtain network prediction results. These prediction results are compared with the true labels in the validation set to obtain verification results, such as evaluation metrics such as accuracy, recall, and F1 value. Based on the verification results, it is determined whether the model suffers from overfitting or underfitting. The parameters of the detection network are then appropriately adjusted, such as adjusting hyperparameters such as network weights and learning rate, to optimize model performance and obtain an optimized detection network.

[0182] In one feasible embodiment, in step 1040, after completing training and validation adjustments, the optimized detection network can be evaluated using a test set of data that was not previously used in training and validation. The network is fed with image data from the test set, and the network's predictions are compared with the true labels in the test set. A series of performance metrics, such as precision, recall, and mean squared error, are calculated. These metrics collectively constitute the performance evaluation results, reflecting the model's generalization ability and actual performance on unknown data.

[0183] In one feasible embodiment, in step 1050, certain performance criteria may be set, such as accuracy reaching a certain threshold, loss value being below a certain value, etc. When the performance evaluation results meet these pre-set criteria, the optimized detection network is deemed to perform well in the river pollution detection task and meet the needs of practical applications. Therefore, it is determined as the final river pollution detection network and can be used in actual river pollution detection work. If the performance evaluation results do not meet the criteria, it may be necessary to return to the previous steps, such as re-adjusting parameters, adding data augmentation, etc., to continue optimizing the model.

[0184] The trained river pollution detection network comprises an image pre-segmentation module, a multimodal feature encoding module, a feature fusion module, a dual-branch context aggregation module, and a multi-scale decoding module. After training and optimization, each module can accurately perform its corresponding processing tasks, collaboratively achieving efficient and accurate river pollution detection.

[0185] In one possible embodiment, Figure 11 Taking the illustrated process as an example, in step 130, when performing pollution detection on a river area, the current multimodal data including optical images and thermal infrared images can be input into the detection network. Specific execution steps include, but are not limited to, steps 1110 to 1190.

[0186] Step 1110: Input the current optical image and the current thermal infrared image into the river pollution detection network, and use the image pre-segmentation module to segment the current optical image to obtain a river mask;

[0187] Step 1120: Using the river channel mask as a constraint condition, use a multimodal feature encoding module to perform feature extraction on the current optical image and the current thermal infrared image to obtain a first feature and a second feature;

[0188] Step 1130: Using a feature fusion module to perform feature fusion processing on the first feature and the second feature to obtain a multimodal feature;

[0189] Step 1140: Using a dual-branch context aggregation module, perform local and global feature extraction on the multimodal features to obtain local feature information and global feature information;

[0190] Step 1150: concatenate the local feature information and the global feature information to obtain context-enhanced features;

[0191] Step 1160: Decode the context enhancement feature using a multi-scale decoding module to obtain multi-scale feature information and global context decoding features;

[0192] Step 1170: performing feature compression processing on the global context decoding features, the context enhancement features, and the multi-scale feature information to obtain prediction features;

[0193] Step 1180: Perform multiple upsampling operations on the predicted features to obtain a pollution prediction map;

[0194] Step 1190: Obtain river pollution detection results based on the pollution prediction map.

[0195] In one feasible embodiment, the current optical image and the current thermal infrared image are input into the river pollution detection network. The image pre-segmentation module then segments the current optical image. The purpose of this step is to identify the location and extent of the river in the optical image, ultimately obtaining a river mask. This mask can be used as a constraint for subsequent processing to define the river area. The specific process of step 1110 can be referred to as shown in equation (1) and will not be repeated here.

[0196] In a feasible embodiment, in step 1120, the river channel mask obtained in step 1110 is used as a constraint, and the multimodal feature encoding module is used to extract features from the current optical image and the current thermal infrared image respectively. The specific operation can be referred to formula (2). The current optical image and the current thermal infrared image corresponding to the area where the river channel mask is 1 are cropped and padded to a size of H×W×3, and are respectively subjected to a ResNet-18 encoder and a 1×1 convolution to obtain the first feature f rgb and the second feature f tir .

[0197] In a feasible embodiment, in step 1130, the feature fusion module combines the first feature f rgb and the second feature f tir The purpose of fusion processing is to integrate the feature information of different modal images and give full play to the advantages of multimodal data, so as to obtain richer and more representative multimodal features f fused Specifically, the feature fusion module includes a multi-scale feature enhancement submodule and a spatial attention fusion submodule. In step 1130, the feature fusion module is used to enhance the first feature f rgb and the second feature f tir Perform feature fusion processing to obtain multimodal features f fused The execution process is as follows Figure 12 As shown, the process may include but is not limited to steps 1210 to 1230.

[0198] Step 1210: using a multi-scale feature enhancement submodule to perform feature enhancement processing on the first feature and the second feature respectively, to obtain a first enhanced feature and a second enhanced feature;

[0199] Step 1220: Using the spatial attention fusion submodule, perform feature extraction processing on the first enhanced feature to obtain a first attention feature, and perform feature extraction processing on the second enhanced feature to obtain a second attention feature;

[0200] Step 1230: Fuse the first attention feature and the second attention feature to obtain a multimodal feature.

[0201] In a feasible embodiment, in step 1210, in the feature fusion module, the multi-scale feature enhancement submodule performs feature enhancement processing on the first feature and the second feature respectively. This process uses a specific algorithm to enhance the expressive power of the feature, highlight the key features, and suppress irrelevant information, thereby obtaining the first enhanced feature and the second enhanced feature. For specific operations, please refer to equations (3) to (5). First, ReLU activation is performed on the first feature and the second feature respectively, and then a 3×3 or 5×5 depth convolution operation is performed to extract richer feature information. Then, the number of channels is compressed by 1×1 convolution to reduce the feature dimension and reduce the amount of calculation. Afterwards, binary masks are generated based on the first feature and the second feature respectively, and convolution operations are performed only on the areas marked as 1 in the mask, focusing on the key feature areas. Finally, batch normalization is performed on the features obtained by the operation to make the feature distribution more stable, and finally the first enhanced feature f1 and the second enhanced feature f2 are obtained.

[0202] In a feasible embodiment, in step 1220, in the spatial attention fusion submodule, feature extraction is performed for the first enhanced feature f1 and the second enhanced feature f2 respectively. Based on the spatial attention mechanism, this module can focus on the key areas in the image, extract core features, and then obtain the first attention feature and the second attention feature. The specific operation can refer to equations (6) to (8). After the first enhanced feature and the second enhanced feature are respectively input into the spatial attention fusion submodule, the maximum pooling and average pooling operations are first performed in parallel, and then the two pooling results are spliced ​​along the channel dimension to integrate feature information from different perspectives. Then, the channel dimension is compressed by 1×1 convolution, and a weight map is generated using the Sigmoid activation function to quantify the importance of each spatial position. Finally, the weight map is expanded to the same spatial size as the input features (first feature, second feature), and the key area feature response is enhanced by element-by-element multiplication, thereby obtaining the first attention feature and the second attention feature.

[0203] In a feasible embodiment, in step 1230, in order to integrate the feature information of different modal images in important areas, the first attention feature and the second attention feature need to be fused to obtain a multimodal feature. The specific operation can refer to equations (9) to (13), and the steps are as follows: First, the first attention feature and the second attention feature output from the spatial attention fusion submodule are added element by element. Then, a weighted probability matrix W is generated by 1×1 convolution and Sigmoid activation function. Then, the first feature and the second feature are multiplied by W and (1-W) respectively to obtain the first weighted feature W hand the second weighted feature W l Finally, the first weighted feature and the second weighted feature are concatenated and processed through a standard convolution layer. The final fusion feature is the multimodal feature f fused .

[0204] In a feasible embodiment, after obtaining the multimodal feature f fused After that, it also includes: performing convolution operation on the multimodal features according to the preset void ratio to obtain convolution features; performing splicing and channel compression processing on the convolution features to obtain the optimized multimodal features f aligned Specifically, after obtaining the multimodal features, the multimodal features are convolved according to the preset dilation rate. Dilation convolution can expand the receptive field of the convolution kernel without increasing the number of parameters, extract a wider range of feature information, and obtain convolution features. The convolution features are then spliced ​​and channel compressed to further integrate feature information and reduce feature dimensions, obtaining the optimized multimodal features f aligned , the multimodal feature f aligned is a ternary feature map with 256 channels. The specific operations can be referred to as shown in equations (14) to (17), which will not be repeated here.

[0205] In one feasible embodiment, in step 1140, the dual-branch context aggregation module extracts local and global features from the multimodal features to generate local feature information and global feature information, respectively. The local feature information focuses on the subtle structure of the image, accurately capturing the local details of river pollution; the global feature information takes a holistic perspective, encompassing the image's global semantics and contextual associations. The specific calculation process can be found in equations (18) to (19) and will not be elaborated here.

[0206] In a feasible embodiment, in order to further enhance the expressiveness of the features, the local feature information obtained in step 1140 is concatenated with the global feature information. This operation can integrate feature information of different scales to form a context-enhanced feature f context , providing a more representative feature representation for subsequent image analysis. The specific calculation process of step 1150 can be referred to equations (20) to (22), which will not be elaborated here.

[0207] In a feasible embodiment, in step 1160, the multi-scale decoding module can efficiently integrate features of different scales and simultaneously complete global background information with the help of dynamic weight allocation and structured state update mechanism, thereby improving the ability to capture the diffusion pattern of the polluted area. The specific operation process can refer to equations (23) to (30), and the steps are as follows: First, perform convolution operations of different step lengths on the context enhancement features to obtain multi-scale feature information. This information contains the first scale features with different spatial resolutions. and second-scale features Then, the context enhancement features f are context and the first scale feature Perform downsampling to obtain the downsampled features f′ and features Next, the second scale feature is transformed into The context-enhanced feature f′ after downsampling and the first scale feature after downsampling Perform splicing to generate splicing features f concat Then, the concatenated features f concat Input the two-dimensional state space modeling module SS2D, and scan the entire image with the help of the state space model. This process can effectively capture long-range dependencies and global semantic information with linear time complexity. The feature f output after state space modeling is SSM , which is the global context decoding feature.

[0208] In one feasible embodiment, in step 1170, concatenation can integrate different types of feature information, while feature compression is intended to reduce the feature dimension, improve computational efficiency, and ultimately obtain the predicted features. The specific computational process of this step can be referred to as shown in equations (31) and (32), and will not be elaborated here.

[0209] In one feasible embodiment, in step 1180, multiple upsampling operations are performed on the predicted features, and the feature map size is gradually enlarged until it fully matches the original image size, ultimately generating a pollution prediction map. This prediction map presents the river boundary mask and the distribution of polluted areas with pixel-level accuracy, visually marking the specific location, scope, and degree of pollution in the river. The specific calculation logic can be referred to in equations (33) to (36), and will not be repeated here.

[0210] It is understandable that the RGB (optical) and TIR (thermal infrared) multimodal images collected in real time by drones during river inspections may have various problems (such as inconsistent spatial positions, different lighting conditions, inaccurate thermal infrared image radiation, etc.). These problems will interfere with the accuracy of pollution detection. Therefore, the images need to be preprocessed to eliminate environmental interference factors and make the images more suitable for input into the river pollution detection network for processing.

[0211] In one feasible embodiment, before inputting the current multimodal image of the river channel area into the river pollution detection network for pollution detection, the current multimodal image can be preprocessed. Specifically, the preprocessing includes: performing image registration on the current optical image and the current thermal infrared image based on feature point matching and affine transformation algorithms to obtain the registered current optical image and the current thermal infrared image; performing illumination normalization on the registered current optical image; and performing radiation correction on the registered current thermal infrared image to obtain the standard detection optical image and the standard detection thermal infrared image. Specifically, before inputting the current multimodal image of the river channel area into the detection network, image registration must first be performed. The method used can be based on feature point matching and affine transformation algorithms. Feature point matching involves finding identical feature points in two images (the current optical image and the current thermal infrared image). These feature points represent significant locations or structures in the images. By finding a sufficient number of feature points, the relative position and posture relationship between the two images can be determined. The affine transformation algorithm transforms one image (such as translation, rotation, and scaling) based on the relationships between the identified feature points to spatially align the two images. The spatial alignment error is required to be ≤ 2 pixels. This means that after registration, the positional deviation between corresponding points in the two images must be very small, ensuring that the positions of the same objects in the two images correspond as accurately as possible, providing a good foundation for subsequent feature fusion and analysis. This process yields the registered current optical image and thermal infrared image. The registered current optical image undergoes illumination normalization. Because optical images are significantly affected by ambient lighting conditions during capture, varying light intensities and angles can lead to inconsistent image brightness and contrast, affecting the representation of object features. Illumination normalization uses a specific algorithm to adjust image brightness and contrast, ensuring similar visual effects under varying lighting conditions. This emphasizes important information and facilitates subsequent analysis. Radiometric correction is performed on the registered current thermal infrared image. The radiometric characteristics of thermal infrared images can be affected by various factors, such as the device itself and ambient temperature, resulting in inaccurate radiometric values. Radiometric correction adjusts the radiometric values ​​of thermal infrared images to more accurately reflect the actual temperature of objects, improving their quality and reliability. This process produces standard optical and thermal infrared images, eliminating most environmental interference factors and serving as standard input for the river pollution detection network.

[0212] It should be noted that the pollution prediction map is a pixel-level result map with the same size as the original multimodal image (that is, the combination of the current optical image and the thermal infrared image). Each pixel in the map corresponds to a channel vector, which is used to accurately characterize the probability distribution of the pixel belonging to different pollution categories. For example, in a two-classification scenario, it can clearly distinguish between "polluted" and "non-polluted" areas; in a multi-classification scenario, it can accurately identify various types of pollution such as "non-pollution", "oil slick", and "algae". Based on this prediction map, through preset analysis rules and algorithms, it is possible to accurately determine whether there is pollution in the river, as well as the specific type, scope and severity of the pollution, and thus obtain comprehensive and accurate river pollution detection results.

[0213] In a feasible embodiment, after obtaining the river pollution detection results, further screening and analysis can be carried out. If the pollution data detected in the river area exceeds the preset threshold, an early warning can be issued immediately; at the same time, a detailed trend analysis report can be generated based on the location of the pollution source shown in the detection results. Specifically, after obtaining the pollution prediction map, key information such as the area, location coordinates and diffusion morphology of the polluted area can be extracted and compared with the preset threshold (such as the pollution area ratio > 5%), and a comprehensive analysis can be performed in combination with the historical pollution database. Once the data exceeds the limit, a real-time alarm will be triggered, and the pollution heat map will be superimposed on the original image for visual display to intuitively present the pollution situation. For pollution signals that continue to appear in the same area in consecutive frames, the location of the potential pollution source can be inferred through data analysis, and a trend analysis report can be generated to provide a decision-making basis for pollution control. For areas with low confidence in the detection results (such as misjudgment areas caused by factors such as water surface reflection and thermal noise interference), they can be marked as samples to be reviewed, and manually annotated and corrected through the interactive interface, and the correction results are fed back to the model for iterative training, thereby continuously improving the accuracy of the detection model. Ultimately, river pollution detection results are linked to geographic information and pushed to management personnel in real time via the monitoring platform. The entire process ensures single-frame image processing latency of ≤200ms, ensuring data processing accuracy while fully meeting the real-time requirements of river inspections.

[0214] See also Figure 13 , Figure 13This is an overall flow chart of river pollution detection provided by an embodiment of the present application. First, the multimodal remote sensing images of the river area are collected using the equipment carried by the drone, covering both optical (RGB) and thermal infrared (TIR) ​​modes. After the acquisition is completed, the original multimodal images are preprocessed and registered, and the optical image and the thermal infrared image are spatially aligned through feature point matching and affine transformation algorithms (the error is controlled within 2 pixels). At the same time, illumination normalization is performed on the optical image and radiation correction is performed on the thermal infrared image to eliminate the interference of environmental factors on the image quality and form a standardized detection image. Subsequently, the processed multimodal image is input into the pre-trained river pollution detection network. The network integrates core modules such as image pre-segmentation, multimodal feature encoding, feature fusion, dual-branch context aggregation, and multi-scale decoding. These modules work together: the image pre-segmentation module first processes the optical image to generate a river channel mask; the multimodal feature encoding module extracts features from each modality based on the mask; the feature fusion module fuses the optical and thermal infrared image features through multi-scale feature enhancement and spatial attention; the dual-branch context aggregation module extracts local and global features and stitches them together for enhancement; and the multi-scale decoding module uses a two-dimensional state-space model to scan the entire image in linear time, capturing long-range dependencies such as pollution diffusion trends. It fuses multi-scale features to expand the global receptive field and ultimately outputs a river channel segmentation mask and a segmentation map of the polluted area (i.e., a predicted map). After obtaining the segmentation results, the polluted area is further analyzed in depth. Key information such as the area, location coordinates, and diffusion pattern of the polluted area is extracted and compared with a historical pollution database. The pollution level is then determined based on a preset threshold (e.g., if the polluted area exceeds 5%). If pollution data exceeds a threshold, a real-time alert is triggered, generating a pollution heat map and overlaying it onto the original image for visual display. If the threshold is not exceeded, the relevant area is marked as a sample for review and transferred to a human for annotation and correction via an interactive interface. After manual review, the final results are linked to geographic information and pushed to management personnel in real time via the monitoring platform. This ensures that the processing delay of a single frame of image does not exceed 200ms, meeting the real-time requirements of river inspections and providing strong support for pollution source tracing, emergency decision-making, and subsequent iterative model optimization.

[0215] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for detecting river pollution, characterized in that: include: Obtain the label dataset of the target river area; Building an initial river pollution detection network, and training the initial river pollution detection network based on the labeled data set to obtain a trained river pollution detection network; A current multimodal image of a river area is obtained, and the current multimodal image is input into the river pollution detection network for pollution detection to obtain a river pollution detection result.

2. The river pollution detection method according to claim 1, characterized in that: The label dataset of the target river area is obtained according to the following steps: Acquire training multimodal images, the training multimodal images including optical images and thermal infrared images, the training multimodal images being collected by a drone while patrolling along a preset route in the target river area; Annotating the optical image pixel by pixel according to a preset category label to obtain a labeled image; performing an affine transformation operation on the thermal infrared image according to the label image to obtain a corrected image; The labeled image and the corrected image are screened and verified to obtain a labeled data set.

3. The river pollution detection method according to claim 2, characterized in that: The label data set includes the label image and the corrected image of the same scene of the target river area; After obtaining the label dataset of the target river area, the method further includes: Performing pixel-level spatial registration processing on the label image and the corrected image respectively to obtain a registered label image and a registered corrected image; Performing brightness normalization and anti-interference processing on the registration label image to obtain the optimized registration label image; Performing temperature radiation correction and denoising processing on the registered and corrected image to obtain an optimized registered and corrected image; The optimized registration label image and the registration correction image are subjected to size standardization and key area extraction processing to obtain a standard data set.

4. The river pollution detection method according to claim 1, characterized in that: The training of the initial river pollution detection network based on the label data set to obtain a trained river pollution detection network includes: Dividing the labeled dataset into a training set, a validation set, and a test set; Using the training set to perform supervised training on the initial river pollution detection network, according to a preset learning rate rule, calculating a loss value based on the training results of the supervised training and an overall loss function, and updating the parameters of the initial river pollution detection network according to the loss value to obtain a trained detection network, wherein the overall loss function is a weighted sum of a first cross entropy loss function, a second cross entropy loss function, and a modal consistency loss function; Verifying the trained detection network according to the verification set to obtain a verification result, and adjusting the parameters of the detection network according to the verification result to obtain the optimized detection network; Using the test set to evaluate the optimized detection network to obtain a performance evaluation result; When the performance evaluation result shows that the standard is met, the optimized detection network is determined as the river pollution detection network.

5. The river pollution detection method according to claim 1, characterized in that: The river pollution detection network includes an image pre-segmentation module, a multimodal feature encoding module, a feature fusion module, a dual-branch context aggregation module, and a multi-scale decoding module; the current multimodal image includes a current optical image and a current thermal infrared image of the river area; Inputting the current multimodal image into the river pollution detection network to perform pollution detection and obtain a river pollution detection result includes: Inputting the current optical image and the current thermal infrared image into the river pollution detection network, and using the image pre-segmentation module to segment the current optical image to obtain a river mask; Using the river channel mask as a constraint condition, using the multimodal feature encoding module to perform feature extraction on the current optical image and the current thermal infrared image respectively to obtain a first feature and a second feature; Using the feature fusion module to perform feature fusion processing on the first feature and the second feature to obtain a multimodal feature; Using the dual-branch context aggregation module to perform local and global feature extraction on the multimodal features, respectively, to obtain local feature information and global feature information; Splicing the local feature information and the global feature information to obtain context-enhanced features; Decoding the context enhancement feature using the multi-scale decoding module to obtain multi-scale feature information and global context decoding features; Performing feature compression processing on the global context decoding feature, the context enhancement feature, and the multi-scale feature information to obtain a prediction feature; Performing multiple upsampling operations on the prediction features to obtain a pollution prediction map; According to the pollution prediction map, the river pollution detection result is obtained.

6. The river pollution detection method according to claim 5, characterized in that: Before inputting the current multimodal image of the river area into the river pollution detection network for pollution detection, the method includes: performing image registration processing on the current optical image and the current thermal infrared image based on feature point matching and affine transformation algorithms to obtain the registered current optical image and the current thermal infrared image; Performing illumination normalization processing on the registered current optical image, and performing radiation correction on the registered current thermal infrared image to obtain a standard detection optical image and a standard detection thermal infrared image.

7. The river pollution detection method according to claim 5, characterized in that: The feature fusion module includes a multi-scale feature enhancement submodule and a spatial attention fusion submodule; The utilizing the feature fusion module to perform feature fusion processing on the first feature and the second feature to obtain a multimodal feature includes: Using the multi-scale feature enhancement submodule to perform feature enhancement processing on the first feature and the second feature respectively, to obtain a first enhanced feature and a second enhanced feature; Using the spatial attention fusion submodule to perform feature extraction processing on the first enhanced feature to obtain a first attention feature, and performing feature extraction processing on the second enhanced feature to obtain a second attention feature; The first attention feature and the second attention feature are fused to obtain a multimodal feature.

8. The river pollution detection method according to claim 5, characterized in that: After obtaining the multimodal features, the method further includes: Performing a convolution operation on the multimodal feature according to a preset void ratio to obtain a convolution feature; The convolution features are spliced ​​and channel compressed to obtain the optimized multimodal features.

9. The river pollution detection method according to claim 5, characterized in that: The using the multi-scale decoding module to decode the context enhancement feature to obtain multi-scale feature information and global context decoding features includes: Performing convolution operations of different step lengths on the context enhancement feature to obtain multi-scale feature information, where the multi-scale feature information includes a first scale feature and a second scale feature, where the first scale feature and the second scale feature have different spatial resolutions; performing downsampling operations on the context enhancement feature and the first scale feature respectively to obtain the downsampled context enhancement feature and the first scale feature; splicing the second-scale feature, the downsampled context-enhanced feature, and the first-scale feature in the channel dimension to obtain a spliced ​​feature; A global context decoding feature is obtained according to the splicing feature.

10. The river pollution detection method according to claim 1, characterized in that: After obtaining the river pollution detection result, the method further includes: An early warning is issued for river areas where pollution data in the river pollution detection results exceeds a preset threshold, and a trend analysis report is generated based on the location of the pollution source displayed by the river pollution detection results.

Images