Cross-modal information fusion duckweed and algae bloom detection method and related device

Abstract

The invention provides a cross-modal information fusion duckweed and algal bloom detection method and a related device, and belongs to the technical field of duckweed and algal bloom detection. The method comprises the following steps: acquiring visible light wave band image data and near-infrared wave band image data in a duckweed area and an algae bloom area; inputting visible light wave band image data and near infrared wave band image data in the duckweed area and the algae bloom area into the trained DC-TransUnet network to obtain a duckweed water body segmentation result; based on the duckweed water body segmentation result, determining the coverage proportion of duckweed and algae bloom in the water body; and obtaining detection results of the duckweed and the algal bloom based on the coverage ratio of the duckweed and the algal bloom in the water body. The problem that the detection precision of the duckweed and the algae bloom is not high is solved.

Description

Technical Field

[0001] This invention belongs to the field of duckweed and algal bloom detection technology, specifically relating to a cross-modal information fusion method and related apparatus for detecting duckweed and algal blooms. Background Technology

[0002] The accumulation of nitrogen and phosphorus emissions from factory production and daily life has led to frequent algal blooms and duckweed outbreaks in inland water bodies in my country. Typically, algal blooms and duckweed completely covering the water surface significantly reduce dissolved oxygen levels, causing aquatic organism mortality. Furthermore, the release of algal toxins further exacerbates water quality deterioration. This poses a serious threat to drinking water supply, aquaculture, and ecosystem balance; therefore, the detection of duckweed and algal blooms is of paramount importance for effective and rational management.

[0003] Duckweed and algal blooms mostly occur in small, scattered ponds or narrow ditches. Traditional manual survey methods are limited by high labor intensity and low efficiency, making them unsuitable for large-scale, rapid detection. Remote sensing-based detection methods offer advantages such as wide coverage and real-time performance, and are currently the main technical means for fine-grained detection of algal blooms and duckweed. These methods can be categorized into three types: thresholding methods based on spectral indices, classification methods based on machine learning, and feature extraction methods based on deep learning.

[0004] The spectral index thresholding method utilizes the differences in spectral characteristics between aquatic vegetation and algal blooms to construct remote sensing indices that express these differences, and then extracts target areas by setting thresholds. Its advantages lie in its simple computation and convenient implementation, making it suitable for rapid detection over large areas. However, this method often struggles to determine a uniform threshold across different regions or imaging conditions, limiting its transferability and adaptability, and making it prone to misjudgments in complex aquatic environments. Machine learning methods typically construct classifiers, such as random forests and support vector machines, to distinguish between algal blooms and duckweed areas, vegetated areas, and non-vegetated areas in aquatic bodies. These methods possess a certain degree of nonlinear modeling capability, which can improve detection accuracy to some extent. However, they are sensitive to noise, have limited model generalization ability, and their detection accuracy is unstable in remote sensing images covering large-scale complex scenes. Deep learning methods leverage the advanced semantic feature extraction capabilities of neural networks to automatically learn the distinguishable spectral and spatial features of different land cover types in remote sensing images. They exhibit strong noise resistance and can achieve accurate detection of algal blooms and duckweed in aquatic bodies, and have been widely applied to the detection of aquatic vegetation and algae in recent years. However, most existing methods simply stitch visible light remote sensing images and other auxiliary information together in the channel dimension at the network input stage, and use a single-branch encoder structure to extract semantic features from the input data. This network structure is prone to insufficient feature extraction from the input image, and may also introduce noise in the extraction of visible light images and auxiliary information. Furthermore, shallow fusion cannot fully exploit the complementarity between different bands, thus limiting its performance improvement in complex aquatic environments.

[0005] In summary, existing methods suffer from limitations in detection accuracy, insufficient adaptability, or simplistic fusion methods to varying degrees. Therefore, there is an urgent need for a novel detection method that can efficiently extract auxiliary band features and achieve deep interaction with visible light images to improve the detection capability of duckweed and algal blooms in water bodies. Summary of the Invention

[0006] The purpose of this invention is to provide a cross-modal information fusion method and related device for detecting duckweed and algal blooms, in order to solve the problem of low accuracy in the detection of duckweed and algal blooms in the prior art.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a method for detecting duckweed and algal blooms through cross-modal information fusion, comprising the following steps: Acquire visible light and near-infrared image data of duckweed and algal bloom areas; Visible light and near-infrared image data of duckweed and algal bloom areas are input into the trained DC-TransUnet network to obtain duckweed water body segmentation results; The DC-TransUnet network includes a ResNet network, a convolutional linear fusion module, and a cross-modal feature correction module; The ResNet network is used to extract features from visible light band image data to obtain visible light band image data features. The convolutional linear fusion module is used to extract features from near-infrared band image data to obtain near-infrared band image data features. The cross-modal feature correction module is used to perform bi-branch feature AC correction on the features of visible light band image data and near-infrared band image data to obtain the corrected features. Based on the results of duckweed water body segmentation, the coverage ratio of duckweed and algal blooms in the water body was determined. The detection results of duckweed and algal blooms were obtained based on the coverage ratio of duckweed and algal blooms in the water body.

[0008] A further improvement of this invention is that, before inputting the visible light band image data and near-infrared band image data of the duckweed region and algal bloom region into the trained DC-TransUnet network to obtain the duckweed water body segmentation result, the visible light band image data and near-infrared band image data of the duckweed region and algal bloom region are augmented.

[0009] A further improvement of the present invention is that the training method of the DC-TransUnet network includes: A dataset was constructed using visible light and near-infrared imagery data from duckweed and algal bloom areas. Based on the constructed dataset, it is divided into training set, validation set and test set according to the proportion, and the DC-TransUnet network is trained using the training set.

[0010] A further improvement of this invention is that the loss function used when training the DC-TransUnet network is a joint loss function, which is expressed as:

[0011] in, For the joint loss function, Let cross-entropy be the loss function. The Dice loss function, and These are weight parameters; The cross-entropy loss function The expression is:

[0012] The Dice loss function The expression is:

[0013] in, Indicates the number of pixels. Indicates the first The label of each sample Indicates the prediction of the first The probability that a sample belongs to class 1. This indicates the smoothing term.

[0014] A further improvement of this invention is that the convolutional linear fusion module is used to extract features from near-infrared band image data to obtain near-infrared band image data features, specifically including: Based on standard convolution, dilated convolution, and attention mechanisms, multi-scale features of near-infrared image data are captured.

[0015] A further improvement of the present invention is that, in the step of determining the coverage ratio of duckweed and algal blooms in the water body based on the duckweed water body segmentation results, an expansion operation is applied to the duckweed area and the algal bloom area to cancel the gap between the duckweed area and the water body in the duckweed water body segmentation results, and whether they are the same water body is determined based on whether an intersection occurs after expansion.

[0016] A further improvement of this invention is that, after obtaining the detection results of duckweed and algal blooms, the water body is classified and risk warnings are issued.

[0017] Secondly, the present invention provides a cross-modal information fusion system for detecting duckweed and algal blooms, comprising: The image data acquisition module is used to acquire visible light and near-infrared image data in the duckweed and algal bloom areas. The duckweed water body segmentation module is used to input visible light band image data and near-infrared band image data of duckweed area and algal bloom area into the trained DC-TransUnet network to obtain duckweed water body segmentation results; The DC-TransUnet network includes a ResNet network, a convolutional linear fusion module, and a cross-modal feature correction module; The ResNet network is used to extract features from visible light band image data to obtain visible light band image data features. The convolutional linear fusion module is used to extract features from near-infrared band image data to obtain near-infrared band image data features. The cross-modal feature correction module is used to perform bi-branch feature AC correction on the features of visible light band image data and near-infrared band image data to obtain the corrected features. The coverage ratio determination module is used to determine the coverage ratio of duckweed and algal blooms in the water body based on the duckweed water body segmentation results. The duckweed and algal bloom detection module is used to obtain duckweed and algal bloom detection results based on the coverage ratio of duckweed and algal blooms in the water body.

[0018] Thirdly, the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the cross-modal information fusion detection method for duckweed and algal blooms described above.

[0019] Fourthly, the present invention provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the cross-modal information fusion detection method for duckweed and algal blooms described above.

[0020] Compared with the prior art, the present invention has the following beneficial effects: The proposed cross-modal information fusion method for detecting duckweed and algal blooms in this invention, on the one hand, incorporates a convolutional linear fusion module and a cross-modal feature correction module within the DC-TransUnet network. These modules enable joint modeling of local and global features, as well as bidirectional interactive compensation of cross-modal features, thereby improving the detection accuracy of duckweed and algal blooms in complex aquatic environments. On the other hand, based on the duckweed water body segmentation results, the coverage ratio of duckweed and algal blooms in the water body is determined. This operation can provide a scientific basis for aquatic ecological management and pollution control. Attached Figure Description

[0021] Figure 1 This is a flowchart of the cross-modal information fusion method for detecting duckweed and algal blooms according to the present invention; Figure 2 This is a schematic diagram of the duckweed and algal bloom detection system based on cross-modal information fusion of the present invention; Figure 3 This is a flowchart of the cross-modal information fusion method for detecting duckweed and algal blooms in Embodiment 4 of the present invention; Figure 4 This is a structural diagram of the convolutional linear fusion module in Embodiment 4 of the present invention; Figure 5 This is a structural diagram of the cross-modal feature correction module in Embodiment 4 of the present invention; Figure 6 This is a schematic diagram of the expansion operation in Embodiment 4 of the present invention; Figure 7 This is a schematic diagram of the structure of the electronic device of the present invention. Detailed Implementation

[0022] To further understand the content of this invention, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments are merely illustrative and not limiting of the invention.

[0023] Example 1: The flowchart of the cross-modal information fusion method for detecting duckweed and algal blooms of this invention is as follows: Figure 1 As shown, the cross-modal information fusion method for detecting duckweed and algal blooms of the present invention includes the following steps: S1. Acquire visible light and near-infrared image data of duckweed and algal bloom areas; S2. Input the visible light band image data and near-infrared band image data of the duckweed area and the algal bloom area into the trained DC-TransUnet network to obtain the duckweed water body segmentation results; The DC-TransUnet network includes a ResNet network, a convolutional linear fusion module, and a cross-modal feature correction module; The ResNet network is used to extract features from visible light band image data to obtain visible light band image data features. The convolutional linear fusion module is used to extract features from near-infrared band image data to obtain near-infrared band image data features. The cross-modal feature correction module is used to perform bi-branch feature AC correction on the features of visible light band image data and near-infrared band image data to obtain the corrected features. S3. Based on the results of duckweed water body segmentation, determine the coverage ratio of duckweed and algal blooms in the water body; S4. Based on the coverage ratio of duckweed and algal blooms in the water body, the detection results of duckweed and algal blooms are obtained.

[0024] Example 2: A schematic diagram of the duckweed and algal bloom detection system based on cross-modal information fusion of this invention is shown below. Figure 2 As shown, the cross-modal information fusion detection system for duckweed and algal blooms of the present invention includes: The image data acquisition module is used to acquire visible light and near-infrared image data in the duckweed and algal bloom areas. The duckweed water body segmentation module is used to input visible light band image data and near-infrared band image data of duckweed area and algal bloom area into the trained DC-TransUnet network to obtain duckweed water body segmentation results; The DC-TransUnet network includes a ResNet network, a convolutional linear fusion module, and a cross-modal feature correction module; The ResNet network is used to extract features from visible light band image data to obtain visible light band image data features. The convolutional linear fusion module is used to extract features from near-infrared band image data to obtain near-infrared band image data features. The cross-modal feature correction module is used to perform bi-branch feature AC correction on the features of visible light band image data and near-infrared band image data to obtain the corrected features. The coverage ratio determination module is used to determine the coverage ratio of duckweed and algal blooms in the water body based on the duckweed water body segmentation results. The duckweed and algal bloom detection module is used to obtain duckweed and algal bloom detection results based on the coverage ratio of duckweed and algal blooms in the water body.

[0025] Example 3: The cross-modal information fusion method for detecting duckweed and algal blooms of the present invention includes the following steps: S1. Acquire visible light and near-infrared image data of duckweed and algal bloom areas.

[0026] S2. Input the visible light and near-infrared image data of the duckweed and algal bloom regions into the trained DC-TransUnet network to obtain the duckweed water body segmentation results.

[0027] The following is a description of the DC-TransUnet network: The DC-TransUnet network includes a ResNet network, a convolutional linear fusion module, and a cross-modal feature correction module.

[0028] The ResNet network is used to extract features from visible light band image data to obtain visible light band image data features.

[0029] The convolutional linear fusion module is used to extract features from near-infrared band image data to obtain near-infrared band image data features.

[0030] The cross-modal feature correction module is used to perform bi-branch feature AC correction on the features of visible light band image data and near-infrared band image data to obtain the corrected features.

[0031] Before obtaining the duckweed water body segmentation results, the visible light and near-infrared image data of the duckweed and algal bloom regions are input into the trained DC-TransUnet network to perform data augmentation on the visible light and near-infrared image data of the duckweed and algal bloom regions.

[0032] The training method for the DC-TransUnet network in this step includes: A dataset was constructed using visible light and near-infrared imagery data from duckweed and algal bloom areas. Based on the constructed dataset, it is divided into training set, validation set and test set in a ratio of 7:2:1. The DC-TransUnet network is trained using the training set.

[0033] The loss function used when training the DC-TransUnet network is the joint loss function, which is expressed as:

[0034] in, For the joint loss function, Let cross-entropy be the loss function. The Dice loss function, and These are the weight parameters.

[0035] Cross-entropy loss function The expression is:

[0036] Dice loss function The expression is:

[0037] in, Indicates the number of pixels. Indicates the first The label of each sample Indicates the prediction of the first The probability that a sample belongs to class 1. This indicates the smoothing term.

[0038] In this step, the convolutional linear fusion module is used to extract features from the near-infrared band image data, specifically including: Based on standard convolution, dilated convolution, and attention mechanisms, multi-scale features of near-infrared image data are captured.

[0039] S3. Based on the results of duckweed water body segmentation, determine the coverage ratio of duckweed and algal blooms in the water body.

[0040] In this step, based on the results of the duckweed water body segmentation, the step of determining the coverage ratio of duckweed and algal blooms in the water body involves using an expansion operation on the duckweed and algal bloom areas to eliminate the gaps between the duckweed and algal bloom areas and the water body in the results of the duckweed water body segmentation. The step of determining whether they are the same water body is based on whether they intersect after expansion.

[0041] S4. Based on the coverage ratio of duckweed and algal blooms in the water body, the detection results of duckweed and algal blooms are obtained.

[0042] After obtaining the test results for duckweed and algal blooms, the water body is classified and risk warnings are issued.

[0043] Example 4: The flowchart of the cross-modal information fusion method for detecting duckweed and algal blooms of this invention is as follows: Figure 3 As shown, the method of the present invention will be described in detail below: Step 1: First, based on the FBPS (Five-Billion-Pixels) dataset, visually observe and manually label the duckweed and algal bloom areas. Crop the original images to 224*224 pixels to construct the duckweed and algal bloom dataset.

[0044] Step 1.1: Manually label the original image, that is, visually observe the original image, extract the areas that are different from green vegetation and green artificial surface, identify samples as duckweed or algal blooms, and classify and label them.

[0045] Step 2: Using the algal bloom and duckweed dataset constructed in Step 1, feature extraction is performed on the image features of algal blooms and duckweed based on the DC-TransUnet network to obtain duckweed water body segmentation results. The objective function of the DC-TransUnet network is optimized using the stochastic gradient algorithm, and the weights of the DC-TransUnet network are adjusted layer by layer using the backpropagation mechanism to establish the duckweed and algal bloom extraction network. Step 3: After obtaining the duckweed water body segmentation results in Step 2, the coverage ratio of duckweed and algal blooms in the water body is further quantified using the coverage calculation module (also called the duckweed and algal bloom coverage calculation module). The quantified results from the coverage calculation module are applied to duckweed water body management and risk early warning, enabling accurate detection of water bodies urgently needing cleaning and long-term monitoring of potentially risky water bodies.

[0046] Step 1 will be explained in detail below: To avoid overfitting during DC-TransUnet network training, all images in the dataset are rotated and mirrored to achieve data augmentation.

[0047] Step 2 will be explained in detail below: Based on the TransUnet network, a dual-branch cross-modal TransUnet network, namely the DC-TransUnet network, is established. In the Encoder stage, a ResNet network is used for feature extraction from visible light image data, while a convolutional linear fusion module is used for feature extraction from near-infrared image data. During feature downsampling, a cross-modal feature correction module performs dual-branch feature exchange correction on the features of both visible light and near-infrared image data, resulting in corrected features (also called corrected image features). These corrected features, after passing through the Transformer layer, are uploaded to the decoder for layer-by-layer upsampling and feature skip connections, used to predict detection results.

[0048] The structure diagram of the convolutional linear fusion module is as follows: Figure 4 As shown, this module first processes the input features using standard convolution and dilated convolution with dilation rates of 1, 2, and 4 to obtain two sets of features with different receptive fields. The two sets of features are then added together to obtain fused local features. Simultaneously, average pooling is performed along the height and width directions of the input feature map to extract contextual information in the corresponding directions, generating a spatial attention map to highlight key regions and suppress irrelevant responses. Further, matrix multiplication is performed between the local features and the spatial attention map to obtain global representation features, which are then fused with the global features. Finally, the fused features are processed through a feedforward structure consisting of standard convolutional layers and pointwise convolutional layers to obtain the module's final output. This module extracts multi-scale information from near-infrared band image data by combining standard convolution, dilated convolution, and attention mechanisms, and further enhances the discriminative ability of the features through global modeling and feedforward mapping.

[0049] The structure diagram of the cross-modal feature correction module is as follows: Figure 5As shown, in terms of channel feature correction, for the input RGB and NIR features, channel information is first extracted through max pooling and average pooling layers, and then concatenated along the channel dimension to obtain a channel feature map. This feature map is then input into a multilayer perceptron module, where it is processed and reshaped to obtain channel weight coefficients. These coefficients represent the importance of each channel. Finally, the channel weights are applied to the RGB and NIR features respectively to obtain the channel correction result. In terms of spatial feature correction, for the input RGB and NIR features, they are first concatenated along the channel dimension to obtain a combined feature map. This combined feature map is then passed through a gating module to obtain modality weight coefficients for spatial location. These modality weight coefficients can adaptively adjust the contribution of different modalities at the pixel level: enhancing the dominant modality in information-rich regions and enhancing the complementary modality in other regions, thereby achieving fine-grained cross-modal spatial interaction. Finally, the spatial weights are applied to the RGB and NIR features respectively to obtain the spatial correction result. This module achieves bidirectional feature compensation through channel correction and spatial correction, thereby improving the detection accuracy of duckweed and algal blooms. The corrected image features (channel correction results and spatial correction results) are further input into the decoder part of the DC-TransUnet network.

[0050] During training, the DC-TransUnet network needs to compare the overlap between manually labeled areas of duckweed and algal blooms when detecting them, facilitating feedback and network adjustment. Therefore, this embodiment uses a joint loss function, expressed as:

[0051] in, For the joint loss function, Let cross-entropy be the loss function. The Dice loss function, and These are the weight parameters.

[0052] Cross-entropy loss function The expression is:

[0053] Dice loss function The expression is:

[0054] in, Indicates the number of pixels. Indicates the first The label of each sample Indicates the prediction of the first The probability that a sample belongs to class 1. This indicates the smoothing term.

[0055] Step 3 will be explained in detail below: In the step of determining the coverage ratio of duckweed and algal blooms in the water body based on the duckweed water body segmentation results, an expansion operation is applied to the duckweed and algal bloom areas to cancel the gaps between the duckweed and algal bloom areas and the water body in the duckweed water body segmentation results, and whether they are the same water body is determined by whether they intersect after expansion.

[0056] The expansion operation diagram is as follows: Figure 6 As shown, Figure 6 In the image, 1 represents duckweed and algal blooms detected by the DC-TransUnet network, 2 represents uncovered water surface, and 3 represents the entire water area. The coverage of duckweed or algal blooms is calculated by applying a dilation operation to the duckweed and algal bloom areas, causing them to intersect with uncovered water surfaces within the same water area, thus obtaining the overall water area. Furthermore, the presence or absence of intersections after dilation determines whether the water bodies belong to the same location, preventing errors in calculating the overall water area due to multiple water bodies appearing in the same image.

[0057] Compared with the prior art, the method of the present invention has the following beneficial effects: 1. This invention constructs a dual-branch encoder structure, extracting features from visible light and near-infrared image data in the encoder stage of the DC-TransUnet network, respectively. Based on this, a convolutional linear fusion module and a cross-modal feature correction module are introduced. The former achieves efficient extraction of local details and global semantics, while the latter promotes cross-modal feature interaction at both the channel and spatial levels, thereby significantly enhancing the detection accuracy of the DC-TransUnet network for duckweed and algal blooms in complex aquatic environments.

[0058] 2. This invention, based on the results of duckweed water body segmentation, quantifies the coverage ratio of duckweed and algal blooms in a complete closed water body (the coverage ratio of duckweed and algal blooms in the water body), and classifies the water body and provides risk warnings. This not only enhances the application value of duckweed and algal bloom detection results but also provides a scientific basis for water body ecological management and pollution prevention and control.

[0059] Example 5: Please see Figure 7 As shown, the present invention also provides an electronic device 100 for a cross-modal information fusion method for detecting duckweed and algal blooms; the electronic device 100 includes a memory 101, at least one processor 102, a computer program 103 stored in the memory 101 and executable on the at least one processor 102, and at least one communication bus 104.

[0060] The memory 101 can be used to store the computer program 103. The processor 102 implements the steps of the cross-modal information fusion method for detecting duckweed and algae blooms described in Embodiment 1 by running or executing the computer program stored in the memory 101 and calling the data stored in the memory 101. The memory 101 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device 100 (such as audio data), etc. In addition, the memory 101 may include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other non-volatile solid-state storage device.

[0061] The at least one processor 102 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 102 may be a microprocessor or any conventional processor. The processor 102 is the control center of the electronic device 100, connecting various parts of the electronic device 100 via various interfaces and lines.

[0062] The memory 101 in the electronic device 100 stores multiple instructions to implement a cross-modal information fusion method for detecting duckweed and algal blooms, and the processor 102 can execute the multiple instructions to achieve the following: Acquire visible light and near-infrared image data of duckweed and algal bloom areas; Visible light and near-infrared image data of duckweed and algal bloom areas are input into the trained DC-TransUnet network to obtain duckweed water body segmentation results; The DC-TransUnet network includes a ResNet network, a convolutional linear fusion module, and a cross-modal feature correction module; The ResNet network is used to extract features from visible light band image data to obtain visible light band image data features. The convolutional linear fusion module is used to extract features from near-infrared band image data to obtain near-infrared band image data features. The cross-modal feature correction module is used to perform bi-branch feature AC correction on the features of visible light band image data and near-infrared band image data to obtain the corrected features. Based on the results of duckweed water body segmentation, the coverage ratio of duckweed and algal blooms in the water body was determined. The detection results of duckweed and algal blooms were obtained based on the coverage ratio of duckweed and algal blooms in the water body.

[0063] Example 6: If the modules / units integrated in the electronic device 100 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0064] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0065] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0066] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0067] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for detecting duckweed and algal blooms through cross-modal information fusion, characterized in that, Includes the following steps: Acquire visible light and near-infrared image data of duckweed and algal bloom areas; Visible light and near-infrared image data of duckweed and algal bloom areas are input into the trained DC-TransUnet network to obtain duckweed water body segmentation results; The DC-TransUnet network includes a ResNet network, a convolutional linear fusion module, and a cross-modal feature correction module; The ResNet network is used to extract features from visible light band image data to obtain visible light band image data features. The convolutional linear fusion module is used to extract features from near-infrared band image data to obtain near-infrared band image data features. The cross-modal feature correction module is used to perform bi-branch feature AC correction on the features of visible light band image data and near-infrared band image data to obtain the corrected features. Based on the results of duckweed water body segmentation, the coverage ratio of duckweed and algal blooms in the water body was determined. The detection results of duckweed and algal blooms were obtained based on the coverage ratio of duckweed and algal blooms in the water body.

2. The method for detecting duckweed and algal blooms through cross-modal information fusion according to claim 1, characterized in that, Before obtaining the duckweed water body segmentation results, the visible light and near-infrared image data of the duckweed and algal bloom regions are input into the trained DC-TransUnet network to perform data augmentation on the visible light and near-infrared image data of the duckweed and algal bloom regions.

3. The method for detecting duckweed and algal blooms through cross-modal information fusion according to claim 1, characterized in that, The training method for the DC-TransUnet network includes: A dataset was constructed using visible light and near-infrared imagery data from duckweed and algal bloom areas. Based on the constructed dataset, it is divided into training set, validation set and test set according to the proportion, and the DC-TransUnet network is trained using the training set.

4. The method for detecting duckweed and algal blooms through cross-modal information fusion according to claim 3, characterized in that, The loss function used when training the DC-TransUnet network is the joint loss function, which is expressed as: in, For the joint loss function, Let cross-entropy be the loss function. The Dice loss function, and These are weight parameters; The cross-entropy loss function The expression is: The Dice loss function The expression is: in, Indicates the number of pixels. Indicates the first The label of each sample, Indicates the prediction of the first The probability that a sample belongs to class 1. This indicates the smoothing term.

5. The method for detecting duckweed and algal blooms through cross-modal information fusion according to claim 1, characterized in that, The convolutional linear fusion module is used to extract features from near-infrared band image data to obtain near-infrared band image data features, specifically including: Based on standard convolution, dilated convolution, and attention mechanisms, multi-scale features of near-infrared image data are captured.

6. The method for detecting duckweed and algal blooms through cross-modal information fusion according to claim 1, characterized in that, In the step of determining the coverage ratio of duckweed and algal blooms in the water body based on the duckweed water body segmentation results, an expansion operation is applied to the duckweed area and the algal bloom area to cancel the gap between the duckweed area and the water body in the duckweed water body segmentation results, and whether they are the same water body is determined by whether they intersect after expansion.

7. The method for detecting duckweed and algal blooms by cross-modal information fusion according to claim 1, characterized in that, After obtaining the test results for duckweed and algal blooms, the water body is classified and risk warnings are issued.

8. A cross-modal information fusion system for detecting duckweed and algal blooms, characterized in that, include: The image data acquisition module is used to acquire visible light and near-infrared image data in the duckweed and algal bloom areas. The duckweed water body segmentation module is used to input visible light band image data and near-infrared band image data of duckweed area and algal bloom area into the trained DC-TransUnet network to obtain duckweed water body segmentation results; The DC-TransUnet network includes a ResNet network, a convolutional linear fusion module, and a cross-modal feature correction module; The ResNet network is used to extract features from visible light band image data to obtain visible light band image data features. The convolutional linear fusion module is used to extract features from near-infrared band image data to obtain near-infrared band image data features. The cross-modal feature correction module is used to perform bi-branch feature AC correction on the features of visible light band image data and near-infrared band image data to obtain the corrected features. The coverage ratio determination module is used to determine the coverage ratio of duckweed and algal blooms in the water body based on the duckweed water body segmentation results. The duckweed and algal bloom detection module is used to obtain duckweed and algal bloom detection results based on the coverage ratio of duckweed and algal blooms in the water body.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method for detecting duckweed and algal blooms by cross-modal information fusion as described in any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the cross-modal information fusion method for detecting duckweed and algal blooms as described in any one of claims 1 to 7.

Images