Multi-modal data fusion unmanned aerial vehicle offshore enteromorpha efficient monitoring method and system

Abstract

The invention provides a multi-modal data fusion unmanned aerial vehicle marine enteromorpha efficient monitoring method and system, and relates to the technical field of image processing, and the method comprises the steps: obtaining a natural optical image and an infrared image of marine enteromorpha collected by an unmanned aerial vehicle; inputting the natural optical image into a first backbone network for feature coding to obtain an optical modal initial feature map; inputting the infrared image into a second backbone network for feature coding to obtain an infrared modal initial feature map; performing cross-modal fusion and enhancement processing on the optical modal initial feature map and the infrared modal initial feature map to obtain an enhanced fusion feature map; and inputting the enhanced fusion feature map into a target detection network, and outputting position information, confidence and category information of the enteromorpha plaque. According to the method, the problems of poor adaptability and small target missing detection of single-mode monitoring in a complex environment are effectively solved, and the accuracy, the robustness and the early warning capability of enteromorpha monitoring are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and system for efficient monitoring of marine seaweed using unmanned aerial vehicles (UAVs) through multimodal data fusion. Background Technology

[0002] Ulva prolifera is a large green algae that can cause severe damage to marine ecosystems when it grows explosively and forms green tides. Specifically, large amounts of Ulva prolifera block sunlight, affecting phytoplankton photosynthesis, disrupting the ecological balance, and leading to a decline in biodiversity. The decaying Ulva prolifera consumes large amounts of dissolved oxygen, causing hypoxia and threatening the survival of marine life. Therefore, timely and effective monitoring of Ulva prolifera is crucial.

[0003] Traditional methods for monitoring *Ulva prolifera* mainly include visual monitoring and satellite remote sensing. Visual monitoring is inefficient and has a limited monitoring range; satellite remote sensing has low resolution, making it difficult to accurately identify small areas of *Ulva prolifera* accumulation, and is greatly affected by weather factors. Therefore, existing methods are insufficient to meet the needs of real-time and accurate monitoring.

[0004] In recent years, deep learning-based target detection technologies (such as the YOLO algorithm) have been introduced into the field of Ulva prolifera monitoring. However, existing methods have a core problem: they rely on visible light image data for monitoring, have poor adaptability in complex sea surface environments (such as strong reflections and haze), and are insufficient in detecting small, scattered patches of Ulva prolifera in the early stages of an outbreak, making it difficult to achieve early warning and comprehensive and accurate monitoring. Summary of the Invention

[0005] To address the technical problems of existing seaweed monitoring methods in the background, such as poor adaptability in complex environments and difficulty in early detection of small seaweed, this invention provides a multimodal data fusion-based UAV-based efficient seaweed monitoring scheme by introducing a dual-backbone network architecture, a cross-attention fusion mechanism, and a small target feature enhancement module based on Hilbert transform. This effectively improves the accuracy, robustness, and early warning capability of the monitoring.

[0006] To achieve the above objectives, the first aspect of the present invention provides a method for efficient monitoring of marine seaweed using unmanned aerial vehicles (UAVs) based on multimodal data fusion, comprising: Acquire natural optical and infrared images of seaweed collected by drones; The natural optical image is input into the first backbone network for feature encoding to obtain an initial feature map of the optical mode; the infrared image is input into the second backbone network for feature encoding to obtain an initial feature map of the infrared mode. The initial feature map of the optical mode and the initial feature map of the infrared mode are subjected to cross-modal fusion and enhancement processing to obtain an enhanced fused feature map; The enhanced fused feature map is input into the target detection network, which outputs the location information, confidence level, and category information of the seaweed patches.

[0007] Furthermore, before inputting the natural optical image and infrared image into the backbone network, image preprocessing is also included; the preprocessing includes: Adaptive threshold denoising and color correction are performed on the natural optical image; The infrared image is subjected to grayscale stretching and thermal noise removal processing.

[0008] Furthermore, after preprocessing, the size normalization process is also performed on the preprocessed natural optical image and infrared image.

[0009] Furthermore, the cross-modal fusion and enhancement processing includes a feature fusion step, which specifically involves: The initial feature map of the optical mode and the initial feature map of the infrared mode are input into the cross-fusion module; In the cross-fusion module, the initial feature maps of the optical mode and the infrared mode are first enhanced by a single-modal self-attention mechanism. Then, through a cross-attention mechanism, the query vector of one mode interacts with the key vector and value vector of another mode to achieve dynamic aggregation of dual-modal features and obtain a fused feature map.

[0010] Furthermore, the cross-modal fusion and enhancement processing also includes a small target feature enhancement step, which is performed after the feature fusion step, specifically as follows: The fused feature map is input into the small target feature enhancement module; In the small target feature enhancement module, Hilbert transforms are performed on the initial optical modal feature map and the initial infrared modal feature map, respectively, to decompose them into real and imaginary parts; the imaginary part of the initial optical modal feature map is fused with the real part of the initial infrared modal feature map to obtain a first composite feature; the imaginary part of the initial infrared modal feature map is fused with the real part of the initial optical modal feature map to obtain a second composite feature; and an inverse Hilbert transform is performed on the first composite feature and the second composite feature to map them back to the spatial domain, resulting in the enhanced fused feature map.

[0011] Furthermore, the target detection network is based on the YOLO architecture, and its detection head receives processed multi-scale feature maps; the multi-scale feature maps include feature maps at least three scales: P3, P4, and P5; the detection head outputs preliminary detection results through anchor box matching, bounding box regression, and category prediction, and eliminates redundant detection boxes through confidence ranking and non-maximum suppression processing, outputting the final information on the location, confidence, and category of the seaweed patches.

[0012] A second aspect of the present invention provides a high-efficiency monitoring system for marine seaweed using a drone based on multimodal data fusion, comprising: The image acquisition module is used to acquire natural optical and infrared images of seaweed collected by the UAV; The feature encoding module is used to input the natural optical image into a first backbone network for feature encoding to obtain an initial feature map of the optical mode; and to input the infrared image into a second backbone network for feature encoding to obtain an initial feature map of the infrared mode. The feature fusion and enhancement module is used to perform cross-modal fusion and enhancement processing on the initial feature map of the optical mode and the initial feature map of the infrared mode to obtain an enhanced fused feature map; The target detection module is used to input the enhanced fused feature map into the target detection network and output the location information, confidence level and category information of the seaweed patches.

[0013] A third aspect of the present invention provides an electronic device including a memory, a processor, and a program stored in the memory and running on the processor, wherein the processor executes the program to implement the steps in the UAV-based efficient monitoring method for marine seaweed using multimodal data fusion as described in the first aspect of the present invention.

[0014] A fourth aspect of the present invention provides a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the steps of the UAV-based efficient monitoring method for marine seaweed using multimodal data fusion as described in the first aspect of the present invention.

[0015] A fifth aspect of the present invention provides a computer program product comprising software code, wherein the program in the software code performs the steps of the UAV-based efficient monitoring method for marine seaweed using multimodal data fusion as described in the first aspect of the present invention.

[0016] Compared with existing technologies, the present invention provides a method and system for efficient monitoring of marine seaweed using multimodal data fusion by unmanned aerial vehicles (UAVs), which has the following advantages: (1) In view of the technical problem that existing single-mode (relying only on visible light images) monitoring methods have poor adaptability in complex sea conditions, this invention extracts the texture and color information of natural optical images and the temperature contour information of infrared images through two independent backbone networks, and uses a cross-fusion module to dynamically aggregate the complementary advantages of the two. This allows the system to make reliable judgments based on the effective information of the other mode even when the imaging quality of a single mode is poor (such as optical images being interfered with by reflections or infrared images lacking texture). This significantly improves the environmental adaptability and detection robustness of the monitoring system under severe weather and complex lighting conditions.

[0017] (2) In view of the technical problems of insufficient detection capability and easy missed detection of small-sized seaweed patches in the existing technology, the present invention maps the features to the complex frequency domain and separates them into low-frequency real part carrying the global background and high-frequency imaginary part containing local details. It also designs an interactive fusion strategy of optical high frequency (details) and infrared low frequency (background), infrared high frequency (edge) and optical low frequency (background), so that small target features can simultaneously obtain dual enhancement of color texture details and edge contours, effectively alleviate the problem of small target detail loss caused by downsampling, significantly improve the recognition accuracy of early sporadic seaweed patches, and provide technical support for realizing early warning of disasters.

[0018] (3) This invention first strengthens the significant Ulva features within each modality through self-attention, and then accurately calculates the correlation weights of features between modalities through cross-attention, thereby achieving on-demand and dynamic information aggregation, avoiding information interference caused by simple feature splicing or addition, and ensuring that the fused feature map has both rich semantic information and accurate spatial positioning capabilities, thus optimizing the overall detection performance.

[0019] (4) The present invention improves the YOLO architecture by receiving and utilizing the multi-scale feature maps (P3, P4, P5) enhanced by the aforementioned modules, and by using its efficient anchor box matching and regression mechanism, so that the improved network can accurately locate and identify seaweed patches of different scales from large to small. While maintaining the detection speed, it greatly improves the detection comprehensiveness and accuracy in complex marine scenarios. Attached Figure Description

[0020] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute an undue limitation of this disclosure.

[0021] Figure 1 This is a flowchart of a multimodal data fusion-based method for efficient monitoring of marine seaweed using unmanned aerial vehicles (UAVs) according to Embodiment 1 of the present invention. Figure 2 This is a framework diagram of the UAV-based efficient monitoring method for marine seaweed using multimodal data fusion, provided in Embodiment 1 of the present invention. Figure 3 This is a schematic diagram of the self-attention mechanism provided in Embodiment 1 of the present invention; Figure 4 This is a schematic diagram of the cross-attention mechanism provided in Embodiment 1 of the present invention; Figure 5 This is a framework diagram of the cross-fusion module provided in Embodiment 1 of the present invention; Figure 6 This is a schematic diagram of the Hilbert transform provided in Embodiment 1 of the present invention; Figure 7This is an architecture diagram of the UAV-based high-efficiency monitoring system for marine seaweed using multimodal data fusion, provided in Embodiment 2 of the present invention. Detailed Implementation

[0022] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0023] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. Furthermore, it should be understood that the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0025] All data acquisition in this embodiment is carried out in accordance with laws and regulations and with user consent, and the data is used legally.

[0026] Before describing the embodiments of the present invention in detail, the technical terms involved in the present invention are explained and defined as follows: 1. Dual-backbone network: In this invention, it specifically refers to a feature extraction architecture composed of two convolutional neural network (CNN) backbones arranged side by side, which have the same or similar structures but do not share parameters. The first backbone network is specifically responsible for processing natural optical images to extract visual features such as texture and color; the other backbone network, the second backbone network, is specifically responsible for processing infrared images to extract thermal radiation features such as temperature and contour.

[0027] 2. Cross-Attention Mechanism: This is the core module used in this invention to achieve multimodal feature fusion. Its working principle is as follows: for a feature of one modality, it is treated as a "query," while features of another modality are treated as "keys" and "values." Attention weights are obtained by calculating the similarity between the "query" and the "key," and then these weights are used to weighted sums of the "values," thereby dynamically filtering and extracting the most relevant complementary information from the other modality to the current modality.

[0028] 3. Hilbert Transform: In this invention, it is a signal processing transform applied to feature maps to map features from the spatial domain to the complex frequency domain. The transformed features are decomposed into real and imaginary parts: the real part mainly carries low-frequency information such as the global contour and background of the feature; the imaginary part reflects high-frequency details such as local edges and textures of the feature.

[0029] Example 1 like Figure 1 This embodiment provides a multimodal data fusion-based method for efficient monitoring of marine seaweed by unmanned aerial vehicles (UAVs), including: Acquire natural optical and infrared images of seaweed collected by drones; The natural optical image is input into the first backbone network for feature encoding to obtain an initial feature map of the optical mode; the infrared image is input into the second backbone network for feature encoding to obtain an initial feature map of the infrared mode. The initial feature map of the optical mode and the initial feature map of the infrared mode are subjected to cross-modal fusion and enhancement processing to obtain an enhanced fused feature map; The enhanced fused feature map is input into the target detection network, which outputs the location information, confidence level, and category information of the seaweed patches.

[0030] This invention separates and optimizes feature extraction from different modalities at the source through a dual-backbone network, laying the foundation for subsequent fusion; cross-modal fusion and enhancement processing deeply mine and integrate complementary information from both modalities; finally, a mature target detection network (YOLO) completes accurate localization and identification. The monitoring method provided in this embodiment, through systematic process design, comprehensively solves the monitoring performance bottleneck problem caused by a single data source and simple information fusion.

[0031] Specifically, before inputting the natural optical image and infrared image into the backbone network, image preprocessing is also included; the preprocessing includes: Adaptive threshold denoising and color correction are performed on the natural optical image; The infrared image is subjected to grayscale stretching and thermal noise removal processing.

[0032] Specifically, after preprocessing, the process also includes size normalization of the preprocessed natural optical image and infrared image.

[0033] To address issues such as surface reflection and shadow noise in natural optical images, and insufficient contrast and thermal noise in infrared images, targeted data cleaning and enhancement are implemented. Adaptive threshold denoising dynamically adapts to changes in illumination in different regions, effectively suppressing specular interference; color correction aims to restore the true color characteristics of *Ulva prolifera* under complex lighting conditions. Grayscale stretching expands the grayscale dynamic range of infrared images, making the temperature difference between *Ulva prolifera* and seawater more apparent. Size normalization ensures the spatial consistency of the dual-modal images, providing the necessary conditions for parallel processing of the subsequent backbone network and multi-scale feature alignment, thus improving the accuracy and stability of subsequent feature extraction and fusion from the data source.

[0034] Specifically, the cross-modal fusion and enhancement processing includes a feature fusion step, which specifically involves: The initial feature map of the optical mode and the initial feature map of the infrared mode are input into the cross-fusion module; In the cross-fusion module, the initial feature maps of the optical mode and the infrared mode are first enhanced by a single-modal self-attention mechanism. Then, through a cross-attention mechanism, the query vector of one mode interacts with the key vector and value vector of another mode to achieve dynamic aggregation of dual-modal features and obtain a fused feature map.

[0035] The cross-fusion module simulates how the human visual system processes multi-source information. The single-modal self-attention mechanism first "focuses" within each modality, assigning higher weights to areas related to *Ulva prolifera* by calculating the correlation between all locations within the feature map, thereby suppressing background interference and highlighting the target. Subsequently, the cross-attention mechanism establishes a "dialogue" bridge between two modalities. For example, it uses the "query" of optical features to find relevant temperature contour information in the "key-value" pairs of infrared features for supplementation, and vice versa. This mechanism can dynamically and selectively aggregate complementary information across modalities, achieving a feature fusion effect of "1+1>2," effectively overcoming the adverse effects of environmental factors on a single modality.

[0036] Specifically, the cross-modal fusion and enhancement processing further includes a small target feature enhancement step, which is performed after the feature fusion step, and specifically as follows: The fused feature map is input into the small target feature enhancement module; In the small target feature enhancement module, Hilbert transforms are performed on the initial optical modal feature map and the initial infrared modal feature map, respectively, to decompose them into real and imaginary parts; the imaginary part of the initial optical modal feature map is fused with the real part of the initial infrared modal feature map to obtain a first composite feature; the imaginary part of the initial infrared modal feature map is fused with the real part of the initial optical modal feature map to obtain a second composite feature; and an inverse Hilbert transform is performed on the first composite feature and the second composite feature to map them back to the spatial domain, resulting in the enhanced fused feature map.

[0037] The small target feature enhancement module is based on frequency domain analysis theory in signal processing. The Hilbert transform maps spatial domain features to the complex frequency domain. The real part mainly contains low-frequency information describing the large-scale background and overall shape of the target, while the imaginary part focuses on high-frequency information depicting details such as target edges and textures. Through cross-modal interaction between the optical imaginary part (high-frequency details), the infrared real part (low-frequency background), and the infrared imaginary part (high-frequency edges) and the optical real part (low-frequency background), the detail resolution capability of the optical mode and the contour discrimination capability of the infrared mode are essentially recombined in the frequency domain. After the inverse transform, small targets simultaneously acquire richer details and clearer boundaries, significantly enhancing their representational ability and directly improving the model's sensitivity and detection rate for early-stage, small *Ulva prolifera* patches.

[0038] Specifically, the target detection network is based on the YOLO architecture, and its detection head receives processed multi-scale feature maps. The multi-scale feature maps include feature maps at least at three scales: P3, P4, and P5. The detection head outputs preliminary detection results through anchor box matching, bounding box regression, and category prediction, and eliminates redundant detection boxes through confidence ranking and non-maximum suppression processing, outputting the final information on the location, confidence, and category of the seaweed patches.

[0039] The YOLO-based object detection network utilizes multi-scale feature maps optimized by the aforementioned modules. P5 (lower resolution) has a large receptive field, suitable for detecting large *Ulva prolifera* clusters; P4 and P3 (resolution increasing sequentially) have smaller receptive fields but retain more spatial details, making them particularly suitable for detecting small to medium-sized patches. This invention inputs the enhanced multi-scale features into the YOLO detection head, enabling the network to fully utilize feature information at different levels. Anchor box matching pre-sets candidate boxes of different sizes and proportions, bounding box regression fine-tunes them to accurately enclose the target, and category prediction determines whether it is *Ulva prolifera*. Confidence ranking and non-maximum suppression are used to filter the final results, removing overlapping and unreliable detection boxes, thereby outputting high-precision *Ulva prolifera* location and category information in complex backgrounds.

[0040] In one specific embodiment, this embodiment proposes a multimodal data fusion-based method for efficient monitoring of marine seaweed using unmanned aerial vehicles (UAVs), including: First, natural optical and infrared images were acquired by a drone. Then, both types of images underwent preprocessing to optimize features: for the natural optical images, adaptive threshold denoising was used to eliminate interference from strong sea surface reflections and wave shadows, followed by color correction to restore the true greenness of the seaweed; for the infrared images, grayscale stretching was used to enhance the temperature contrast between the seaweed and the sea surface, while removing isolated thermal noise. Finally, the preprocessed images were normalized to a preset resolution. The core purpose of this approach is to provide standardized, high-quality input data for the subsequent feature encoding, self-attention, and cross-attention feature fusion modules of the dual-backbone network, avoiding the impact of data differences on feature extraction accuracy.

[0041] The overall approach of the scheme described in this embodiment is as follows: An improved YOLO framework for multimodal marine seaweed monitoring is designed. This framework is based on the original YOLO backbone network structure and optimized to construct a dual-backbone network parallel coding architecture. One backbone network is specifically adapted to the feature extraction requirements of natural optical images, while the other backbone network is adapted to the feature extraction requirements of infrared images, respectively used to accurately capture the texture and color features and temperature contour features of seaweed. After the two types of feature extraction are completed, the saliency of the seaweed features is first enhanced through a single-modal self-attention mechanism, and then a cross-attention fusion module is introduced to perform cross-modal aggregation of the dual-modal features extracted by the two backbone networks. Finally, relying on the original YOLO detection head, the location information and category prediction results of the seaweed are output.

[0042] The improved YOLO framework proposed in this embodiment focuses on addressing the information limitations of traditional single-modal YOLO in marine seaweed monitoring: When only a single natural optical image is input, the YOLO backbone network is easily affected by environmental interference such as strong reflections from the sea surface and wave shadows, resulting in blurred boundaries between seaweed and the background, making it difficult to accurately distinguish the target; when only a single infrared image is input, although it can clearly show the temperature difference between seaweed and the sea surface, the features extracted by the YOLO backbone network lack texture details, easily misclassifying non-seaweed high-temperature areas (such as floating objects, local heat flow) as seaweed. The core reason lies in the fundamental difference in feature expression logic between the two modalities: natural optical images rely on local features of color and texture to distinguish targets, while infrared images rely on global comparisons of temperature differences to distinguish targets. The traditional single-backbone network structure of YOLO cannot simultaneously adapt to the feature extraction needs of both modalities. This framework adapts to two types of images using a dual-backbone network, fully leveraging the information advantages of each modality. Simultaneously, the introduction of a cross-fusion module avoids the drawback of indiscriminate mixing of modal information in traditional fusion methods, achieving "on-demand fusion" through dynamic calculation of association weights, ensuring the accurate utilization of effective information. Therefore, the key idea of ​​this scheme is: based on a dual-backbone network, a cross-fusion module establishes precise associations of high-level features across the two modalities, enabling the YOLO framework to simultaneously utilize the detail resolution of natural optical images and the target discrimination of infrared images, significantly improving the accuracy and environmental robustness of marine seaweed monitoring.

[0043] The overall process of this embodiment is as follows: The first step is to acquire natural optical images and infrared images captured by the drone. Adaptive threshold denoising and color correction are performed on the natural optical images in sequence, and grayscale stretching and thermal noise removal are performed on the infrared images in sequence. Then, the two types of pre-processed images are normalized to a preset resolution. The second step, as Figure 1 As shown, a dual-backbone network is built based on the YOLO framework—one branch is designed for natural optical images, extracting the texture and color features of Ulva prolifera through multi-scale convolution operations; the other branch is adapted for infrared images, capturing the temperature differences and contour features of Ulva prolifera by adjusting the convolution parameters, and finally outputting a dual-modal high-level feature map. The third step is to connect a cross-fusion module at the output of the dual backbone network: first, the salience of the seaweed features is enhanced through a single-modal self-attention mechanism, and then the association weights of the dual-modal features are quantified through cross-modal cross-attention, dynamically aggregating complementary information to generate a fused feature map. The fourth step involves inputting the fused feature map into the YOLO detection head. After steps such as anchor box matching, bounding box regression, and category prediction, the location coordinates, confidence level, and category information of the seaweed patches are output, thus completing the multimodal seaweed monitoring process.

[0044] Specifically, such as Figure 2 As shown, in this embodiment, the acquired images are preprocessed, namely, the natural optical images are subjected to adaptive threshold denoising and color correction, the infrared images are subjected to grayscale stretching and thermal noise removal, and the two types of images are normalized to a preset resolution [640, 640].

[0045] After image preprocessing, the natural optical images were respectively... (Size [3, 640, 640]) and infrared images The inputs (size [1, 640, 640]) into the first and second backbone networks will produce a set of convolutional feature maps: feature map P3 (size [80, 80]), suitable for detecting larger objects; feature map P4 (size [40, 40]), suitable for detecting medium-sized objects; and feature map P5 (size [20, 20]), suitable for detecting smaller objects.

[0046]

[0047]

[0048] in, and These are the first backbone network and the second backbone network, respectively. These represent the output feature maps P3, P4, and P5 of the first backbone network, respectively. P3, P4, and P5 represent the output feature maps of the second backbone network, respectively.

[0049] like Figure 3 As shown, the feature map of a single-modal (natural optical or infrared) image after being extracted by a backbone network. (Where H and W are the feature map dimensions, and C is the number of channels), through 3 Convolutional layer (corresponding weight matrix) , , Perform a linear transformation on X to generate a query vector Q, a key vector K, and a value vector V.

[0050]

[0051] in , They are respectively , The dimension is usually set as (h represents the number of attention heads). Next, the transpose of Q and the dot product of K are calculated and normalized to obtain the attention score matrix:

[0052] The score matrix is ​​normalized using the Softmax function to generate the attention weight matrix A, which quantifies the association strength at each location in the feature space.

[0053] Finally, the weight matrix A and the value vector V are multiplied by weight to obtain the feature map after self-attention enhancement. To preserve the original feature information and ensure stable training, then... Convolution (corresponding to the weight matrix W) yields a self-attention feature map, which is the enhanced single-modal feature.

[0054] This process significantly enhances the recognizability of Ulva prolifera features within a single modality by dynamically assigning high weights to Ulva prolifera feature regions.

[0055] Next, bidirectional cross-attention is applied to the self-attention-based multimodal features at the same scale, aiming to enhance the expressive power of the multimodal features by strengthening the complementary information between the two modalities. For example... Figure 4 As shown, firstly, the natural optical feature map after self-attention... and infrared feature map Perform a linear transformation. For each mode, use three independent sets of... The convolutional layer maps its feature map, generating query (Q), key (K), and value (V) vectors. Specifically, this is done for natural optical feature maps. ,pass Convolution generates query vectors key vector Sum value vector Similarly, for infrared feature maps Calculate the query vector and key-value vectors and .

[0056] Next, the cross-modal correlation score is calculated. In the direction from the natural optical mode to the infrared mode ( The query vector is calculated using the dot product. and key vector The degree of matching is used to obtain the association score:

[0057] in, It is the feature dimension. This is the scaling factor. The attention weight matrix is ​​calculated using the Softmax activation function:

[0058] Similarly, in the direction from infrared mode to natural optical mode ( ), calculate query vector and key vector The dot product yields the associated score:

[0059] Then, the corresponding attention weight matrix is ​​obtained using Softmax:

[0060] The attention weight matrix obtained through calculation and The features of the two modalities are weighted and aggregated to extract complementary information. For the natural optical modality... From infrared mode The complementary information is calculated using the following formula:

[0061] For infrared mode From natural optical modes Complementary information, calculate:

[0062] Then, these two aggregated features and They are assembled together according to the channels, and connected through a... The convolutional layer performs dimensionality compression to obtain the final fused features. :

[0063] To further enhance the correlation between modalities while preserving the original feature information, residual connections are introduced. Specifically, the fused features are compared with the original natural optical feature map. and infrared feature map Add them together to form the final result:

[0064] This bidirectional cross-attention mechanism effectively improves the fusion effect of multimodal features by enhancing the complementary information between the natural optical mode and the infrared mode, and can significantly improve performance.

[0065] The final output feature map (Output) is then fed into the Neck section of the YOLO network to complete the detection of seaweed.

[0066] This embodiment proposes an innovative, multimodal data fusion-based method for efficient monitoring of marine seaweed using unmanned aerial vehicles (UAVs). It employs a combination of dual-backbone networks and a cross-attention mechanism, fusing features from natural optical and infrared images to improve monitoring accuracy. Compared to the traditional single-modal YOLO method, this approach uses parallel dual-backbone networks to extract features from natural optical and infrared images separately, effectively avoiding the limitations of a single modality, such as sea surface reflection and background interference. Furthermore, the cross-attention module enables complementary information fusion between the two modalities, further enhancing detection accuracy and robustness.

[0067] To address the issue of intermodal feature differences, a cross-fusion module is employed to accurately correlate high-level features of both modalities. For example... Figure 5 As shown, the cross-fusion module first enhances the saliency of each modality's features through a single-modal self-attention mechanism during feature fusion. Then, it dynamically filters complementary information through cross-modal cross-attention, ensuring accurate fusion of information from different modalities and avoiding information redundancy and feature confusion. Through this design, the scheme effectively improves target discrimination and boundary clarity in the *Ulva prolifera* monitoring task, especially significantly enhancing robustness in complex environments.

[0068] Compared to traditional single-backbone structures, this approach fully utilizes the texture and color features of natural optical images, as well as the temperature profile features of infrared images, solving the problem of insufficient single-modal feature representation and further improving the accuracy of *Ulva prolifera* detection. This embodiment successfully enhances the performance of multimodal data fusion by combining a dual-backbone network with a cross-fusion module, thus improving the overall performance of this task.

[0069] In one specific embodiment, this embodiment proposes a multimodal data fusion-based method for efficient monitoring of marine seaweed using unmanned aerial vehicles (UAVs), including: Acquire preprocessed natural optical and infrared images; The preprocessed natural optical image and infrared image are input into the first backbone network and the second backbone network of the dual backbone network, respectively, to complete the initial feature encoding and obtain the initial feature map of the optical mode and the initial feature map of the infrared mode. Hilbert Transform (HT) is performed on the initial feature maps of the two modalities respectively, mapping the spatial domain features to the complex frequency domain and decomposing them into real and imaginary parts: the real part represents the global morphology of the features, containing low-frequency information such as background contours and large-scale scene distribution, providing stable global environmental constraints for the detection task; the imaginary part focuses on local detail features, carrying high-frequency discriminative information such as texture and edges of small targets. To address the dual requirements of global background stability and local detail discriminativeness in small target detection, a cross-modal high- and low-frequency information interaction strategy is designed: by fusing the imaginary part of the optical modality (encoding high-frequency details of small target color and texture) with the real part of the infrared modality (providing a low-frequency reference for the global background), a composite feature focusing on small target details is constructed; simultaneously, the imaginary part of the infrared modality (enhancing high-frequency features of small target edge contours) is combined with the real part of the optical modality (providing an accurate low-frequency structure of the global scene) to generate a composite representation that enhances the edge discriminativeness of small targets. After the composite features of the two types mentioned above are inversely mapped back to the spatial domain, the high-frequency features of small targets can simultaneously acquire dual-modal information gain—optical high frequency supplements color and texture details, and infrared high frequency enhances edge and contour features, thereby effectively alleviating the problem of blurred details and background interference of small targets under single mode, and providing a significant feature enhancement effect for accurate detection of small targets.

[0070] The overall process of this embodiment is as follows: the preprocessed natural optical image and infrared image are input into the first backbone network and the second backbone network in the dual backbone network, respectively, to complete the encoding of the initial features and obtain the optical modal feature map. and infrared modal feature map .

[0071] Next, HT is performed on the same-scale feature maps of the two modalities respectively:

[0072] Among them, HT ( ) indicates the feature map Perform a Hilbert transform to generate a feature representation in the complex frequency domain. The Hilbert transform maps the feature map (X) to the complex domain and decomposes it into real and imaginary parts:

[0073] Real part and It mainly contains low-frequency information, which corresponds to the global background contour and large-scale scene distribution in the optical and infrared modes, respectively, providing stable global environmental constraints for the detection task.

[0074] virtual part and It contains high-frequency information, mainly reflecting the local details of small targets, such as edges and textures, and is an important basis for distinguishing small targets from the background.

[0075] To address the dual requirements of global background stability and local detail recognition in small target detection tasks, a cross-modal high- and low-frequency information interaction strategy was designed. For example... Figure 6 As shown, the imaginary part (high-frequency details) of the optical mode is first fused with the real part (low-frequency background) of the infrared mode to obtain a composite feature of focused small target details:

[0076] Similarly, by fusing the imaginary part (high-frequency edges) of the infrared mode with the real part (low-frequency background) of the optical mode, a composite characterization that enhances the edge discrimination of small targets is obtained:

[0077] Then, return to the spatial domain via the inverse Hilbert transform:

[0078]

[0079] in, IHT This represents the inverse Hilbert transform, used to recover the feature map from the complex frequency domain to the spatial domain.

[0080] This method effectively fuses high-frequency and low-frequency information from optical and infrared modes, enhancing the details and edge features of small targets, thereby improving the accuracy and robustness of small target detection. This deep fusion of cross-modal information provides a significant feature enhancement effect for the accurate detection of small targets.

[0081] Example 2 like Figure 7 As shown, this embodiment provides a multimodal data fusion-based UAV-based high-efficiency monitoring system for marine seaweed, including: The image acquisition module is used to acquire natural optical and infrared images of seaweed collected by the UAV; The feature encoding module is used to input the natural optical image into a first backbone network for feature encoding to obtain an initial feature map of the optical mode; and to input the infrared image into a second backbone network for feature encoding to obtain an initial feature map of the infrared mode. The feature fusion and enhancement module is used to perform cross-modal fusion and enhancement processing on the initial feature map of the optical mode and the initial feature map of the infrared mode to obtain an enhanced fused feature map; The target detection module is used to input the enhanced fused feature map into the target detection network and output the location information, confidence level and category information of the seaweed patches.

[0082] Example 3 Embodiment 3 of the present invention provides an electronic device.

[0083] An electronic device includes a memory, a processor, and a program stored in the memory and running on the processor. When the processor executes the program, it implements the steps in the UAV-based efficient monitoring method for marine seaweed using multimodal data fusion as described in Embodiment 1 of the present invention.

[0084] The detailed steps are the same as those of the UAV-based efficient monitoring method for marine seaweed using multimodal data fusion provided in Example 1, and will not be repeated here.

[0085] Example 4 Embodiment 4 of the present invention provides a computer-readable storage medium.

[0086] A computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in the UAV-based efficient monitoring method for marine seaweed using multimodal data fusion as described in Embodiment 1 of the present invention.

[0087] The detailed steps are the same as those of the UAV-based efficient monitoring method for marine seaweed using multimodal data fusion provided in Example 1, and will not be repeated here.

[0088] Example 5 Embodiment 5 of the present invention provides a computer program product.

[0089] A computer program product includes software code, wherein the program in the software code performs the steps of the UAV-based efficient monitoring method for marine seaweed using multimodal data fusion as described in Embodiment 1 of the present invention.

[0090] The detailed steps are the same as those of the UAV-based efficient monitoring method for marine seaweed using multimodal data fusion provided in Example 1, and will not be repeated here.

[0091] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present invention can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0092] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0093] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0094] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0095] The above description is merely a preferred embodiment of this practice and is not intended to limit the scope of this practice. Various modifications and variations can be made to this practice by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this practice should be included within the protection scope of this practice.

Claims

1. A multi-modal data fusion unmanned aerial vehicle offshore green algae efficient monitoring method, characterized in that, The method comprises the following steps: acquiring natural optical images and infrared images of marine green algae collected by a UAV; inputting the natural optical images into a first backbone network for feature encoding to obtain initial feature maps of the optical modality; inputting the infrared images into a second backbone network for feature encoding to obtain initial feature maps of the infrared modality; performing cross-modal fusion and enhancement processing on the initial feature maps of the optical modality and the initial feature maps of the infrared modality to obtain enhanced fusion feature maps; inputting the enhanced fusion feature maps into a target detection network to output position information, confidence and category information of green algae patches.

2. The method of claim 1, wherein, Before the natural optical images and the infrared images are inputted into the backbone network, the method further comprises pre-processing the images; the pre-processing comprises: performing adaptive threshold denoising processing and color correction processing on the natural optical images; performing gray scale stretching processing and thermal noise removal processing on the infrared images.

3. The method of claim 2, wherein, After the pre-processing, the method further comprises performing size normalization processing on the pre-processed natural optical images and infrared images.

4. The method of claim 1, wherein, The cross-modal fusion and enhancement processing comprises a feature fusion step, which specifically comprises: inputting the initial feature maps of the optical modality and the initial feature maps of the infrared modality into a cross fusion module; in the cross fusion module, first, feature strengthening is performed on the initial feature maps of the optical modality and the initial feature maps of the infrared modality respectively through a single modality self-attention mechanism; then, through a cross-attention mechanism, the query vector of one modality interacts with the key vector and the value vector of the other modality to realize dynamic aggregation of the features of the two modalities, thereby obtaining fusion feature maps.

5. The method of claim 4, wherein, The cross-modal fusion and enhancement processing further comprises a small target feature enhancement step, which is executed after the feature fusion step, and specifically comprises: inputting the fusion feature maps into a small target feature enhancement module; in the small target feature enhancement module, the initial feature maps of the optical modality and the initial feature maps of the infrared modality are decomposed into real and imaginary parts through Hilbert transform; the imaginary part of the initial feature maps of the optical modality is fused with the real part of the initial feature maps of the infrared modality to obtain first composite features; the imaginary part of the initial feature maps of the infrared modality is fused with the real part of the initial feature maps of the optical modality to obtain second composite features; inverse Hilbert transform is performed on the first composite features and the second composite features to map them back to the spatial domain, thereby obtaining the enhanced fusion feature maps.

6. The method of claim 1, wherein, The target detection network is based on a YOLO architecture, and a detection head of the target detection network receives processed multi-scale feature maps; the multi-scale feature maps at least comprise feature maps of three scales of P3, P4 and P5; the detection head outputs preliminary detection results through anchor box matching, boundary box regression and category prediction, and eliminates redundant detection boxes through confidence sorting and non-maximum suppression processing to output final position information, confidence and category information of green algae patches.

7. A high-efficiency monitoring system for marine seaweed using unmanned aerial vehicles (UAVs) with multimodal data fusion, characterized in that: The method comprises the following steps: an image acquisition module is configured to acquire natural optical images and infrared images of marine green algae collected by a UAV; The feature coding module is configured to input the natural optical image into a first backbone network for feature coding to obtain an optical modality initial feature map; and input the infrared image into a second backbone network for feature coding to obtain an infrared modality initial feature map. The feature fusion and enhancement module is configured to perform cross-modality fusion and enhancement processing on the optical modality initial feature map and the infrared modality initial feature map to obtain an enhanced fusion feature map. The target detection module is configured to input the enhanced fusion feature map into a target detection network to output position information, confidence and category information of the Enteromorpha prolifera patch.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor executes the program to implement the steps of the multi-modal data fusion unmanned aerial vehicle offshore Enteromorpha prolifera efficient monitoring method according to any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps of the multi-modal data fusion unmanned aerial vehicle offshore Enteromorpha prolifera efficient monitoring method according to any one of claims 1 to 6.

10. A computer program product comprising software code, characterized in that, The program in the software code performs the steps of the multi-modal data fusion unmanned aerial vehicle offshore Enteromorpha prolifera efficient monitoring method according to any one of claims 1 to 6.

Images