Complex sea condition small target detection method based on multi-modal fusion

Through the combination of adaptive-Laplace fusion and U-Net subnet, the problems of insufficient multimodal data fusion, small object detection difficulties and poor environmental adaptability in maritime target detection are solved, and high accuracy and real-time are achieved , suitable for offshore target detection in complex sea conditions.

CN118941944AActive Publication Date: 2024-11-12NAT INNOVATION INST OF DEFENSE TECH PLA ACAD OF MILITARY SCI

Patent Information

Application Number
CN202410932274.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-12
Publication Date
2024-11-12
Estimated Expiration
2044-07-12

AI Technical Summary

Technical Problem

The existing offshore target detection technology has defects in insufficient multimodal data fusion, difficulty in small target detection, poor environmental adaptability and insufficient real-time performance, especially in complex sea conditions, it is difficult to achieve high accuracy and rapid response.

Method used

Adaptive-Laplace fusion strategy is adopted to fuse infrared, visible light and hyperspectral images, combined with the step-by-step feature fusion mechanism of the U-Net subnet and the dynamic label calibration mechanism to improve the accuracy and real-timeness of object detection.

Benefits of technology

It significantly improves the accuracy of small-object detection and environmental adaptability in complex sea conditions, reduces calculation costs, and can work stably at night or inclement weather conditions to meet the needs of real-time or near-real-time detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941944A_ABST
    Figure CN118941944A_ABST
Patent Text Reader

Abstract

The invention provides a complex sea condition small target detection method based on multi-modal fusion, and belongs to the technical field of sea target detection. The method comprises the following steps: S1, in an image fusion stage, fusing an infrared image, a visible light image and a hyperspectral image into a self-adaptive image based on a self-adaptive-Laplacian fusion strategy; s2, in a target detection stage, a feature fusion layer is calculated through a down-sampling layer of an encoder and a convolutional layer of a dense block, and step-by-step feature fusion for the adaptive image is realized; s3, in a target detection stage, performing target detection on the fused features through a target detection model; wherein other pixels which do not intersect with the positive pixel of the positive label are excluded, and mistakenly detected pixels in the background are excluded from the target detection result; step S4, executing average weighted summation between pixels to update labels, and gradually expanding point labels to improve the detection accuracy; and S5, outputting the generated detection image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of marine target detection, and in particular relates to a small target detection method in complex marine conditions based on multimodal fusion. Background Art

[0002] Maritime target detection technology plays a vital role in the marine monitoring system. It is not only related to maritime traffic safety, but also involves the protection of marine resources, the monitoring of maritime boundaries, and the monitoring of the marine environment. However, the complexity of the marine environment has brought many challenges to maritime target detection.

[0003] In order to achieve effective marine target detection, researchers have developed a variety of technical means. Radar technology detects targets by emitting electromagnetic waves and receiving reflected signals. It has the advantage of long-distance detection, but the resolution is usually low and it is easily affected by weather conditions such as rain and fog. Optical imaging systems can provide high-resolution images, so that details such as the shape, color and texture of the target can be clearly displayed. However, the effect of optical imaging is limited in poor lighting conditions or extreme climates. Sonar mainly detects underwater targets, such as submarines, by the propagation of sound waves in water, and the detection of surface targets is relatively limited. Although large-scale ocean monitoring using satellite-borne sensors can cover a wide range of sea areas, it is usually not as real-time and resolution as offshore monitoring methods due to the limitations of satellite orbits and sensor performance. In order to integrate the advantages of these technologies and overcome their respective shortcomings, multimodal data fusion technology has emerged. By fusing these different modal data, the shortcomings of a single sensor can be made up.

[0004] In recent years, object detection based on infrared, visible light, and hyperspectral images has gradually become a research hotspot and has achieved a series of innovative results. In the early days, object detection mainly relied on manual feature extraction and traditional machine learning algorithms such as support vector machines (SVMs) and random forests. These methods require expertise to design features and often have limitations in detection performance. Over time, the emergence of deep learning has greatly promoted the advancement of object detection technology. Deep learning models, especially convolutional neural networks (CNNs), have become mainstream due to their superior performance in image recognition and classification tasks.

[0005] However, there are still certain defects and challenges in the existing technology. (1) Insufficient fusion of multimodal data. The existing technology often relies on single-modal data and fails to effectively integrate the advantages of different sensor data, resulting in limited target detection performance in complex sea conditions. (2) Small target detection is difficult. Small target detection in the marine environment faces more challenges, such as small target size and weak signal. Existing technology is difficult to accurately identify and locate these small targets. (3) Poor environmental adaptability. The marine environment is changeable, including changes in lighting, wave fluctuations, etc. The detection efficiency and accuracy of existing technologies at night or in severe weather conditions are insufficient. (4) Insufficient real-time performance. In application scenarios that require rapid response, the computational cost of existing technologies is high and cannot meet real-time or near real-time detection requirements. Summary of the invention

[0006] In response to the above technical problems, the present invention proposes a small target detection solution in complex sea conditions based on multimodal fusion.

[0007] The first aspect of the present invention discloses a method for detecting small targets in complex sea conditions based on multimodal fusion, the method comprising:

[0008] Step S1: In the image fusion stage, based on the adaptive-Laplacian fusion strategy, the infrared image, the visible light image, and the hyperspectral image are fused into an adaptive image;

[0009] Step S2: In the target detection stage, the feature fusion layer is calculated through the downsampling layer of the encoder and the convolution layer of the dense block to achieve step-by-step feature fusion for the adaptive image;

[0010] Step S3, in the target detection stage, target detection is performed on the fused features through the target detection model; wherein other pixels that have no intersection with the positive pixels of the positive label are excluded, and the misdetected pixels in the background are excluded from the target detection results;

[0011] Step S4, performing average weighted summation between pixels to update labels, and gradually expanding point labels to improve detection accuracy;

[0012] Step S5: output the generated detection image.

[0013] According to the method of the first aspect of the present invention, in step S1:

[0014] In the first stage of integration:

[0015] Adaptively fuse the infrared image, visible light image, and hyperspectral image to form a mixed image data set, which is used to integrate the basic information of the three images and form a first fused image;

[0016] At the same time, the three images are fused using the Laplacian pyramid to retain high-frequency detail information and form a second fused image;

[0017] In the second stage of fusion, the first fused image and the second fused image are adaptively fused again to obtain the adaptive image.

[0018] According to the method of the first aspect of the present invention, in step S1, the adaptive-Laplacian fusion process is:

[0019]

[0020] Among them, I IR ,I VIS ,I HS Respectively represent infrared image, visible light image, and hyperspectral image, w IR 、w VIS 、w HS are the weights in the first stage of adaptive fusion process, w I ' R , w′ VIS , w′ HS are the weights in the first stage of Laplace fusion, w A 、w L are their weights in the second stage adaptive fusion process, N represents the number of levels of the Laplacian pyramid, They are the detail images of the kth layer of the Laplacian pyramid for infrared, visible light and hyperspectral images respectively.

[0021] According to the method of the first aspect of the present invention, in step S2, a U-Net sub-network based on a step-by-step feature fusion mechanism is used to perform step-by-step feature fusion for an adaptive image; wherein the U-Net sub-network based on the step-by-step feature fusion mechanism aggregates feature information of different sizes through multiple interactions, thereby achieving step-by-step feature fusion, and the process is as follows:

[0022]

[0023] Among them, L i,j represents the output of the jth convolutional layer along the i-th downsampling layer of the encoder and the dense block along the plain skip path, F represents multiple cascaded convolutional layers, and P max Represents a maximum pooling layer.

[0024] According to the method of the first aspect of the present invention, in step S3, target detection is performed using a target detection model based on a dynamic label calibration mechanism; wherein: the dynamic label calibration mechanism is used to exclude other pixels that have no intersection with positive pixels of positive labels, and exclude misdetected pixels in the background from the target detection result; in step S4, an average weighted summation between pixels is performed simultaneously to achieve label update, and the point labels are gradually expanded by gradually updating the labels to improve the accuracy of detection; the above process is:

[0025]

[0026] Where ⊙ represents element-by-element multiplication, h and w are the height and width of the input image, r is set to 0.15%, and T b is the minimum threshold, k is used to control the threshold growth rate, It is for The mask, is the updated label in round n, which is the new supervision for training in round n+1.

[0027] The second aspect of the present invention discloses a small target detection system in complex sea conditions based on multimodal fusion, the system comprising a processing unit, the processing unit being configured to execute:

[0028] In the image fusion stage, infrared images, visible light images, and hyperspectral images are fused into adaptive images based on the adaptive-Laplacian fusion strategy;

[0029] In the target detection stage, the feature fusion layer is calculated through the downsampling layer of the encoder and the convolution layer of the dense block to achieve step-by-step feature fusion for adaptive images;

[0030] In the target detection stage, the fused features are used for target detection through the target detection model; among them, other pixels that have no intersection with the positive pixels of the positive label are excluded, and the misdetected pixels in the background are excluded from the target detection results;

[0031] Perform average weighted summation between pixels to update labels and gradually expand point labels to improve detection accuracy;

[0032] Output the generated detection image.

[0033] The third aspect of the present invention discloses an electronic device. The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the method for detecting small targets in complex sea conditions based on multimodal fusion described in the first aspect of the present disclosure is implemented.

[0034] The fourth aspect of the present invention discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for detecting small targets in complex sea conditions based on multimodal fusion as described in the first aspect of the present disclosure.

[0035] In summary, the present invention utilizes the adaptive-Laplacian fusion strategy to optimize the integration process of different modal data, improve the quality of the fused image, and enhance the performance of target detection. The structure based on the U-Net subnetwork is adopted, and the extraction and representation of small target features are strengthened through the step-by-step feature fusion mechanism, thereby improving the detection accuracy. A dynamic label calibration mechanism is proposed to optimize the label update during the training process, improve the model's ability to recognize small targets, especially the detection performance in complex backgrounds. Through model design, the real-time performance of the system is improved, so that it can adapt to the needs of marine target detection at night or in severe weather conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 A schematic diagram of a small target detection framework in complex sea conditions based on multimodal fusion according to an embodiment of the present invention;

[0038] Figure 2 The figure is a flow chart of a method for detecting small targets in complex sea conditions based on multimodal fusion according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0040] In view of the shortcomings of the existing technology, such as the insufficiency of multimodal data fusion, the challenge of small target detection, poor environmental adaptability and real-time limitations, the present invention proposes a marine target detection method based on multimodal image fusion.

[0041] The above method aims to significantly enhance the feature representation capability through a step-by-step feature fusion mechanism and a dynamic label calibration mechanism, thereby greatly improving the accuracy of target detection. Infrared, visible light, and hyperspectral images are used as input data, and the three input images are cleverly fused through an adaptive-Laplacian fusion strategy to generate a comprehensive image that is both rich in details and highly consistent overall. This fusion strategy not only retains the key information in the image, but also enhances the quality and usability of the image. Furthermore, the present invention also proposes a U-Net subnetwork based on a step-by-step feature fusion mechanism, which can efficiently aggregate feature information from different scales, thereby achieving more accurate feature extraction and target positioning. In addition, the present invention also proposes a dynamic label calibration mechanism, which can gradually expand the point labels of the target by gradually updating the labels, and can effectively improve the accuracy and reliability of detection.

[0042] The technical solution proposed in the present invention is not only technically innovative, but also has wide applicability and significant advantages in practical applications, and can provide strong technical support for fields such as maritime safety monitoring, marine resource management and maritime rescue.

[0043] The first aspect of the present invention discloses a method for detecting small targets in complex sea conditions based on multimodal fusion. Figure 1-2 As shown, the method includes:

[0044] Step S1: In the image fusion stage, based on the adaptive-Laplacian fusion strategy, the infrared image, the visible light image, and the hyperspectral image are fused into an adaptive image;

[0045] Step S2: In the target detection stage, the feature fusion layer is calculated through the downsampling layer of the encoder and the convolution layer of the dense block to achieve step-by-step feature fusion for the adaptive image;

[0046] Step S3, in the target detection stage, target detection is performed on the fused features through the target detection model; wherein other pixels that have no intersection with the positive pixels of the positive label are excluded, and the misdetected pixels in the background are excluded from the target detection results;

[0047] Step S4, performing average weighted summation between pixels to update labels, and gradually expanding point labels to improve detection accuracy;

[0048] Step S5: output the generated detection image.

[0049] According to the method of the first aspect of the present invention, in step S1:

[0050] In the first stage of integration:

[0051] Adaptively fuse the infrared image, visible light image, and hyperspectral image to form a mixed image data set, which is used to integrate the basic information of the three images and form a first fused image;

[0052] At the same time, the three images are fused using the Laplacian pyramid to retain high-frequency detail information and form a second fused image;

[0053] In the second stage of fusion, the first fused image and the second fused image are adaptively fused again to obtain the adaptive image.

[0054] According to the method of the first aspect of the present invention, in step S1, the adaptive-Laplacian fusion process is:

[0055]

[0056] Among them, I IR ,I VIS ,I HS Respectively represent infrared image, visible light image, and hyperspectral image, w IR 、w VIS 、w HS are the weights in the first stage of adaptive fusion, w′ IR , w′ VIS , w′ HS are the weights in the first stage of Laplace fusion, w A 、w L are their weights in the second stage adaptive fusion process, N represents the number of levels of the Laplacian pyramid, They are the detail images of the kth layer of the Laplacian pyramid for infrared, visible light and hyperspectral images respectively.

[0057] According to the method of the first aspect of the present invention, in step S2, a U-Net sub-network based on a step-by-step feature fusion mechanism is used to perform step-by-step feature fusion for an adaptive image; wherein the U-Net sub-network based on the step-by-step feature fusion mechanism aggregates feature information of different sizes through multiple interactions, thereby achieving step-by-step feature fusion, and the process is as follows:

[0058]

[0059] Among them, L i,j represents the output of the jth convolutional layer along the i-th downsampling layer of the encoder and the dense block along the plain skip path, F represents multiple cascaded convolutional layers, and P max Represents a maximum pooling layer.

[0060] According to the method of the first aspect of the present invention, in step S3, target detection is performed using a target detection model based on a dynamic label calibration mechanism; wherein: the dynamic label calibration mechanism is used to exclude other pixels that have no intersection with positive pixels of positive labels, and exclude misdetected pixels in the background from the target detection result; in step S4, an average weighted summation between pixels is performed simultaneously to achieve label update, and the point labels are gradually expanded by gradually updating the labels to improve the accuracy of detection; the above process is:

[0061]

[0062]

[0063] Where ⊙ represents element-by-element multiplication, h and w are the height and width of the input image, r is set to 0.15%, and T b is the minimum threshold, k is used to control the threshold growth rate, It is for The mask, is the updated label in round n, which is the new supervision for training in round n+1.

[0064] First embodiment

[0065] Aiming at the special needs of marine target detection, the present invention proposes a marine target detection method based on multimodal image fusion, which can effectively improve the detection accuracy and efficiency of marine targets. The method is divided into two parts: multimodal image fusion and marine target detection.

[0066] In the multimodal image fusion stage, an innovative two-stage fusion strategy is proposed: the adaptive-Laplacian fusion strategy. In the first stage of fusion, the infrared image, visible light image and hyperspectral image are adaptively fused to form a mixed image data set, which quickly integrates the basic information of the three images. At the same time, the three images are fused using the Laplacian pyramid to retain more high-frequency detail information. In the second stage of fusion, the two fused images generated in the first stage are adaptively fused again to obtain a final image that is both rich in details and has overall consistency. Adaptive-Laplacian fusion can be expressed as:

[0067]

[0068] Among them, I IR ,I VIS ,I HS Respectively represent infrared image, visible light image, and hyperspectral image, w IR 、w VIS 、w HS are the weights in the first stage of adaptive fusion, w′ IR , w′ VIS, w′ HS are the weights in the first stage of Laplace fusion, w A 、w L are their weights in the second stage adaptive fusion process, N represents the number of levels of the Laplacian pyramid, They are the detail images of the kth layer of the Laplacian pyramid for infrared, visible light and hyperspectral images respectively.

[0069] In the target detection stage, a U-net subnetwork is proposed. Based on U-Net, this network aggregates feature information from different scales through multiple interactions to achieve step-by-step feature fusion. This aggregation method enhances feature representation and improves the accuracy of target detection. The calculation process can be expressed as:

[0070]

[0071] Among them, L i,j Represents node L i,j Where i is the Ith downsampling layer along the encoder, j is the jth convolutional layer of the dense block along the plain skip path, F represents multiple cascaded convolutional layers, P max Represents a maximum pooling layer.

[0072] Based on this network, this patent proposes a dynamic label calibration mechanism. This mechanism excludes other pixels that have no intersection with the positive pixels of the positive label, excludes the misdetected pixels in the background from the target detection results, and reduces false positives. The calculation process is shown in the formula:

[0073]

[0074] At the same time, perform the average weighted summation between pixels to achieve label update. By gradually updating the label, the point label can be gradually expanded to improve the accuracy of detection. The calculation process is shown in the formula:

[0075]

[0076] ⊙ represents element-by-element multiplication, h and w are the height and width of the input image, r is 0.15%, T b is the minimum threshold, k controls the threshold growth rate, for The mask, is the updated label in round n, which is used as the new supervision for training in round n+1.

[0077] Second embodiment

[0078] The marine target detection technology proposed in this invention not only fully utilizes the information in multimodal data through the adaptive-Laplacian fusion method, but also significantly improves the overall performance of marine target detection technology through the step-by-step feature fusion mechanism and dynamic label calibration mechanism. The specific process is as follows (combined with Figure 2 shown):

[0079] Step 1: The infrared image, visible light image and hyperspectral image are fused into an adaptive image through the adaptive Laplace fusion strategy.

[0080] Step 2: Through the downsampling layer of the encoder and the convolution layer of the dense block, the feature fusion layer is calculated to achieve step-by-step feature fusion.

[0081] Step 3: Exclude other pixels that have no intersection with the positive pixels of the positive label, and exclude the misdetected pixels in the background from the target detection results.

[0082] Step 4: Calculate the average weighted sum between pixels to update the label, gradually expand the point label, and improve the detection accuracy.

[0083] Step 5: Output the generated detection image.

[0084] It can be seen that the marine target detection method based on multimodal image fusion has shown significant technical advantages compared with the existing technology. First, this method effectively integrates the information of infrared, visible light and hyperspectral images through the adaptive-Laplacian fusion strategy, which greatly improves the detection accuracy of small targets in complex sea conditions. Secondly, this technical solution has excellent environmental adaptability and can work stably at night or in severe weather conditions, ensuring the continuity and reliability of detection. In addition, this patent reduces the dependence on a large amount of labeled data, reduces the cost of data collection and processing, and simplifies system deployment and operation procedures. The introduction of the step-by-step feature fusion mechanism and the dynamic label calibration mechanism further enhances the robustness of the system and ensures stable operation in a changing marine environment. By providing accurate marine target detection, this patent provides strong technical support for marine safety monitoring and maritime search and rescue operations, greatly improves the rescue efficiency, and has important social and economic value.

[0085] The second aspect of the present invention discloses a small target detection system in complex sea conditions based on multimodal fusion, the system comprising a processing unit, the processing unit being configured to execute:

[0086] In the image fusion stage, infrared images, visible light images, and hyperspectral images are fused into adaptive images based on the adaptive-Laplacian fusion strategy;

[0087] In the target detection stage, the feature fusion layer is calculated through the downsampling layer of the encoder and the convolution layer of the dense block to achieve step-by-step feature fusion for adaptive images;

[0088] In the target detection stage, the fused features are used for target detection through the target detection model; among them, other pixels that have no intersection with the positive pixels of the positive label are excluded, and the misdetected pixels in the background are excluded from the target detection results;

[0089] Perform average weighted summation between pixels to update labels and gradually expand point labels to improve detection accuracy;

[0090] Output the generated detection image.

[0091] The third aspect of the present invention discloses an electronic device. The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the method for detecting small targets in complex sea conditions based on multimodal fusion described in the first aspect of the present disclosure is implemented.

[0092] The fourth aspect of the present invention discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for detecting small targets in complex sea conditions based on multimodal fusion as described in the first aspect of the present disclosure.

[0093] In summary, the present invention utilizes the adaptive-Laplacian fusion strategy to optimize the integration process of different modal data, improve the quality of the fused image, and enhance the performance of target detection. The structure based on the U-Net subnetwork is adopted, and the extraction and representation of small target features are strengthened through the step-by-step feature fusion mechanism, thereby improving the detection accuracy. A dynamic label calibration mechanism is proposed to optimize the label update during the training process, improve the model's ability to recognize small targets, especially the detection performance in complex backgrounds. Through model design, the real-time performance of the system is improved, so that it can adapt to the needs of marine target detection at night or in severe weather conditions.

[0094] Please note that the technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. The above-mentioned embodiments only express several implementation methods of the present application, and their descriptions are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, without departing from the concept of the present application, several variations and improvements can be made, which all belong to the scope of protection of the present application. Therefore, the scope of protection of the patent in this application shall be based on the attached claims.

Claims

1. A small target detection method in complex sea conditions based on multimodal fusion, characterized in that: The method comprises: Step S1: In the image fusion stage, based on the adaptive-Laplacian fusion strategy, the infrared image, the visible light image, and the hyperspectral image are fused into an adaptive image; Step S2: In the target detection stage, the feature fusion layer is calculated through the downsampling layer of the encoder and the convolution layer of the dense block to achieve step-by-step feature fusion for the adaptive image; Step S3, in the target detection stage, target detection is performed on the fused features through the target detection model; wherein other pixels that have no intersection with the positive pixels of the positive label are excluded, and the misdetected pixels in the background are excluded from the target detection results; Step S4, performing average weighted summation between pixels to update labels, and gradually expanding point labels to improve detection accuracy; Step S5: output the generated detection image.

2. According to claim 1, a method for detecting small targets in complex sea conditions based on multimodal fusion is characterized in that: In step S1: In the first stage of integration: Adaptively fuse the infrared image, visible light image, and hyperspectral image to form a mixed image data set, which is used to integrate the basic information of the three images and form a first fused image; At the same time, the three images are fused using the Laplacian pyramid to retain high-frequency detail information and form a second fused image; In the second stage of fusion, the first fused image and the second fused image are adaptively fused again to obtain the adaptive image.

3. The method for detecting small targets in complex sea conditions based on multimodal fusion according to claim 2 is characterized in that: In step S1, the adaptive-Laplacian fusion process is: Among them, I IR ,I VIS ,I HS Respectively represent infrared image, visible light image, and hyperspectral image, w IR 、w VIS 、w HS are the weights in the first stage of adaptive fusion process, w I ' R , w′ VIS , w′ HS are the weights in the first stage of Laplace fusion process, w A 、w L are their weights in the second stage adaptive fusion process, N represents the number of levels of the Laplacian pyramid, They are the detail images of the kth layer of the Laplacian pyramid for infrared, visible light and hyperspectral images respectively.

4. The method for detecting small targets in complex sea conditions based on multimodal fusion according to claim 3 is characterized in that: In step S2, a U-Net subnetwork based on a step-by-step feature fusion mechanism is used to perform step-by-step feature fusion for the adaptive image; wherein the U-Net subnetwork based on the step-by-step feature fusion mechanism aggregates feature information of different sizes through multiple interactions, thereby achieving step-by-step feature fusion, and the process is as follows: Among them, L i,j represents the output of the jth convolutional layer along the i-th downsampling layer of the encoder and the dense block along the plain skip path, F represents multiple cascaded convolutional layers, and P max Represents a maximum pooling layer.

5. The method for detecting small targets in complex sea conditions based on multimodal fusion according to claim 4, characterized in that: In step S3, target detection is performed using a target detection model based on a dynamic label calibration mechanism; wherein: the dynamic label calibration mechanism is used to exclude other pixels that have no intersection with the positive pixels of the positive label, and exclude misdetected pixels in the background from the target detection result; In step S4, the average weighted summation between pixels is performed simultaneously to achieve label update, and the point labels are gradually expanded by gradually updating the labels to improve the accuracy of detection; The above process is: Where ⊙ represents element-by-element multiplication, h and w are the height and width of the input image, r is set to 0.15%, and T b is the minimum threshold, k is used to control the threshold growth rate, It is for The mask, is the updated label in round n, which is the new supervision for training in round n+1.

6. A small target detection system in complex sea conditions based on multimodal fusion, characterized in that: The system comprises a processing unit configured to perform: In the image fusion stage, infrared images, visible light images, and hyperspectral images are fused into adaptive images based on the adaptive-Laplacian fusion strategy; In the target detection stage, the feature fusion layer is calculated through the downsampling layer of the encoder and the convolution layer of the dense block to achieve step-by-step feature fusion for adaptive images; In the target detection stage, the fused features are used for target detection through the target detection model; among them, other pixels that have no intersection with the positive pixels of the positive label are excluded, and the misdetected pixels in the background are excluded from the target detection results; Perform an average weighted sum across pixels to update labels and gradually expand point labels to improve detection accuracy; Output the generated detection image.

7. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the method for detecting small targets in complex sea conditions based on multimodal fusion as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, it implements the method for detecting small targets in complex sea conditions based on multimodal fusion as described in any one of claims 1 to 5.

Citation Information

Patent Citations

  • Infrared and visible light image fusion method based on multi-mode features

    CN114639002A

  • Infrared small target detection method and device based on data enhancement

    CN117409192A

  • Salient target detection method based on infrared and visible light image fusion

    CN117935006A

  • Infrared and visible light fusion method based on double-layer optimization

    WO2021120408A1

Cited By

  • Multispectral imaging underwater high-speed image sensor and imaging method

    CN120091232A