Sea surface floating small target detection method based on multi-modal image fusion

Abstract

The invention relates to a sea surface floating small target detection method based on multi-modal image fusion. Comprising the following steps: acquiring a plurality of image pairs, preprocessing the acquired image pairs, and constructing an image sample data set by using the preprocessed image pairs; enhancing the contrast and detail information of the infrared polarization image by using an optimization algorithm; extracting low-pass sub-band and band-pass directional sub-band information of different modal images by using multi-scale transformation, determining fusion weights of the two parts according to the three-channel simplified pulse coupling neural network, and finally obtaining a final fusion result by using inverse transformation of the multi-scale transformation; and inputting the three-mode fusion image into a fixed detection network to obtain a target detection result. According to the method, the relative effect of the infrared polarization, the infrared intensity and the visible light image is effectively measured, the contrast and detail information of the infrared polarization image are effectively enhanced, the fusion feature is enhanced, and better fusion image quality and target detection performance are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and target detection, and in particular to a method for detecting small floating targets on the sea surface based on multimodal image fusion. Background Art

[0002] In recent years, traditional single-modal imaging technology has been unable to meet the increasingly complex application requirements. With the rapid development of hardware, multimodal imaging technology has been widely used in monitoring, autonomous driving, target detection and other fields. In particular, the combination of infrared images and visible light images has shown significant advantages in improving the processing effect of subsequent tasks. Infrared detectors can capture scene information that is difficult to detect under conventional visible light by detecting the thermal radiation of objects, which makes them perform well in dark or obstructed environments. However, due to the low pixel density of infrared detectors and the limitations of the optical system, infrared images have low resolution, poor image quality and lack of sufficient texture details. Visible light images have rich scene detail information, but their imaging effects are easily affected by environmental factors such as obstructions and bad weather.

[0003] There is a close connection between image fusion and other advanced computer vision tasks: as a technique for input enhancement, image fusion can significantly improve the accuracy of other computer vision tasks. In addition, the preprocessing effect of image fusion in tasks such as multi-view feature matching, pedestrian semantic segmentation, target detection and tracking has been verified, indicating that it has effective application value in these fields.

[0004] Although existing target detection methods based on infrared and visible light image fusion can effectively integrate the common and complementary information in infrared and visible light images, they usually rely only on intensity information and ignore the key information from light polarization. This limits the performance of image fusion in certain scenes or materials. Polarization is an inherent property of light and provides unique information about the target, such as surface smoothness, 3D normals, and material composition. Due to the differences in surface materials, roughness, and physicochemical properties of target objects, they exhibit significant infrared polarization characteristics, which can effectively identify artificial targets hidden in natural backgrounds. Although existing methods have achieved remarkable results, there are still some challenges in the fusion process of infrared polarization, infrared intensity, and visible light trimodal images. First, infrared polarization images are usually acquired under low radiation conditions and have weak signals, which is not conducive to retaining significant target information from the image during the fusion process. Second, due to the differences in the salient parts of multimodal features themselves, existing methods are difficult to ensure the actual effect of interactive fusion of multimodal features. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a method for detecting small floating targets on the sea surface based on multimodal image fusion, which effectively measures the relative effects of infrared polarization, infrared intensity and visible light images, effectively enhances the contrast and detail information of infrared polarization images, and fully considers the information between infrared polarization, infrared intensity and visible light images and the differences in prominent areas during feature fusion, so as to enhance the fused features and have better fused image quality and target detection performance.

[0006] To achieve the above object, the present invention adopts the following technical solution: a method for detecting small floating targets on the sea surface based on multimodal image fusion, comprising the following steps:

[0007] Step S1: obtaining a plurality of image pairs, preprocessing the obtained image pairs, and constructing an image sample data set using the preprocessed image pairs;

[0008] Step S2: using an optimization algorithm to enhance the contrast and detail information of the infrared polarization image;

[0009] Step S3: Use multi-scale transform to extract low-pass subband and band-pass directional subband information of different modal images, determine the fusion weights of the two parts according to the three-channel simplified pulse coupled neural network, and fuse the low-pass subband and the band-pass directional subband respectively, and finally use the inverse transform of the multi-scale transform to obtain the final fusion result;

[0010] Step S4: Input the trimodal fusion image into a fixed detection network to obtain the target detection result.

[0011] In a preferred embodiment, the specific steps of step S1 are:

[0012] Step S11: synchronously collecting infrared polarization, infrared intensity and visible light data sets, using a collection system equipped with an infrared polarization camera and a visible light camera to collect images, and the collected infrared polarization, infrared intensity and visible light images contain a variety of scene information;

[0013] Step S12: preprocessing of the infrared polarization, infrared intensity and visible light data sets includes performing distortion correction operations on the infrared polarization, infrared intensity and visible light images; and performing registration operations on the infrared polarization, infrared intensity and visible light images in the image pair;

[0014] Step S13: construct a sample data set using the pre-processed infrared polarization, infrared intensity and visible light images of different scenes.

[0015] In a preferred embodiment, the specific process of enhancing the contrast and detail information of the infrared polarization image is as follows:

[0016] Step S21: using a particle swarm optimization algorithm to adjust the calculation formula of infrared linear polarization degree to determine the parameters that need to be optimized;

[0017] Step S22: constructing the fitness function of the particle swarm optimization algorithm, setting the initial parameters of the optimization algorithm, and iteratively optimizing the parameters to be optimized in step S21;

[0018] Step S23: using the optimal parameters to obtain an infrared polarization image with enhanced contrast and detail information.

[0019] In a preferred embodiment, the specific process of image fusion is as follows:

[0020] Step S31: using non-subsampled contourlet transform to decompose the enhanced infrared polarization image, infrared intensity image and visible light image respectively to obtain two parts: a low-pass sub-band and a band-pass directional sub-band;

[0021] Step S32: determining the fusion weights of the two parts according to the three-channel simplified pulse coupled neural network, the multi-scale morphological gradient and the fractal dimension;

[0022] Step S33: using the fusion weights of step S32, respectively fuse the low-pass sub-band and the band-pass directional sub-band;

[0023] Step S34: Obtain the final tri-modal image fusion result through inverse transformation of non-subsampled contourlet transform.

[0024] In a preferred embodiment, the fixed detection network adopts Yolov5 network.

[0025] In a preferred embodiment, the calculation formula of the infrared linear polarization degree is:

[0026]

[0027] Among them, S0 is the total light intensity; S1 is the linear polarization component in a specific direction between the horizontal and vertical directions; S2 is the linear polarization component in another specific direction perpendicular to the direction of S1; and They represent S1 to the power of a and S2 to the power of b respectively; the parameters that need to be optimized are the powers of S1 and S2 and the value of the coefficient before S0, i.e. a, b and c in the calculation formula.

[0028] In a preferred embodiment, the fitness function of the optimization algorithm is expressed as:

[0029] fitness=-(PSNR+EN)

[0030] Among them, PSNR represents the peak signal-to-noise ratio between the infrared polarization image and the infrared intensity image calculated by optimizing parameters; EN represents the information entropy of the infrared polarization image calculated by optimizing parameters.

[0031] In a preferred embodiment, the specific steps of the three-channel simplified pulse coupled neural network are:

[0032] Inputting the multi-modal low-pass sub-band and the band-pass directional sub-band into an input and modulation module, receiving external stimulation and stimulation from surrounding neurons through dendrites, wherein the multi-modal low-pass sub-band and the band-pass directional sub-band serve as external stimulation;

[0033] The fusion information area measures the activity level of neurons and integrates it to obtain a fused image;

[0034] The pulse generating section generates output pulses.

[0035] In a preferred embodiment, the expression of the multi-scale morphological gradient is:

[0036]

[0037] Among them, X represents different modes, m is the number of scales, G t (i, j) is the morphological gradient of X at scale t at the (i, j)th pixel, W t =1 / (2t+1) is G t The weight of .

[0038] In a preferred embodiment, the expression of the fractal dimension is:

[0039]

[0040] Among them, X represents different modes, g max and g min are the minimum and maximum grayscale of the respective style.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] 1. The present invention aims to solve the problem that the infrared polarization image has low contrast, which makes the detail information unclear and difficult to be used for fusion. The calculation formula of infrared linear polarization degree is improved through an optimization algorithm to better retain the unique information of infrared polarization image during the fusion process.

[0043] 2. The present invention proposes a three-modal image fusion method combining infrared polarization, infrared intensity and visible light images of infrared polarization images to better retain the common and complementary information from different modal images of the same scene and improve the target detection performance;

[0044] 3. The present invention conducts in-depth mining of feature relationships, avoids insufficient fusion of features of different modalities, and refines and enhances unfinished fusion features, thereby providing better scene representation capabilities and improving target detection performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is an overall flow chart of a target detection method according to an embodiment of the present invention;

[0046] Figure 2 is a comparison diagram of an original infrared polarization image and an enhanced infrared polarization image according to an embodiment of the present invention;

[0047] Figure 3 Schematic diagram of a deep channel attention module according to an embodiment of the present invention;

[0048] Figure 4 It is a schematic diagram of a three-channel simplified pulse coupled neural network according to an embodiment of the present invention;

[0049] Figure 5 A comparison diagram of the fusion effect between the image fusion method of an embodiment of the present invention and the existing method;

[0050] Figure 6 The figure is a comparison chart of the target detection result of the embodiment of the present invention and the detection result of the existing method. DETAILED DESCRIPTION

[0051] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0052] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0053] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0054] like Figure 1-6 As shown, the present invention provides a method for detecting small floating targets on the sea surface based on multimodal image fusion, the method comprising:

[0055] S1: obtaining multiple pairs of images, preprocessing the obtained image pairs, and constructing an image sample data set using the preprocessed image pairs;

[0056] Specifically, the data set used in the present invention is a data set collected by a synchronous acquisition system equipped with an infrared polarization camera and a visible light camera, including infrared polarization images, infrared intensity images and visible light images under various environments. The images in the data set are subjected to distortion correction, and then the images in the data set are registered using the RIFT method. Finally, the infrared polarization images, infrared intensity images and visible light images are normalized to form a sample data set, thereby completing the preprocessing of the data set.

[0057] S2: Use optimization algorithm to enhance the contrast and detail information of infrared polarization images;

[0058] Specifically, the specific process of enhancing infrared polarization images in the present invention is as follows:

[0059] S21: using a particle swarm optimization algorithm to adjust the calculation formula of infrared linear polarization degree and determine the parameters that need to be optimized;

[0060] Specifically, the parameters that need to be optimized in the linear polarization degree of the present invention are as follows:

[0061]

[0062] Among them, S0 is the total light intensity; S1 is the linear polarization component in a specific direction between the horizontal and vertical directions; S2 is the linear polarization component in another specific direction perpendicular to the direction of S1; the parameters that need to be optimized are the powers of S1 and S2 and the value of S0, that is, a, b and c in the calculation formula.

[0063] S22: construct the fitness function of the particle swarm optimization algorithm, set the initial parameters of the optimization algorithm, and iteratively optimize the parameters that need to be optimized in S21;

[0064] Specifically, the fitness function of the optimization algorithm used in the present invention is expressed as:

[0065] f itness =-(PSNR+EN)

[0066] Among them, PSNR represents the peak signal-to-noise ratio between the infrared polarization image and the infrared intensity image calculated by optimizing parameters; EN represents the information entropy of the infrared polarization image calculated by optimizing parameters;

[0067] Specifically, the initial parameters of the optimization algorithm used in the present invention are:

[0068] Inertia weight 0.2, individual learning factor 2, group learning factor 2, particle swarm size 40, particle dimension 3, maximum number of iterations 30, position parameter limits a = [2, 10], b = [2, 10] and c = [0.1, 10] (where a and b are integers and c is a decimal place) and speed parameter limit v∈[-1, 1].

[0069] S23: Using the optimal parameters, an infrared polarization image with enhanced contrast and detail information is obtained;

[0070] Specifically, the original infrared polarization image and the enhanced infrared polarization image, such as Figure 2 As shown in the figure, it can be seen that the enhanced infrared polarization image has more obvious contour details than the original image, and the polarization information is enhanced.

[0071] S3: Use multi-scale transform to extract the low-pass subband and band-pass directional subband information of different modal images, determine the fusion weights of the two parts according to the three-channel simplified pulse coupled neural network, and fuse the low-pass subband and band-pass directional subband respectively. Finally, use the inverse transform of the multi-scale transform to obtain the final fusion result;

[0072] Specifically, the overall framework of the image fusion method constructed by the present invention is as follows: Figure 3 As shown, the specific method flow is:

[0073] S31: using non-subsampled contourlet transform to decompose the enhanced infrared polarization image, infrared intensity image and visible light image respectively, and obtain two parts: low-pass sub-band and band-pass directional sub-band;

[0074] S32: Determine the fusion weights of the two parts based on the three-channel simplified pulse coupled neural network, multi-scale morphological gradient and fractal dimension;

[0075] Specifically, the schematic diagram of the constructed three-channel simplified pulse-coupled neural network is as follows: Figure 4 As shown, the method flow is:

[0076] The multi-modal low-pass sub-band and the band-pass directional sub-band are input to the input and modulation module, and the external stimulus and the stimulus from the surrounding neurons are received through the dendrites, wherein the multi-modal low-pass sub-band and the band-pass directional sub-band are used as the external stimulus; the fusion information area measures the activity level of the neurons and integrates it to obtain a fusion image; the pulse generation area generates an output pulse;

[0077] The three-channel simplified pulse coupled neural network model can be expressed by the following formula:

[0078]

[0079]

[0080] in, and are the external stimuli of the three-channel simplified pulse coupled neural network model corresponding to the infrared intensity, enhanced infrared polarization and the (i, j)th pixel of the visible light image; the three feedback inputs of the (i, j)th neuron and Store external stimuli separately and L ij [n] is the connection input of the (i, j)th neuron at the nth iteration; and Constitutes the internal activity of the (i, j)th neuron obtained after the nth iteration, corresponding to the nonlinear modulation of the feedback and connected input; U ij [n] is the internal state of the three-channel simplified pulse coupled neural network model after the nth iteration corresponding to the (i, j)th neuron, which can be used to generate a fused image; the external output and dynamic threshold of the (i, j)th neuron after the nth iteration are represented by Y ij [n] and E ij [n] means;

[0081] and is corresponding to and The connection strength is calculated by the following formula:

[0082]

[0083] Among them, X∈{1,2,3}; Here, The value depends on and This ensures that corresponding pixels of the source image have different connection strengths. The mechanism will be introduced later;

[0084] α f , calculated by the following formula:

[0085]

[0086] in, and They are S 1 , S 2 and S 3 The standard deviation of f > 0, which prevents U[n] from growing exponentially with respect to n;

[0087] V E and α e They are calculated by the following formulas:

[0088]

[0089]

[0090] Among them, S 1’ , S 2’ and S 3’ Respectively represent S 1 , S 2 and S 3 The Otsu threshold; YesS X The maximum value of pixels.

[0091] Specifically, the specific process of the fusion rule of the passband directional sub-band described in the present invention is:

[0092] Calculate the bandpass directional subband

[0093]

[0094] in, is the fractal dimension FD of the (i, j)th pixel of X calculated using the following expression; and FD X The minimum and maximum values ​​of

[0095] Here, g min and g max are the minimum and maximum grayscale of the respective style;

[0096] After executing the three-channel simplified pulse coupled neural network model for n iterations, the fused bandpass directional subband It is represented by the following expression:

[0097]

[0098] in, and After the nth iteration, and internal activities;

[0099] Specifically, the specific process of the fusion rule of the low-pass sub-band described in the present invention is as follows:

[0100] Calculate the low-pass subband

[0101]

[0102] in, is the multiscale morphological gradient MSMG of the (i, j)th pixel of X calculated using the following expression; and MSMG X The minimum and maximum values ​​of

[0103]

[0104] Where m is the scale number, G t (i, j) is the morphological gradient of X at scale t, W t =1 / (2t+1) is G t The weight of

[0105] In the implementation of the three-channel simplified pulse coupled neural network model n After iterations, the fused bandpass directional subband It is expressed by the following expression:

[0106]

[0107] in, and They are n After iterations and internal activities;

[0108] S33: Use the fusion weight of S32 to fuse the low-pass sub-band and the band-pass directional sub-band respectively;

[0109] S34: The final tri-modal image fusion result is obtained by inverse transformation of non-subsampled contourlet transform.

[0110] Specifically, the self-collected data set is used as the test input fusion method to obtain the image fusion result. The comparison between the image fusion method of the present invention and the existing image fusion method on the fusion effect of the data set is as follows: Figure 5 As shown in the figure, it can be seen that the method not only retains the scene information of the visible light image, but also maintains the salient objects of the infrared image, and also retains more polarization information. This is due to the fusion rule of the three-channel pulse-coupled neural grid model, which adaptively focuses on the salient areas in the infrared image, the polarization information in the infrared polarization image and the background in the visible light image; since other existing methods are two-modal image fusion methods, the present invention adopts two fusions to obtain a three-modal image fusion result.

[0111] Furthermore, the comparison results of the evaluation indicators of the image fusion method of the present invention with the LatLRR method, the MDLatLRR method, the DTCWT method, the U2Fusion method, the Res2Fusion method and the SwinFusion method in the test data set are as follows:

[0112] EN SD <![CDATA[Q AB / F ]]> SF LatLRR 6.9119 37.5614 0.3198 5.8548 MDLatLRR 7.3189 47.5627 0.4615 7.7894 DTCWT 6.9127 35.9289 0.3953 7.3044 U2Fusion 6.8927 40.1423 0.2352 5.5168 SwinFusion 7.1311 46.3109 0.3567 6.7238 This method 7.3742 52.7605 0.5121 7.8261

[0113] In the table, EN (Entropy) represents information entropy. The larger the index is, the better the fused image can retain the information of the source image and the richer the details are. SD (Standard Deviation) represents standard deviation. The larger the index is, the more significant the grayscale change of the fused image is, and the richer the contrast and details of the image are. QAB / F (Edge Preservation Evaluation for Fusion) represents the edge preservation quality of the fused image. The larger the index is, the more the fused image can retain the edge information from the source image, thus having better performance in detail preservation, target contour and visual quality. SF (Spatial Frequency) represents spatial frequency. The larger the index is, the more high-frequency information is in the fused image, the richer the details are, and the higher the clarity of the image is.

[0114] S4: Input the tri-modal fusion image into the fixed detection network to obtain the target detection result.

[0115] Specifically, the self-collected data set is input into a fixed target detection network to obtain the target detection result. The target detection result of the present invention is compared with the detection result of the existing method in terms of effect. Figure 6 As shown in the figure, it can be seen that the target detection result of this method has better detection performance and higher detection accuracy.

[0116] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting small floating targets on the sea surface based on multimodal image fusion, characterized in that: The steps include: Step S1: obtaining a plurality of image pairs, preprocessing the obtained image pairs, and constructing an image sample data set using the preprocessed image pairs; Step S2: using an optimization algorithm to enhance the contrast and detail information of the infrared polarization image; Step S3: Use multi-scale transform to extract low-pass subband and band-pass directional subband information of different modal images, determine the fusion weights of the two parts according to the three-channel simplified pulse coupled neural network, and fuse the low-pass subband and the band-pass directional subband respectively, and finally use the inverse transform of the multi-scale transform to obtain the final fusion result; Step S4: Input the trimodal fusion image into a fixed detection network to obtain the target detection result.

2. According to claim 1, a method for detecting small floating targets on the sea surface based on multimodal image fusion is characterized in that: The specific steps of step S1 are: Step S11: synchronously collecting infrared polarization, infrared intensity and visible light data sets, using a collection system equipped with an infrared polarization camera and a visible light camera to collect images, and the collected infrared polarization, infrared intensity and visible light images contain a variety of scene information; Step S12: preprocessing of the infrared polarization, infrared intensity and visible light data sets includes performing distortion correction operations on the infrared polarization, infrared intensity and visible light images; and performing registration operations on the infrared polarization, infrared intensity and visible light images in the image pair; Step S13: construct a sample data set using the pre-processed infrared polarization, infrared intensity and visible light images of different scenes.

3. The method for detecting small floating targets on the sea surface based on multimodal image fusion according to claim 1, characterized in that: The specific process of enhancing the contrast and detail information of the infrared polarization image is as follows: Step S21: using a particle swarm optimization algorithm to adjust the calculation formula of infrared linear polarization degree to determine the parameters that need to be optimized; Step S22: constructing the fitness function of the particle swarm optimization algorithm, setting the initial parameters of the optimization algorithm, and iteratively optimizing the parameters to be optimized in step S21; Step S23: using the optimal parameters to obtain an infrared polarization image with enhanced contrast and detail information.

4. The method for detecting small floating targets on the sea surface based on multimodal image fusion according to claim 1, characterized in that: The specific process of image fusion is as follows: Step S31: using non-subsampled contourlet transform to decompose the enhanced infrared polarization image, infrared intensity image and visible light image respectively to obtain two parts: a low-pass sub-band and a band-pass directional sub-band; Step S32: determining the fusion weights of the two parts according to the three-channel simplified pulse coupled neural network, the multi-scale morphological gradient and the fractal dimension; Step S33: using the fusion weights of step S32, respectively fuse the low-pass sub-band and the band-pass directional sub-band; Step S34: Obtain the final tri-modal image fusion result through inverse transformation of non-subsampled contourlet transform.

5. The method for detecting small floating targets on the sea surface based on multimodal image fusion according to claim 1, characterized in that: The fixed detection network adopts Yolov5 network.

6. The method for detecting small floating targets on the sea surface based on multimodal image fusion according to claim 3, characterized in that: The calculation formula of the infrared linear polarization degree is: Among them, S0 is the total light intensity; S1 is the linear polarization component in a specific direction between the horizontal and vertical directions; S2 is the linear polarization component in another specific direction perpendicular to the direction of S1; and They represent S1 to the power of a and S2 to the power of b respectively; the parameters that need to be optimized are the powers of S1 and S2 and the value of the coefficient before S0, that is, a, b and c in the calculation formula.

7. The method for detecting small floating targets on the sea surface based on multimodal image fusion according to claim 3, characterized in that: The fitness function of the optimization algorithm is expressed as: fitness=-(PSNR+EN) Among them, PSNR represents the peak signal-to-noise ratio between the infrared polarization image and the infrared intensity image calculated by optimizing parameters; EN represents the information entropy of the infrared polarization image calculated by optimizing parameters.

8. The method for detecting small floating targets on the sea surface based on multimodal image fusion according to claim 4, characterized in that: The specific steps of the three-channel simplified pulse coupled neural network are: Inputting the multi-modal low-pass sub-band and the band-pass directional sub-band into an input and modulation module, receiving external stimulation and stimulation from surrounding neurons through dendrites, wherein the multi-modal low-pass sub-band and the band-pass directional sub-band serve as external stimulation; The fusion information area measures the activity level of neurons and integrates it to obtain a fused image; The pulse generating section generates output pulses.

9. The method for detecting small floating targets on the sea surface based on multimodal image fusion according to claim 4, characterized in that: The expression of the multi-scale morphological gradient is: Among them, X represents different modes, m is the number of scales, G t (i, j) is the morphological gradient of X at scale t at the (i, j)th pixel, W t =1 / (2t+1) is G t The weight of .

10. The method for detecting small floating targets on the sea surface based on multimodal image fusion according to claim 4, characterized in that: The expression of the fractal dimension is: Among them, X represents different modes, g max and g min are the minimum and maximum grayscale of the respective style.

Images