A sea surface floating small target detection method based on multi-modal image fusion

Abstract

The present application relates to a kind of sea surface floating small target detection method based on multi-modal image fusion. Including: obtaining multiple pairs of image pairs, and the image pair obtained is preprocessed, and the image sample data set is constructed using the image pair after preprocessing;The contrast and detail information of infrared polarized image are enhanced using optimization algorithm;Different modal image low-pass sub-band and band-pass directional sub-band information are extracted using multi-scale transformation, the fusion weight of two parts is determined according to three-channel simplified pulse coupling neural network, and finally the final fusion result is obtained using the inverse transformation of multi-scale transformation;Three modal fusion image is input into fixed detection network to obtain target detection result. The present application effectively measures the relative effect of infrared polarization, infrared intensity and visible light image, effectively enhances the contrast and detail information of infrared polarized image, enhances the fusion feature, and has better fusion image quality and target detection performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing and target detection, and particularly relates to a sea surface floating small target detection method based on multi-modal image fusion. BACKGROUND

[0002] In recent years, traditional single-modal imaging technology has been difficult to meet the increasingly complex application requirements. With the rapid development of hardware, multi-modal imaging technology has been widely applied in monitoring, automatic driving, target detection and other fields. In particular, the combination of infrared images and visible light images shows significant advantages in improving the processing effect of subsequent tasks. Infrared detectors can capture scene information that is difficult to perceive under conventional visible light by detecting the thermal radiation of objects, which makes them perform well in dark or obstructed environments. However, due to the low pixel density of infrared detectors, the limitations of optical systems, and other reasons, the resolution of infrared images is low, the image quality is poor, and there is a lack of sufficient texture details. While visible light images have rich scene detail information, their imaging effect is easily affected by environmental factors such as obstructions and bad weather.

[0003] There is a close relationship between image fusion and other advanced computer vision tasks: as a technique for input enhancement, image fusion can significantly improve the accuracy of other computer vision tasks. In addition, the pre-processing effect of image fusion in multi-view feature matching, pedestrian semantic segmentation, target detection and tracking tasks has been verified, indicating that it has effective application value in these fields.

[0004] Although the existing target detection method based on infrared and visible light image fusion can effectively integrate the common and complementary information in infrared and visible light images, it usually only relies on intensity information and ignores the key information from light polarization. This limits the performance of image fusion in certain scenes or materials. Polarization is an inherent property of light, providing unique information about the target, such as surface smoothness, 3D normal and material composition. Due to the differences in surface material, roughness and physical and chemical properties of target objects, they exhibit significant infrared polarization characteristics, which can effectively identify artificial targets hidden in natural backgrounds. Although existing methods have achieved significant results, there are still some challenges in the process of infrared polarization, infrared intensity and visible light three-modal image fusion. First, infrared polarization images are usually obtained under low radiation conditions, and the signal is weak, which is not conducive to preserving significant target information from this image in the fusion process. Second, due to the differences in the highlighted parts of multi-modal features themselves, existing methods are difficult to ensure the actual effect of multi-modal feature interactive fusion. SUMMARY

[0005] Therefore, the present application aims to provide a sea surface floating small target detection method based on multi-modal image fusion, effectively measures the relative effects of infrared polarization, infrared intensity and visible light images, effectively enhances the contrast and detail information of the infrared polarization image, and fully considers the information and differences of prominent areas among the infrared polarization, infrared intensity and visible light images during feature fusion, so that the fused features are enhanced, and the fused image quality and target detection performance are better.

[0006] To achieve the above-mentioned purpose, the present application adopts the following technical scheme: a sea surface floating small target detection method based on multi-modal image fusion, comprising the following steps:

[0007] Step S1: acquiring multiple pairs of images and pre-processing the acquired image pairs, and using the pre-processed image pairs to construct an image sample data set;

[0008] Step S2: using an optimization algorithm to enhance the contrast and detail information of the infrared polarization image;

[0009] Step S3: using multi-scale transformation to extract low-pass sub-band and band-pass directional sub-band information of different modal images, determining the fusion weights of the two parts according to a three-channel simplified pulse coupled neural network, and respectively fusing the low-pass sub-band and the band-pass directional sub-band, and finally using inverse transformation of the multi-scale transformation to obtain the final fusion result;

[0010] Step S4: inputting the three-modal fusion image into a fixed detection network to obtain a target detection result.

[0011] In a preferred embodiment, the step S1 comprises the following specific steps:

[0012] Step S11: synchronous acquisition of infrared polarization, infrared intensity and visible light data sets, using a collection system carrying an infrared polarization camera and a visible light camera to collect images, and the collected infrared polarization, infrared intensity and visible light images contain a variety of scene information;

[0013] Step S12: pre-processing of the infrared polarization, infrared intensity and visible light data set, including distortion correction operation on the infrared polarization, infrared intensity and visible light images; and registration operation on the infrared polarization, infrared intensity and visible light images in the image pair;

[0014] Step S13: using the pre-processed infrared polarization, infrared intensity and visible light images of different scenes to construct a sample data set.

[0015] In a preferred embodiment, the specific process of enhancing the contrast and detail information of the infrared polarization image is as follows:

[0016] Step S21: adjust the calculation formula of the infrared linear polarization degree using a particle swarm optimization algorithm to determine the parameters that need to be optimized;

[0017] Step S22: construct a fitness function of the particle swarm optimization algorithm, set the initial parameters of the optimization algorithm, and iteratively optimize the parameters that need to be optimized in step S21.

[0018] Step S23: use the optimal parameters to obtain the infrared polarization image with enhanced contrast and detail information.

[0019] In a preferred embodiment, the specific process of image fusion is as follows:

[0020] Step S31: use non-subsampled contourlet transform to decompose the enhanced infrared polarization image, the infrared intensity image, and the visible light image respectively to obtain low-pass subbands and band-pass directional subbands;

[0021] Step S32: determine the fusion weights of the two parts according to the three-channel simplified pulse coupled neural network, the multi-scale morphological gradient, and the fractal dimension;

[0022] Step S33: use the fusion weights of step S32 to fuse the low-pass subbands and the band-pass directional subbands respectively;

[0023] Step S34: obtain the final three-modal image fusion result through inverse non-subsampled contourlet transform.

[0024] In a preferred embodiment, the fixed detection network adopts a Yolov5 network.

[0025] In a preferred embodiment, the calculation formula of the infrared linear polarization degree is as follows:

[0026]

[0027] wherein S0 is the total light intensity; S1 is the linear polarization component in a certain specific direction between the horizontal and vertical directions; S2 is the linear polarization component in another specific direction perpendicular to S1 direction; and represent the a-th power of S1 and the b-th power of S2 respectively; the parameters that need to be optimized are the powers of S1 and S2 and the value of the coefficient before S0, i.e., a, b, and c in the calculation formula.

[0028] In a preferred embodiment, the expression of the fitness function of the optimization algorithm is as follows:

[0029] fitness = -(PSNR + EN)

[0030] Wherein, the PSNR represents the peak signal-to-noise ratio between the infrared polarization image and the infrared intensity image calculated by the optimization parameter; the EN represents the information entropy of the infrared polarization image calculated by the optimization parameter.

[0031] In a preferred embodiment, the specific steps of the three-channel simplified pulse-coupled neural network are as follows:

[0032] The multi-modal low-pass sub-band and the band-pass directional sub-band are input to the input and modulation module, and the external stimulus and the stimulus from the surrounding neurons are received by the dendrites, wherein the multi-modal low-pass sub-band and the band-pass directional sub-band are used as the external stimulus;

[0033] The fusion information area measures the activity level of the neuron and integrates the fusion image;

[0034] The pulse generation area generates an output pulse.

[0035] In a preferred embodiment, the expression of the multi-scale morphological gradient is as follows:

[0036]

[0037] Wherein, X represents different modalities, m is the scale number, G t (i, j) is the morphological gradient of X at the (i, j) pixel at scale t, W t =1 / (2t+1) is the weight of G t .

[0038] In a preferred embodiment, the expression of the fractal dimension is as follows:

[0039]

[0040] Wherein, X represents different modalities, g max and g min are the minimum and maximum gray scales of each style.

[0041] Compared with the prior art, the present application has the following beneficial effects:

[0042] 1. The present application is aimed at the problem that the contrast of the infrared polarization image is low, resulting in unclear details and difficulty in fusion, and the calculation formula of the infrared linear polarization degree is improved by an optimization algorithm, so that the unique information of the infrared polarization image can be better preserved in the fusion process.

[0043] 2. The present application proposes a three-modality image fusion method combining the infrared polarization, infrared intensity and visible light images of the infrared polarization image, so that the common and complementary information from different modality images of the same scene can be better preserved, and the target detection performance is improved.

[0044] 3. The application deeply mines the feature relationship, avoids insufficient fusion of different modal features, refines and enhances the incomplete fused features, makes them have better scene representation ability, and improves the target detection performance. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 is a target detection method of an embodiment of the application;

[0046] Figure 2 is a comparison chart of an original infrared polarization image and an enhanced infrared polarization image of an embodiment of the application;

[0047] Figure 3 is a schematic diagram of a deep channel attention module of an embodiment of the application;

[0048] Figure 4 is a schematic diagram of a three-channel simplified pulse coupled neural network of an embodiment of the application;

[0049] Figure 5 is a fusion effect comparison chart of an image fusion method and an existing method of an embodiment of the application;

[0050] Figure 6 is a comparison chart of a target detection result and a detection result of an existing method of an embodiment of the application. DETAILED DESCRIPTION

[0051] The application will be further described below in conjunction with the drawings and embodiments.

[0052] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise indicated, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0053] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and furthermore, it should be understood that when the terms "comprise" and / or "include" are used in the specification, there is a feature, step, operation, device, component and / or combination thereof.

[0054] As shown in Figures 1-6 The application provides a sea surface floating small target detection method based on multi-modal image fusion, the method comprising:

[0055] S1: Obtain a plurality of image pairs, and pre-process the obtained image pairs, and use the pre-processed image pairs to construct an image sample data set;

[0056] Specifically, the data set used in the present application is a data set collected by using a synchronous acquisition system equipped with an infrared polarization camera and a visible light camera, including infrared polarization images, infrared intensity images and visible light images in multiple environments. The images of the data set are corrected for distortion, and then the images of the data set are registered using the RIFT method, and finally the infrared polarization images, the infrared intensity and the visible light images are normalized to form a sample data set, so as to complete the preprocessing of the data set.

[0057] S2: enhancing the contrast and detail information of the infrared polarization image by using an optimization algorithm;

[0058] Specifically, the specific process of enhancing the infrared polarization image in the present application is as follows:

[0059] S21: adjusting the calculation formula of the infrared linear polarization degree by using a particle swarm optimization algorithm to determine the parameters to be optimized;

[0060] Specifically, the parameters to be optimized in the linear polarization degree in the present application are as follows:

[0061]

[0062] wherein S0 is the total light intensity; S1 is the linear polarization component in a certain specific direction between the horizontal and vertical directions; S2 is the linear polarization component in another specific direction perpendicular to S1; the parameters to be optimized are the power of S1 and S2 and the value of S0, i.e. a, b and c in the calculation formula.

[0063] S22: constructing a fitness function of the particle swarm optimization algorithm, setting the initial parameters of the optimization algorithm, and iteratively optimizing the parameters to be optimized in S21;

[0064] Specifically, the expression of the fitness function of the optimization algorithm used in the present application is as follows:

[0065] f itness =-(PSNR+EN)

[0066] wherein PSNR represents the peak signal-to-noise ratio between the infrared polarization image and the infrared intensity image calculated by the optimization parameters; EN represents the information entropy of the infrared polarization image calculated by the optimization parameters.

[0067] Specifically, the initial parameters of the optimization algorithm used in the present application are as follows:

[0068] Inertia weight 0.2, individual learning factor 2, group learning factor 2, particle swarm size 40, particle dimension 3, maximum iteration number 30, position parameter limit a=[2, 10], b=[2, 10] and c=[0.1, 10] (wherein a, b take integer, c takes one decimal) and velocity parameter limit v∈[-1, 1].

[0069] S23: using the optimal parameters, obtaining the infrared polarization image with enhanced contrast and detail information;

[0070] Specifically, the original infrared polarization image and the enhanced infrared polarization image are as shown in the figure. Figure 2 As shown in the figure, the enhanced infrared polarization image has more obvious contour details than the original image, and the polarization information is enhanced.

[0071] S3: using multi-scale transformation to extract low-pass sub-band and band-pass directional sub-band information of different modal images, determining the fusion weight of the two parts according to the three-channel simplified pulse coupled neural network, and respectively fusing the low-pass sub-band and the band-pass directional sub-band, and finally using the inverse transformation of the multi-scale transformation to obtain the final fusion result;

[0072] Specifically, the overall framework of the image fusion method constructed by the application is as shown in the figure. Figure 3 The specific method flow is as follows:

[0073] S31: using non-subsampled contourlet transform to respectively decompose the enhanced infrared polarization image, the infrared intensity image and the visible light image, to obtain two parts of low-pass sub-band and band-pass directional sub-band;

[0074] S32: determining the fusion weight of the two parts according to the three-channel simplified pulse coupled neural network, multi-scale morphological gradient and fractal dimension;

[0075] Specifically, the schematic diagram of the three-channel simplified pulse coupled neural network constructed is as shown in the figure. Figure 4 The method flow is as follows:

[0076] The multi-modal low-pass sub-band and band-pass directional sub-band are input into the input and modulation module, and the external stimulus and the stimulus from the surrounding neurons are received through the dendrites, wherein the multi-modal low-pass sub-band and band-pass directional sub-band are taken as the external stimulus; the fusion information area measures the activity level of the neurons and integrates to obtain a fusion image; the pulse generation area generates an output pulse;

[0077] The three-channel simplified pulse coupled neural network model can be represented by the following formula:

[0078]

[0079]

[0080] where, and are the external stimuli of the three-channel reduced pulse coupled neural network model corresponding to the (i,j)th pixel of the infrared intensity, enhanced infrared polarization and visible light images, respectively; the three feedback inputs of the (i,j)th neuron and store the external stimuli and L ij [n] is the connection input of the (i,j)th neuron at the nth iteration; and constitute the internal activity on the (i,j)th neuron obtained after the nth iteration, corresponding to the nonlinear modulation of the feedback and connection inputs; U ij [n] is the internal state of the three-channel reduced pulse coupled neural network model corresponding to the (i,j)th neuron after the nth iteration, which can be used to generate the fused image; the external output and dynamic threshold of the (i,j)th neuron after the nth iteration are denoted by Y ij [n] and E ij [n], respectively;

[0081] and are the connection strengths corresponding to and are calculated by the following formula:

[0082]

[0083] where X ∈ {1,2,3}; here, The value of depends on and This ensures that the corresponding pixels of the source images have different connection strengths, The mechanism of is introduced later;

[0084] α f is calculated by the following formula:

[0085]

[0086] where, and are the standard deviations of S 1 , S 2 and S 3 , respectively, and in general, α f > 0, which prevents U[n] from growing exponentially with respect to n;

[0087] V E and α e are calculated by the following formula:

[0088]

[0089]

[0090] Among them, S 1’ S 2’ and S 3’ S represents 1 S 2 and S 3 Otsu threshold; It is S X The maximum pixel value.

[0091] Specifically, the specific process of the fusion rule for the bandpass directional sub-bands described in this invention is as follows:

[0092] Calculate the bandpass directional subband

[0093]

[0094] in, FD is the fractal dimension of the (i, j)th pixel of X, calculated using the following expression; and They are FD X The minimum and maximum values;

[0095] Here, g min and g max These are the minimum and maximum gray levels for their respective styles;

[0096] After n iterations of the three-channel simplified pulse-coupled neural network model, the fused bandpass directional subband is obtained. It is represented by the following expression:

[0097]

[0098] in, and They are respectively after the nth iteration and Internal activities;

[0099] Specifically, the specific process of the low-pass subband fusion rule described in this invention is as follows:

[0100] Calculate low-pass subband

[0101]

[0102] in, is the multiscale morphological gradient of the (i, j)th pixel of X calculated using the following expression; and are the minimum and maximum values of the multiscale morphological gradient of X X , respectively;

[0103]

[0104] where m is the number of scales, G t (i, j) is the morphological gradient of X at scale t, W t = 1 / (2t+1) is the weight of G t ;

[0105] After performing the three-channel simplified pulse coupled neural network model n after the (k-1)th iteration, the fused bandpass directional subband is represented by the following expression:

[0106]

[0107] where, and are the internal activities of the (k-1)th iteration of the lowpass subband and the bandpass directional subband, respectively; n and are the internal activities of the kth iteration of the lowpass subband and the bandpass directional subband, respectively;

[0108] S33: using the fusion weights of S32, the lowpass subband and the bandpass directional subband are fused, respectively;

[0109] S34: the final three-modal image fusion result is obtained by inverse non-subsampled contourlet transform.

[0110] Specifically, the self-collected data set is used as a test input to fuse the method to obtain an image fusion result, and the comparison of the image fusion method of the present application and the existing image fusion method in the fusion effect of the data set is as shown in the following table: Figure 5 As can be seen from the figure, the present method not only retains the scene information of the visible light image, but also maintains the significant objects of the infrared image, and also retains more polarization information, which benefits from the fusion rule of the three-channel pulse coupled neural grid model, which adaptively focuses on the significant areas in the infrared image, the polarization information in the infrared polarization image and the background in the visible light image; since other existing methods are two-modal image fusion methods, the present application adopts fusion twice to obtain a three-modal image fusion result.

[0111] Further, the comparison results of the evaluation indexes of the image fusion method of the present application and the LatLRR method, the MDLatLRR method, the DTCWT method, the U2Fusion method, the Res2Fusion method and the SwinFusion method in the test data set are as follows:​

[0112] EN SD Q AB / F ]] SF LatLRR 6.9119 37.5614 0.3198 5.8548 MDLatLRR 7.3189 47.5627 0.4615 7.7894 DTCWT 6.9127 35.9289 0.3953 7.3044 U2Fusion 6.8927 40.1423 0.2352 5.5168 SwinFusion 7.1311 46.3109 0.3567 6.7238 The method 7.3742 52.7605 0.5121 7.8261

[0113] In the table, EN (Entropy) represents information entropy, the greater the index, the better the fusion image can retain the information of the source image, and the more details; SD (Standard Deviation) represents standard deviation, the greater the index, the more significant the gray scale change of the fusion image, and the richer the contrast and details of the image; QAB / F (Edge Preservation Evaluation for Fusion) represents the edge retention quality of the fusion image, the greater the index, the more edge information from the source image that the fusion image can retain, thereby having better performance in detail retention, target contour and visual quality; and SF (Spatial Frequency) represents spatial frequency, the greater the index, the more high-frequency information in the fusion image, the richer the details, and the higher the definition of the image.

[0114] S4: inputting the three-modal fusion image into a fixed detection network to obtain a target detection result.

[0115] Specifically, the self-collected data set is input into the fixed target detection network to obtain a target detection result, and the comparison of the target detection result of the present application with the detection result of the prior art in effect is as shown in the following table. Figure 6 As shown in the figure, the detection performance of the target detection result of the present application is better, and the detection precision is higher.

[0116] The above is only a preferred embodiment of the present application, and does not limit the present application, any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A sea surface floating small target detection method based on multi-modal image fusion, characterized in that, The method comprises the following steps: Step S1: acquiring a plurality of pairs of images, and preprocessing the acquired pairs of images, and constructing an image sample data set using the preprocessed pairs of images; Step S2: enhancing the contrast and detail information of the infrared polarization image using an optimization algorithm; Step S3: extracting low-pass sub-band and band-pass directional sub-band information of different modal images using multi-scale transformation, determining fusion weights of the two parts according to a three-channel simplified pulse coupled neural network, and fusing the low-pass sub-band and the band-pass directional sub-band respectively, and finally obtaining a final fusion result using inverse transformation of the multi-scale transformation; Step S4: inputting the three-modal fusion image into a fixed detection network to obtain a target detection result; The specific process of enhancing the contrast and detail information of the infrared polarization image is as follows: Step S21: adjusting the calculation formula of the infrared linear polarization degree using a particle swarm optimization algorithm to determine the parameters that need to be optimized; Step S22: constructing a fitness function of the particle swarm optimization algorithm, setting initial parameters of the optimization algorithm, and iteratively optimizing the parameters that need to be optimized in step S21; Step S23: using the optimal parameters to obtain the infrared polarization image with enhanced contrast and detail information; The specific process of image fusion is as follows: Step S31: decomposing the enhanced infrared polarization image, the infrared intensity image and the visible light image using non-subsampled contourlet transformation to obtain low-pass sub-band and band-pass directional sub-band; Step S32: determining fusion weights of the two parts according to a three-channel simplified pulse coupled neural network, multi-scale morphological gradient and fractal dimension; Step S33: using the fusion weights of step S32 to fuse the low-pass sub-band and the band-pass directional sub-band respectively; Step S34: obtaining the final three-modal image fusion result through inverse transformation of the non-subsampled contourlet transformation; The fixed detection network adopts a Yolov5 network.

2. The sea surface floating small target detection method based on multi-modal image fusion according to claim 1, characterized in that, The specific steps of step S1 are as follows: Step S11: synchronous acquisition of infrared polarization, infrared intensity and visible light data sets, using a collection system with an infrared polarization camera and a visible light camera to collect images, and the collected infrared polarization, infrared intensity and visible light images contain a variety of scene information; Step S12: preprocessing of the infrared polarization, infrared intensity and visible light data set, including distortion correction operation on the infrared polarization, infrared intensity and visible light images; and registration operation on the infrared polarization, infrared intensity and visible light images in the image pair; Step S13: constructing a sample data set using the preprocessed infrared polarization, infrared intensity and visible light images of different scenes.

3. The sea surface floating small target detection method based on multi-modal image fusion according to claim 1, characterized in that, The calculation formula of the infrared linear polarization degree is as follows: where S0 is the total light intensity; S1 is the linear polarization component in a certain direction between the horizontal and vertical directions; S2 is the linear polarization component in another certain direction perpendicular to S1 direction; and represent the a-th power of S1 and the b-th power of S2, respectively; the parameters to be optimized are the powers of S1 and S2 and the value of the coefficient before S0, i.e., a, b and c in the calculation formula.

4. The sea surface floating small target detection method based on multi-modal image fusion according to claim 1, characterized in that, The expression of the fitness function of the optimization algorithm is as follows: fitness = -(PSNR + EN) Wherein, PSNR represents the peak signal-to-noise ratio between the infrared polarization image and the infrared intensity image calculated by the optimization parameter; EN represents the information entropy of the infrared polarization image calculated by the optimization parameter.

5. The sea surface floating small target detection method based on multi-modal image fusion according to claim 1, characterized in that, The specific steps of the three-channel simplified pulse coupled neural network are as follows: The multi-modal low-pass sub-band and the band-pass directional sub-band are input into an input and modulation module, receive external stimulation and stimulation from surrounding neurons through dendrites, wherein the multi-modal low-pass sub-band and the band-pass directional sub-band are external stimulation; The fusion information area measures the activity level of the neuron and integrates the fusion image; The pulse generation area generates an output pulse.

6. The sea surface floating small target detection method based on multi-modal image fusion according to claim 1, characterized in that, The expression of the multi-scale morphological gradient is: where X denotes different modalities, m is the scale number, G t (i,j) is the morphological gradient of X at the (i,j)th pixel at scale t, W t = 1 / (2t+1) is the weight of G t .

7. The sea surface floating small target detection method based on multi-modal image fusion according to claim 1, characterized in that, The expression of the fractal dimension is: where X denotes different modalities, g max and g min are the minimum and maximum gray levels of each style, respectively.

Images