A target detection method and device based on dual-modal fusion network

Abstract

The present invention provides a target detection method based on a dual-modal fusion network, comprising acquiring an image pair dataset, establishing a dual-modal image dataset, constructing a module to encode infrared images and visible light images respectively, constructing a dual-path gated fusion network, inputting fused features into the third layer of a YOLOv5-s network of a benchmark method to obtain a final detection network, and inputting the processed image pair dataset into the final detection network to obtain a detection result. The method also includes a target detection device based on the method of the present invention. The method of the present invention adopts an embedded gated fusion network to enable the model to determine the contribution of two modal images to detection in different scenarios; introduces the idea of ​​multi-task learning, adaptively adjusts weight distribution, and realizes cross-modal feature fusion; and has good robustness in complex environments such as the night.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a target detection method based on a dual-modal fusion network. Background Art

[0002] Current object detection methods primarily use unimodal images relevant to the detection task as training data. However, using only unimodal images can sometimes make it difficult to detect objects in complex real-world scenarios. To address this issue, a method using multimodal images as training data has been proposed. Multimodal images, such as infrared and visible light images, offer complementary advantages. Infrared images rely on the heat generated by the target object and are unaffected by lighting conditions, but they cannot capture detailed information about the target. Visible light images, on the other hand, clearly capture the target's texture features and details, but are susceptible to lighting conditions. In the existing technology, DEVAGUPTAPU C et al. (2019 IEEE / CVF Conference on Computer Vision and Pattern Recognition Workshops, 2019, Borrow from anywhere: Pseudo multi-modal object detection in thermal imagery.) used CycleGAN to create synthetic infrared images, and used the illumination-aware fusion framework to fuse visible light and infrared images to help improve the performance of object detection in infrared images; YANG L et al. (arXiv preprint, 2201.03786, 2022, Drone Object Detection Using RGB / IR Fusion.) used an illumination subnetwork to assign weights and jointly detected pedestrians with visible light and infrared images.

[0003] The method proposed by DEVAGUPTAPU C et al. uses the same network to extract features from two images. The extracted features are not weighted and are simply concatenated. This method fails to consider the differences between images or the varying contributions of different source images to detection in different scenarios. The method proposed by YANG L et al. uses the same network to detect two images, assigns weights to the detection results, and fails to eliminate redundant features at the feature level.

[0004] The above methods do not extract features from bimodal images separately, and their fusion of multimodal features is insufficient. They also fail to find a suitable method to pre-process some redundant feature information, and further efforts are needed to improve the complementary advantages between multimodal feature information. Furthermore, the above methods do not use different networks to extract features from different image sources, but rather only use the same network to extract features. Furthermore, their fusion of bimodal features is insufficient, and further efforts are needed to improve the complementary advantages between bimodal feature information. Summary of the Invention

[0005] In order to overcome the problem that the existing technology is not sufficient in fusing bimodal features, the present invention proposes an object detection method based on a bimodal fusion network. Its benchmark method is YOLOV5-s, which adopts early fusion and embeds a gated fusion network, so that the model can determine the contribution of the two modal images to detection in different scenarios.

[0006] Specifically, including:

[0007] Step A: Acquire an image pair dataset, i.e., use an infrared imaging device to capture an infrared image, use a visible light imaging device to capture a visible light image, and then align the two images to obtain an image pair dataset;

[0008] Step B: Establish a bimodal image dataset, that is, perform data enhancement processing on the registered image dataset to form a bimodal dataset;

[0009] Step C: construct modules to encode infrared images and visible light images respectively;

[0010] Step D: Construct a dual-path gated fusion network, that is, calculate the infrared feature weights and visible light feature weights through the constructed gated fusion network, adaptively adjust the weight distribution of the two-path features, obtain the fusion feature, and complete the gated fusion network training;

[0011] Step E: Input the fused features into the third layer of the YOLOv5-s network of the baseline method to obtain the final detection network;

[0012] Step F: Input the image pair dataset processed in step B into the final detection network to obtain the detection results.

[0013] Further,

[0014] Step A includes:

[0015] Step A1: Acquire infrared and visible light images. This involves taking photos using a visible light camera and an infrared camera in a fully enclosed environment. Hardware is used to align the optical axes of the cameras so that the image ranges of the two photos are roughly consistent.

[0016] Step A2: performing image registration on the acquired infrared image and visible light image, i.e., using an alignment strategy and adjusting the image size to achieve image registration;

[0017] Step B includes:

[0018] Step B1: performing rough processing on the image pair data set to obtain a rough processed image pair data set, wherein the rough processing method includes: enhancing edges, sharpening, smoothing, and detail;

[0019] Step B2: arranging the roughly processed image data set, including rotation, flipping, and brightness adjustment;

[0020] Step C includes:

[0021] Step C1: Construct an infrared encoder to obtain infrared image information and neuron weights, and then obtain infrared image features;

[0022] Step C2: construct a visible light encoder, input a visible light image, and obtain visible light image features;

[0023] Step C3: combining the infrared image features obtained in step C1 and the visible light image features obtained in step C2 to obtain a combined feature;

[0024] The input of step D is the combined features generated in step C3, and the output of step D is the fused features after fusion;

[0025] Step D includes:

[0026] Step D1: Divide the combined feature slices into infrared feature blocks and visible light feature blocks;

[0027] Step D2: Generate two gated networks to calculate infrared probability features and visible light probability features;

[0028] Step D3: Calculating an adaptive infrared feature and an adaptive visible light feature based on the infrared probability feature and the visible light probability feature generated in step D2;

[0029] Step D4: adding the adaptive infrared feature and the adaptive visible light feature;

[0030] Step D5: Downsample and pool the infrared image features obtained in step C1 and the visible light image features obtained in step C2;

[0031] Step D5 includes:

[0032] Step D5-1: Transform the infrared image feature F IR , through the maximum pooling operation, the pooled infrared feature F is obtained IRc ;

[0033] Step D5-2: Transform the visible light image feature F VS , through the maximum pooling operation, the pooled visible light feature F is obtained IRc ;

[0034] Step D6: Concatenate the result of step D4, the pooled infrared features, and the pooled visible light features according to the channel dimension to obtain a fused feature.

[0035] Further,

[0036] Step B1 includes:

[0037] Step B1-1: Enhance image edges;

[0038] Step B1-2: Sharpening, using contour filtering method to extract all contour information in the image;

[0039] Step B1-3: Smoothing, using a smoothing filter method to make the image brightness change smoothly and gradually, reducing sudden gradients;

[0040] Step B1-4: Details, using detail enhancement filtering method to make the details in the image more obvious;

[0041] Step B2 includes:

[0042] Step B2-1: Rotate the image dataset;

[0043] Step B2-2: flip the image dataset;

[0044] Step B2-3: Adjust the brightness and contrast of the image dataset;

[0045] In step C1, the infrared encoder is composed of an infrared mini residual module. The infrared mini residual module has a three-layer structure consisting of a maximum pooling layer and two consecutive 1×1 convolutional layers;

[0046] Step C2 includes:

[0047] Step C2-1: Input the visible light image output from step B2 into a visible light encoder to obtain detailed feature information of the visible light image; the visible light encoder is composed of a visible light mini residual module, which is a three-layer structure consisting of a maximum pooling layer, a 1×1 convolutional layer, and a 3×3 convolutional layer;

[0048] Step C2-2: Use the coordinate attention method to calculate the visible light image feature weights in the vertical and horizontal directions, and then obtain the weights in the vertical spatial direction and the weights in the horizontal spatial direction;

[0049] Step C2-3: Multiply the results of step C2-1 and step C2-2 pixel by pixel to obtain visible light image features;

[0050] Step D2 includes:

[0051] Step D2-1: Construct an infrared image gating network and a visible light image gating network. The gating network consists of two layers: a 1×1 convolutional layer and a softmax layer.

[0052] Step D3 includes:

[0053] Step D3-1: Multiply the infrared probability feature in the infrared image gating network and the infrared feature block element by element and adaptively weight them to obtain an adaptive infrared feature;

[0054] Step D3-2: Multiply the visible light probability features in the visible light image gating network and the visible light feature blocks element by element and adaptively weight them to obtain adaptive visible light features.

[0055] Further,

[0056] Step A1: The visible light camera model is PointGrey Flea3, and the infrared camera model is FLIR-A35;

[0057] In step A2-3, the spatial resolution of the visible light camera is 640×480 pixels, with a vertical field of view of 103.6°. The thermal infrared camera has a spatial resolution of 320×256 pixels, with a vertical field of view of 39°. By reducing the boundary area of ​​the visible light image, a visible light image registered with the original thermal infrared image is obtained in the aligned image domain to obtain a registered image pair dataset. The image size is 640×512, with a resolution of 320×256 pixels.

[0058] The rough processing method in step B1 is implemented using the PIL class in Python;

[0059] The sorting process in step B2 is implemented using the OpenCV class in Python;

[0060] The brightness and contrast adjustment in step B2-3 is a simple linear transformation of the pixel. The output pixel value output_pixel is calculated using the formula output_pixel = alpha * (input_pixel) + beta, where input_pixel is the input pixel, alpha is the transparency factor, and beta is the brightness bias factor.

[0061] In step C1, the infrared image first undergoes a maximum pooling operation, and then passes through two consecutive 1×1 convolutional layers to linearly combine the 24 infrared image features;

[0062] Energy function factor The calculation formula is:

[0063]

[0064] Where M = H × W is the number of neurons on the channel to be calculated, H and W are the length and width of the image feature map respectively, and x i It is a feature map input feature of the neighboring neurons of the current neuron, i is the number of feature maps, i ranges from 1 to M;

[0065] Energy function factor The calculation formula is:

[0066]

[0067] Energy function Energy function factor We get the energy function The calculation formula is:

[0068]

[0069] Where λ is a hyperparameter and its default value is 10 -4 , t is the single channel input feature of the target neuron;

[0070] The neuron weights are calculated through the energy function The inverse of is

[0071] Multiply the infrared image information and the neuron weight pixel by pixel to obtain the infrared image features;

[0072] In step C2-1, the visible light image first passes through the maximum pooling layer, then through the 1×1 convolution layer, and finally through the 3×3 convolution layer to obtain the detailed feature information of the visible light image;

[0073] Step C2-2 includes:

[0074] Step C2-2-1: Extract visible light image features from the vertical direction using global average pooling, set the number of output feature maps to 1, and obtain a one-dimensional vertical vector;

[0075] Step C2-2-2: Extract visible light image features from the horizontal direction using a global average pooling operation, set the number of output feature maps to 1, and obtain a one-dimensional horizontal vector;

[0076] Step C2-2-3: Concatenate the one-dimensional vertical vector obtained in step C2-2-1 and the one-dimensional horizontal vector obtained in step C2-2-2 according to the channel dimension to obtain a combined vector;

[0077] Step C2-2-4: Use 1×1 convolution to change the channel of the combined vector to 8, and obtain the transpose of the combined vector;

[0078] Step C2-2-5: Apply batch normalization to the transpose of the combined vector, calculate the mean and variance, and then pass it through the hswish nonlinear activation function layer to obtain the intermediate vector;

[0079] Step C2-2-6: Slice the intermediate vector into two parts: a vertical tangent vector and a horizontal tangent vector according to the channel dimension;

[0080] Step C2-2-7: The vertical tangent vector is passed through a 1×1 convolution to adjust the number of channels to 24, and then normalized through the Sigmoid activation function to obtain the weight in the vertical spatial direction;

[0081] Step C2-2-8: The horizontal tangent vector is passed through a 1×1 convolution to adjust the number of channels to 24, and then normalized through a Sigmoid activation function to obtain the weight in the horizontal spatial direction;

[0082] Step D2-1 includes:

[0083] Step D2-1-1: Pass the infrared feature image block through a 1×1 convolution layer, set the number of channels to 128, and obtain the infrared weight;

[0084] Step D2-1-2: Pass the visible light feature image block through a 1×1 convolutional layer, set the number of channels to 128, and obtain the visible light weight;

[0085] Step D2-1-3: Multiply the infrared feature block and the infrared weight pixel by pixel, and record it as the weighted infrared feature;

[0086] Step D2-1-4: Multiply the visible light feature block and the visible light weight pixel by pixel, and record it as the weighted visible light feature;

[0087] Step D2-1-5: Use the Softmax layer to calculate the probability of the weighted infrared feature, and record the infrared probability feature in the infrared image gating network as S IR , the calculation process is:

[0088]

[0089] in, For infrared feature blocks, is the visible light feature block, Q1 is the infrared weight, Q2 is the visible light weight, and e represents the exponential function;

[0090] Step D2-1-6: Use the Softmax layer to calculate the probability of weighted visible light features, and record the calculation of the visible light probability feature S in the visible light image gating network G2 VS , the calculation process is:

[0091]

[0092] e stands for exponential function.

[0093] A target detection device based on the above method includes: a dual-mode image acquisition unit 1, a computing unit 2, and a display unit 3. The dual-mode image acquisition unit 1 acquires infrared images and visible light images, and then sends them to the computing unit 2. The computing unit 2 completes dual-mode image recognition and then sends the results to the display unit 3. The dual-mode image acquisition unit 1 includes a sealed box 11, a beam splitter 12, an infrared camera 13, a three-axis platform 14, and a visible light camera 15; a light-collecting port 111 is provided at the lower portion of one side of the sealed box 11, the beam splitter 12 is fixedly mounted on the bottom surface of the sealed box 11, the lower portion is close to the light-collecting port 111, and has an angle with the bottom surface; the upper portion of the beam splitter 12 is fixed to the side wall of the sealed box 11, and the upper fixed portion of the beam splitter 12 extends upward at the same time to form a fixed bracket; the infrared camera 13 is connected to the fixed bracket of the beam splitter 12, vertically downward, and the lens of the infrared camera 13 is directly facing the center position of the beam splitter 12; the three-axis platform 14 is fixedly installed on the side of the closed box 11, and a movable rod 141 is provided at the bottom thereof, and the position of the movable rod 141 in the box can be adjusted; the visible light camera 15 is installed at the bottom of the movable rod 141 of the three-axis platform 14, and the lens is facing the light opening 111 of the closed box 11, and the position of the visible light camera 15 can be adjusted; the computing unit 2 is configured with an implementation program corresponding to the method of the present invention; the display unit 3 is a display; external light passes through the light opening 111 and is divided into two paths through the beam splitter 12, one path is the infrared thermal band, which is reflected by the beam splitter and enters the infrared camera 13, and the other path is the visible light band, which is transmitted through the beam splitter 12 to the visible light camera 15, and the infrared camera 13 and the visible light camera 15 adopt global shutter synchronization control to collect images; the collected infrared image and visible light image are transmitted to the computing unit 2, and the computing unit 2 processes the input infrared image and visible light image through the aforementioned invention method, and finally outputs the result to the display unit 3.

[0094] Further,

[0095] The dimensions of the sealed box 11 are 80 cm long, 40 cm wide, and 80 cm high; the light opening 111 is circular with a diameter of 15 cm; the beam splitter 12 has an angle of 40 degrees with the bottom surface and an angle of 40 degrees with the bottom surface; the three-axis platform 14 can adjust the position of the movable rod 141 in the box through a knob, and then adjust the position of the visible light camera 15; the collected infrared image and visible light image are transmitted to the calculation unit 2 via wired or wireless means, and the calculation unit 2 outputs the results to the display unit 3 via a cable or wirelessly.

[0096] Further,

[0097] The beam splitter 12 is made of zinc-coated silicon wafer; the infrared camera 13 is a FLIR-A35; the computing unit 2 is a computer or an image processor FPGA; and the display unit 3 is a warning light or a warning bell.

[0098] Compared with existing technologies, the present invention offers the following advantages: It embeds a gated fusion network, enabling the model to determine the contribution of two modal images to detection in different scenarios; it incorporates multi-task learning to adaptively adjust weight distribution and achieve cross-modal feature fusion; and it exhibits robustness in complex environments, such as darkness. Furthermore, the present method can also detect targets in single-input visible light or infrared images, with significantly improved detection performance compared to the baseline method, YOLOv5-s. BRIEF DESCRIPTION OF THE DRAWINGS

[0099] Figure 1 Flow chart of the steps for implementing the method of the present invention;

[0100] Figure 2 This is a structural diagram of the hardware device corresponding to the method of the present invention;

[0101] Figure 3 This is a comparison diagram between the method of the present invention and the YOLOv5-s method, where Figure 3 (a) represents the target that should be detected, Figure 3 (b) is the detection result of the YOLOv5-s method. Figure 3 (c) is the detection result of the method of the present invention.

[0102] Description of reference numerals:

[0103] 1. Dual-mode image acquisition unit; 2. Computing unit; 3. Display unit; 11. Sealed box; 12. Beam splitter; 13. Infrared camera; 14. Three-axis platform; 15. Visible light camera; 111. Lighting port; 141. Movable rod. DETAILED DESCRIPTION

[0104] In order to make the objectives, technical methods and advantages of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0105] YOLOV5-s is an existing detection method, but it can only input single-modal images for detection and has poor performance on infrared images, resulting in a limited range of applications. Therefore, the present invention further improves on this method. The YOLOV5-s architecture consists of three layers: the backbone network, the fusion network, and the prediction head. The backbone network is composed of a CSP structure with 12 layers.

[0106] The fusion network consists of a pyramid fusion and path aggregation network; the prediction head primarily uses 1×1 convolution for prediction, calculated using the GIOU loss function. This paper uses YOLOv5-s as a benchmark method. Specifically, the fused features are input into the third layer of the YOLOv5-s network, the benchmark method, to obtain the final detection network, which is then used to obtain the detection results.

[0107] The method of the present invention mainly implements the steps as follows Figure 1 As shown, it mainly includes:

[0108] Step A: Obtain an image pair dataset. This involves capturing infrared images using an infrared imaging device and visible light images using a visible light imaging device. The two images are then registered to obtain an image pair dataset.

[0109] Step A includes:

[0110] Step A1: Acquire infrared and visible light images. Use a visible light camera (PointGrey Flea3) and an infrared camera (FLIR-A35) to capture images in a fully enclosed environment. Use hardware to align the camera optical axes so that the image ranges of the two images are roughly consistent.

[0111] Step A2: Perform image registration on the acquired infrared image and visible light image. Use an alignment strategy to adjust the image size and achieve image registration.

[0112] Step A2 includes:

[0113] Step A2-1: Adjust the two cameras to similar imaging parameters, including exposure time, aperture type, backlight, 3D digital noise reduction, and mirror mode.

[0114] Step A2-2: Align the common horizons of the two cameras at the pixel level;

[0115] Step A2-3: Register the two camera images. The visible light camera has a spatial resolution of 640 × 480 pixels and a vertical field of view of 103.6°. The thermal infrared camera has a spatial resolution of 320 × 256 pixels and a vertical field of view of 39°. The visible light camera has a larger field of view than the thermal infrared camera. By reducing the boundary area of ​​the visible light image, a visible light image is obtained in the aligned image domain that is registered with the original thermal infrared image. This results in a dataset of registered image pairs, all with an image size of 640 × 512 and a resolution of 320 × 256 pixels.

[0116] Step B: Establish a bimodal image dataset. That is, perform data augmentation on the registered image dataset to form a bimodal dataset.

[0117] Step B includes:

[0118] Step B1: Perform rough processing on the image pair dataset to obtain a rough processed image pair dataset. The rough processing methods include: edge enhancement, sharpening, smoothing, and detail enhancement, all of which are implemented using the PIL class in Python;

[0119] Step B1 includes:

[0120] Step B1-1: Enhance the image edges using the deep edge enhancement filtering method in the ImageFilter class in PIL;

[0121] Step B1-2: Sharpening, using contour filtering method to extract all contour information in the image;

[0122] Step B1-3: Smoothing, using a smoothing filter method to make the image brightness change smoothly and gradually, reducing sudden gradients;

[0123] Step B1-4: Details, using detail enhancement filtering method to make the details in the image more obvious;

[0124] Step B2: Optimize the roughly processed image pair dataset. This optimisation process expands the image pair dataset, enabling the detection method to see a wider range of target types, resulting in higher detection accuracy and a more robust detector. This optimisation process includes rotation, flipping, and brightness adjustment, all implemented using OpenCV classes in Python.

[0125] Step B2 includes:

[0126] Step B2-1: Rotate the image dataset using cv2.warpAffine in OpenCV;

[0127] Step B2-2: Flip the image dataset using cv2.flip in OpenCV.

[0128] Step B2-3: Adjust the brightness and contrast of the image dataset. The brightness and contrast adjustment is a simple linear transformation of the pixels. The output pixel value output_pixel is calculated using the formula output_pixel = alpha * (input_pixel) + beta, where input_pixel is the input pixel, alpha is the transparency factor, and beta is the brightness bias factor.

[0129] The output of step B2 is the processed infrared image and visible light image.

[0130] Step C: Construct a module to encode the infrared image and the visible light image separately. That is, pass the infrared image through the designed infrared encoder to extract infrared features; pass the visible light image through the designed visible light encoder to extract visible light features in both vertical and horizontal spatial directions.

[0131] Step C includes:

[0132] Step C1: Construct an infrared encoder to obtain infrared image information and neuron weights, and then obtain infrared image features;

[0133] The infrared encoder consists of an infrared mini-residual module, which has a three-layer structure consisting of a max pooling layer and two consecutive 1×1 convolutional layers. The infrared image first undergoes max pooling, which selects and retains the maximum pixel value within a specified region. The infrared image then undergoes two consecutive 1×1 convolutional layers to linearly combine the 24 infrared image features. This allows these 24 features to be interconnected, facilitating subsequent fusion operations and ultimately yielding the infrared image information.

[0134] The neuron weights are calculated by the energy function. First, the energy function factor is calculated.

[0135] The calculation formula is:

[0136]

[0137] Where M = H × W is the number of neurons on the channel to be calculated, H and W are the length and width of the image feature map respectively, and x i It is a feature map input feature of the neighboring neurons of the current neuron, i is the number of feature maps, and i ranges from 1 to M.

[0138] Then calculate the energy function factor The calculation formula is:

[0139]

[0140] Energy function Energy function factor The energy function is used to measure the differentiation between neurons. The lower the energy, the greater the difference between the current neuron and the surrounding neurons. The calculation formula is:

[0141]

[0142] Where λ is a hyperparameter, and the default value is set to 10 in this invention. -4 , t is the single channel input feature of the target neuron.

[0143] The neuron weights are calculated through the energy function The inverse of is

[0144] Multiply the infrared image information and the neuron weight pixel by pixel to obtain the infrared image feature F IR .

[0145] Step C2: construct a visible light encoder, input a visible light image, and obtain visible light image features;

[0146] Step C2 includes:

[0147] Step C2-1: Input the visible light image outputted in step B2 into a visible light encoder to obtain detail feature information of the visible light image.

[0148] The visible light encoder is composed of a visible light mini residual module, which is a three-layer structure consisting of a maximum pooling layer, a 1×1 convolution layer, and a 3×3 convolution layer. The 3×3 convolution layer can increase the receptive field, making it easier to extract the detailed feature information of the visible light image. The 1×1 convolution layer enables these visible light image detail features to be interconnected, which is more conducive to subsequent fusion operations. The visible light image first passes through the maximum pooling layer, then the 1×1 convolution layer, and finally the 3×3 convolution layer to obtain the visible light image detail feature information X(c,s), where c is the vertical direction of the visible light image feature and s is the horizontal direction of the visible light image feature.

[0149] Step C2-2: Use the coordinate attention method to calculate the visible light image feature weights in the vertical and horizontal directions, and then obtain the weight g(c) in the vertical spatial direction and the weight g(s) in the horizontal spatial direction;

[0150] Step C2-2 includes:

[0151] Step C2-2-1: Extract visible light image features from the vertical direction using a global average pooling operation, set the number of output feature maps to 1, and obtain a one-dimensional vertical vector V1;

[0152] Step C2-2-2: Extract visible light image features from the horizontal direction using a global average pooling operation, set the number of output feature maps to 1, and obtain a one-dimensional horizontal vector V2;

[0153] Step C2-2-3: Concatenate the one-dimensional vertical vector V1 obtained in step C2-2-1 and the one-dimensional horizontal vector V2 obtained in step C2-2-2 according to the channel dimension to obtain a combined vector V;

[0154] Step C2-2-4: Use 1×1 convolution to change the channel of the combined vector V to 8, and obtain the transpose V' of the combined vector;

[0155] Step C2-2-5: The transpose V' of the combined vector is normalized by batch normalization to calculate the mean and variance, and then the intermediate vector VS is obtained by the hswish nonlinear activation function layer.

[0156] Step C2-2-6: Slice VS according to the channel dimension and divide it into two parts: vertical tangent vector VS1 and horizontal tangent vector VS2;

[0157] Step C2-2-7: VS1 is subjected to 1×1 convolution to adjust the number of channels to 24, and then normalized by the Sigmoid activation function. The output of the Sigmoid function is between (0, 1), and the weight is between (0, 1), with a limited range. This is used as the output layer to obtain g(c);

[0158] Step C2-2-8: VS2 is passed through a 1×1 convolution to adjust the number of channels to 24, and then normalized through the Sigmoid activation function. The output of the Sigmoid function is between (0, 1) and the weight range is limited. It is used as the output layer to obtain g(s).

[0159] Step C2-3: Multiply the results of step C2-1 and step C2-2 pixel by pixel to obtain the visible light image feature F VS =X(c,s)×g(c)×g(s).

[0160] Step C3: The infrared image feature F obtained in step C1 IR and the visible light image feature F obtained in step C2 VS Splicing to get the combined feature X F .

[0161] Step D: Construct a dual-path gated fusion network; calculate the infrared feature weights and visible light feature weights through the constructed gated fusion network, adaptively adjust the weight distribution of the two-path features, realize cross-modal feature fusion, and ultimately achieve dual-mode feature complementarity, obtain fused features, and complete gated fusion network training;

[0162] The input of step D is the combined feature X generated in step C3 F , the output of step D is the fused feature F M , step D comprises:

[0163] Step D1: Combine the features X F Slices are divided into infrared feature blocks and visible light feature tiles

[0164] Step D2: Generate two gated networks to calculate infrared probability features and visible light probability features;

[0165] Step D2 includes:

[0166] Step D2-1: Construct the infrared image gating network G1 and the visible light image gating network G2. The gating network consists of two layers: a 1×1 convolutional layer and a softmax layer.

[0167] Step D2-1 includes:

[0168] Step D2-1-1: Through the 1×1 convolution layer, the number of channels is reduced to 128 to obtain the infrared weight Q1;

[0169] Step D2-1-2: Through the 1×1 convolution layer, the number of channels is reduced to 128, and the visible light weight Q2 is obtained;

[0170] Step D2-1-3: Multiply pixel by pixel with Q1, that is Denoted as weighted infrared features;

[0171] Step D2-1-4: Multiply Q2 pixel by pixel, that is Denoted as weighted visible light feature;

[0172] Step D2-1-5: Use the Softmax layer to calculate the probability of the weighted infrared feature, and record the infrared probability feature S in the infrared image gating network G1 IR , the calculation process is:

[0173]

[0174] Step D2-1-6: Use the Softmax layer to calculate the probability of weighted visible light features, and record the calculation of the visible light probability feature S in the visible light image gating network G2 VS , the calculation process is:

[0175]

[0176] Where e represents the exponential function.

[0177] Step D3: Based on the infrared probability feature S generated in step D2 IR and the visible light probability characteristic S VS , the adaptive infrared feature Y1 and the adaptive visible light feature Y2 are calculated. That is, the pixel weights on each feature map are calculated, and the two original input features are corrected to obtain the adaptive infrared feature Y1 and the adaptive visible light feature Y2;

[0178] Step D3-1: Infrared probability feature S in infrared image gating network G1 IR With infrared feature tiles Multiply element by element and adaptively weight to obtain the adaptive infrared feature Y1. The calculation formula is:

[0179] Step D3-2: The visible light probability feature S in the visible light image gating network G2 is VS Visible light feature tiles Multiply element by element and adaptively weight to obtain the adaptive visible light feature Y2, which is calculated as follows:

[0180] Step D4: Add the results of step D3-1 and step D3-2, Y1+Y2;

[0181] Step D5: The infrared image feature F obtained in step C1 IR and the visible light image feature F obtained in step C2 VS Downsampling pooling;

[0182] Step D5-1: Transform the infrared image feature F IR , through the maximum pooling operation, the pooled infrared feature F is obtained IRc ;

[0183] Step D5-2: Transform the visible light image feature F VS , through the maximum pooling operation, the pooled visible light feature F is obtained IRc ;

[0184] Step D6: Add the result Y1+Y2 of step D4 and the result F of step D5-1 IRc and the result F of D5-2 IRc The fusion feature F is obtained by splicing according to the channel dimension M ;

[0185] Step E: Fusion feature F M Input into the third layer of the baseline method YOLOv5-s network to obtain the final detection network.

[0186] Step F: Input the image pair dataset processed in step B into the final detection network to obtain the detection results. The image pair dataset can be stored in a fixed local path and then input into the final detection network for processing, or it can be directly collected into the final detection network for processing after step B.

[0187] The present invention also includes a target detection device based on the method of the present invention, including: a dual-mode image acquisition unit 1, a calculation unit 2, and a display unit 3.

[0188] The dual-mode image acquisition unit 1 acquires infrared images and visible light images, and then sends them to the calculation unit 2. The calculation unit 2 completes the dual-mode image recognition and then sends the result to the display unit 3.

[0189] The dual-mode image acquisition unit 1 includes a sealed box 11 , a beam splitter 12 , an infrared camera 13 , a three-axis platform 14 , and a visible light camera 15 .

[0190] The size of the sealed box 11 is 80 cm long, 40 cm wide and 80 cm high. A light opening 111 is opened at the lower part of one side of the sealed box. The light opening is circular and has a diameter of 15 cm. The beam splitter 12 is fixedly installed on the bottom surface of the sealed box 11, with the lower part close to the light opening 111 and an angle of 40 degrees with the bottom surface. The upper part of the beam splitter 12 is fixed to the side wall of the sealed box 11, and the upper fixed part of the beam splitter 12 extends upward to form a fixed bracket. The beam splitter 12 is made of zinc-coated silicon wafer. The infrared camera 13 is connected to the fixed bracket of the beam splitter 12 and is vertically Below, the lens of the infrared camera 13 is directly facing the center of the beam splitter 12. The model of the infrared camera 13 is FLIR-A35; the three-axis platform 14 is fixedly installed on the side of the closed box 11, and there is a movable rod 141 at the bottom. The position of the movable rod 141 in the box can be adjusted by the knob; the visible light camera 15 is installed at the bottom of the movable rod 141 of the three-axis platform 14, and the lens is facing the light opening 111 of the closed box 11. The position of the visible light camera 15 can be adjusted by adjusting the knob on the three-axis platform 14. The model of the visible light camera 15 is PointGrey Flea3.

[0191] The computing unit 2 is a computer or an image processor FPGA, which is configured with an implementation program corresponding to the method of the present invention;

[0192] The display unit 3 is a display or other sound and light alarm devices, such as a warning light or a warning bell.

[0193] The optical axes of the two cameras are aligned in parallel. The specific method is: adjust the knob on the three-axis platform 14 so that the optical axis of the visible light camera 15 at the lower part of the movable rod 141 coincides with the optical axis of the infrared camera 13.

[0194] External light passes through daylighting port 111 and is split into two paths by beam splitter 12. One path is infrared, which is reflected by the beam splitter and enters infrared camera 13. The other path is visible light, which passes through beam splitter 12 and reaches visible light camera 15. Infrared and visible light cameras 13 and 15 use global shutter synchronization to capture images. The captured infrared and visible light images are transmitted to computing unit 2 via wired or wireless means. Computing unit 2 processes the input infrared and visible light images using the aforementioned inventive method and outputs the results to display unit 3 via cable or wireless means.

[0195] To verify the effectiveness of the method of the present invention, this method is compared with the prior art YOLOv5-s method, and the entire process is carried out under exactly the same test environment.

[0196] The experimental environment was set to Ubuntu 16.04, the CPU was i5-8400, the GPU was NVIDIA GeForce GTX1080Ti (video memory 11GB), the CUDA and CUDNN versions were 10.0 and CUDNN 7.4.2 respectively; the experimental platform was Python and PyTorch; the network training process parameters were set, and the SGD optimizer was used to iteratively update the network parameters during training. The momentum parameter was set to 0.937, the batch size was set to 8, and a total of 150 epochs were trained.

[0197] The specific testing process of the present invention is as follows:

[0198] Step A: using the bimodal dataset image established by the present invention;

[0199] Step B: Input into the designed dual encoder;

[0200] Step C: input into the dual-path gate fusion network;

[0201] Step D: Output the third layer of the baseline method YOLOv5-s network;

[0202] Step E: Obtain detection accuracy and speed;

[0203] Step F: Output visualization results;

[0204] The specific process of the comparison method is as follows:

[0205] Step A: using the visible light image in the dual-modal dataset established by the present invention alone;

[0206] Step B: Input to the YOLOv5-s detector;

[0207] Step C: Obtain detection accuracy and speed;

[0208] Step D: Output visualization results;

[0209] Step E: Using the infrared image in the dual-modal data set established by the present invention alone.

[0210] Step F: Input to the YOLOv5-s detector;

[0211] Step G: Obtain detection accuracy and speed;

[0212] Step H: Output visualization results;

[0213] Table 1 shows the results of the comparative experiment. The first row shows the results of the YOLOv5-s method using visible light images as input, the second row shows the results of the YOLOv5-s method using infrared images as input, and the third row shows the detection accuracy and speed of the present invention using both visible light and infrared images as input. The experimental results show that the present invention improves over the baseline method in both accuracy and speed. The evaluation metrics for this experiment are: accuracy 0.5:0.95, precision 0.5, and speed. Accuracy 0.5:0.95 represents the average accuracy calculated at intervals of 0.05 for confidence thresholds from 0.50 to 0.95, while accuracy 0.5 represents the accuracy when the confidence threshold is set to 0.5.

[0214] Table 1

[0215]

[0216] Comparative experimental results on the KAIST dataset. The KAIST pedestrian dataset includes a total of 95,328 images, each of which contains two versions: RGB color image and infrared image.

[0217] Visualization results such as Figure 3 As shown, Figure 3 (a) represents the target that should be detected, Figure 3 (b) is the detection result of the YOLOv5-s method. Figure 3 (c) is the detection result of the method of the present invention. Obviously, the method of the present invention better detects all targets, and one of the targets is in the dark, which proves the robustness of the method.

[0218] Compared to existing technologies, this method employs an embedded gated fusion network, enabling the model to determine the contribution of both modal images to detection in different scenarios. It also incorporates multi-task learning to adaptively adjust weight distribution, achieving cross-modal feature fusion. Furthermore, this method demonstrates robustness in complex environments, such as darkness. Furthermore, it can also detect targets in single-input visible light or infrared images, significantly improving detection performance compared to the baseline method, YOLOv5-s.

Claims

1. A target detection method based on a dual-modal fusion network, comprising: Step A: Acquire an image pair dataset, i.e., use an infrared imaging device to capture an infrared image, use a visible light imaging device to capture a visible light image, and then align the two images to obtain an image pair dataset; Step B: Establish a bimodal image dataset, that is, perform data enhancement processing on the registered image dataset to form a bimodal dataset; Step C: construct modules to encode infrared images and visible light images respectively; Step D: Construct a dual-path gated fusion network, that is, calculate the infrared feature weights and visible light feature weights through the constructed gated fusion network, adaptively adjust the weight distribution of the two-path features, obtain the fusion feature, and complete the gated fusion network training; Step E: Input the fused features into the third layer of the YOLOv5-s network of the baseline method to obtain the final detection network; Step F: Input the image pair dataset processed in step B into the final detection network to obtain the detection results; in: Step A includes: Step A1: Acquire infrared and visible light images. This involves taking photos using a visible light camera and an infrared camera in fully enclosed conditions. Hardware is used to align the camera optical axes so that the image ranges of the two photos are roughly consistent. Step A2: performing image registration on the acquired infrared image and visible light image, i.e., using an alignment strategy and adjusting the image size to achieve image registration; Step B includes: Step B1: performing rough processing on the image pair data set to obtain a rough processed image pair data set, wherein the rough processing method includes: enhancing edges, sharpening, smoothing, and detail; Step B2: arranging the roughly processed image data set, including rotation, flipping, and brightness adjustment; Step C includes: Step C1: Construct an infrared encoder to obtain infrared image information and neuron weights, and then obtain infrared image features; Step C2: construct a visible light encoder, input a visible light image, and obtain visible light image features; Step C3: combining the infrared image features obtained in step C1 and the visible light image features obtained in step C2 to obtain a combined feature; The input of step D is the combined features generated in step C3, and the output of step D is the fused features after fusion; Step D includes: Step D1: Divide the combined feature slices into infrared feature blocks and visible light feature blocks; Step D2: Generate two gated networks to calculate infrared probability features and visible light probability features; Step D3: Calculating an adaptive infrared feature and an adaptive visible light feature based on the infrared probability feature and the visible light probability feature generated in step D2; Step D4: adding the adaptive infrared feature and the adaptive visible light feature; Step D5: Downsample and pool the infrared image features obtained in step C1 and the visible light image features obtained in step C2; Step D5 includes: Step D5-1: Infrared image features , through the maximum pooling operation, the pooled infrared features are obtained ; Step D5-2: Visible light image features , through the maximum pooling operation, the pooled visible light features are obtained ; Step D6: Concatenate the result of step D4, the pooled infrared features, and the pooled visible light features according to the channel dimension to obtain a fused feature.

2. The target detection method based on a dual-modal fusion network according to claim 1, wherein: Step B1 includes: Step B1-1: Enhance image edges; Step B1-2: Sharpening, using contour filtering method to extract all contour information in the image; Step B1-3: Smoothing, using a smoothing filter method to make the image brightness change smoothly and gradually, reducing sudden gradients; Step B1-4: Details, using detail enhancement filtering method to make the details in the image more obvious; Step B2 includes: Step B2-1: Rotate the image dataset; Step B2-2: flip the image dataset; Step B2-3: Adjust the brightness and contrast of the image dataset; In step C1, the infrared encoder is composed of an infrared mini residual module. The infrared mini residual module has a three-layer structure consisting of a maximum pooling layer and two consecutive 1×1 convolutional layers; Step C2 includes: Step C2-1: Input the visible light image output from step B2 into a visible light encoder to obtain detailed feature information of the visible light image; the visible light encoder is composed of a visible light mini residual module, which is a three-layer structure consisting of a maximum pooling layer, a 1×1 convolutional layer, and a 3×3 convolutional layer; Step C2-2: Use the coordinate attention method to calculate the visible light image feature weights in the vertical and horizontal directions, and then obtain the weights in the vertical spatial direction and the weights in the horizontal spatial direction; Step C2-3: Multiply the results of step C2-1 and step C2-2 pixel by pixel to obtain visible light image features; Step D2 includes: Step D2-1: Construct infrared image gating network and visible light image gating network. The gating network includes two layers of modules, namely Convolutional layer and Softmax layer; Step D3 includes: Step D3-1: Multiply the infrared probability feature in the infrared image gating network and the infrared feature block element by element and adaptively weight them to obtain an adaptive infrared feature; Step D3-2: Multiply the visible light probability features in the visible light image gating network and the visible light feature blocks element by element and adaptively weight them to obtain adaptive visible light features.

3. The target detection method based on a dual-modal fusion network according to claim 2, characterized in that: Step A1: The visible light camera model is PointGrey Flea3, and the infrared camera model is FLIR-A35; In step A2-3, the spatial resolution of the visible light camera is 640×480 pixels, with a vertical field of view of 103.6°. The thermal infrared camera has a spatial resolution of 320×256 pixels, with a vertical field of view of 39°. By reducing the boundary area of ​​the visible light image, a visible light image registered with the original thermal infrared image is obtained in the aligned image domain, resulting in a dataset of registered image pairs. The image size is 640×512, with a resolution of 320×256 pixels. The rough processing method in step B1 is implemented using the PIL class in Python; The sorting process in step B2 is implemented using the OpenCV class in Python; The brightness and contrast adjustment in step B2-3 is a simple linear transformation of the pixel. The output pixel value output_pixel is calculated using the formula output_pixel=alpha*(input_pixel)+beta, where input_pixel is the input pixel, alpha is the transparency factor, and beta is the brightness bias factor. In step C1, the infrared image first undergoes a maximum pooling operation, and then passes through two consecutive 1×1 convolutional layers to linearly combine the 24 infrared image features; Energy function factor The calculation formula is: in is the number of neurons on the channel to be calculated, H and W are the length and width of the image feature map respectively, It is a feature map input feature of the neighboring neurons of the current neuron, is the number of feature maps, From 1 to M; Energy function factor , the calculation formula is: Energy function Energy function factor We get the energy function The calculation formula is: in is a hyperparameter, the default value is , Input features of a single channel to the target neuron; The neuron weights are calculated through the energy function The reciprocal of is ; Multiply the infrared image information and the neuron weight pixel by pixel to obtain the infrared image features; In step C2-1, the visible light image first passes through the maximum pooling layer, then through the 1×1 convolution layer, and finally through the 3×3 convolution layer to obtain the detailed feature information of the visible light image; Step C2-2 includes: Step C2-2-1: Extract visible light image features from the vertical direction using global average pooling, set the number of output feature maps to 1, and obtain a one-dimensional vertical vector; Step C2-2-2: Extract visible light image features from the horizontal direction using a global average pooling operation, set the number of output feature maps to 1, and obtain a one-dimensional horizontal vector; Step C2-2-3: Concatenate the one-dimensional vertical vector obtained in step C2-2-1 and the one-dimensional horizontal vector obtained in step C2-2-2 according to the channel dimension to obtain a combined vector; Step C2-2-4: Use 1×1 convolution to change the channel of the combined vector to 8, and obtain the transpose of the combined vector; Step C2-2-5: Apply batch normalization to the transpose of the combined vector, calculate the mean and variance, and then pass it through the hswish nonlinear activation function layer to obtain the intermediate vector; Step C2-2-6: Slice the intermediate vector into two parts: a vertical tangent vector and a horizontal tangent vector according to the channel dimension; Step C2-2-7: The vertical tangent vector is passed through a 1×1 convolution to adjust the number of channels to 24, and then normalized through the Sigmoid activation function to obtain the weight in the vertical spatial direction; Step C2-2-8: The horizontal tangent vector is passed through a 1×1 convolution to adjust the number of channels to 24, and then normalized through the Sigmoid activation function to obtain the weight in the horizontal spatial direction; Step D2-1 includes: Step D2-1-1: Pass the infrared feature block through Convolutional layer, set the number of channels to 128 and get the infrared weight; Step D2-1-2: Pass the visible light feature block through Convolutional layer, set the number of channels to 128 and obtain the visible light weight; Step D2-1-3: Multiply the infrared feature block and the infrared weight pixel by pixel, and record it as the weighted infrared feature; Step D2-1-4: Multiply the visible light feature block and the visible light weight pixel by pixel, and record it as the weighted visible light feature; Step D2-1-5: Use the Softmax layer to calculate the probability of weighted infrared features. The infrared probability feature in the infrared image gating network is recorded as , the calculation process is: in, For infrared feature blocks, is the visible light feature block, Q1 is the infrared weight, Q2 is the visible light weight, and e represents the exponential function; Step D2-1-6: Use the Softmax layer to calculate the probability of weighted visible light features, and remember to calculate the visible light probability features in the visible light image gating network G2 , the calculation process is: Here, e represents the exponential function.

4. A target detection device using the method according to claim 1, comprising: a dual-mode image acquisition unit (1), a computing unit (2), and a display unit (3), wherein the dual-mode image acquisition unit (1) acquires an infrared image and a visible light image, and then sends the images to the computing unit (2), the computing unit (2) performs dual-mode image recognition, and then sends the results to the display unit (3), characterized in that: The dual-mode image acquisition unit 1 includes a sealed box (11), a beam splitter (12), an infrared camera (13), a three-axis platform (14), and a visible light camera (15); a light opening (111) is opened at the lower part of one side of the sealed box (11); the beam splitter (12) is fixedly mounted on the bottom surface of the sealed box (11), and the lower part is close to the side of the light opening (111) and has an angle with the bottom surface; the upper part of the beam splitter (12) is fixed to the side wall of the sealed box (11); the upper fixed part of the beam splitter (12) extends upward to form a fixed bracket; the infrared camera (13) is connected to the fixed bracket of the beam splitter (12) and is vertically downward. The lens of the infrared camera (13) is directly facing the center position of the beam splitter (12); the three-axis platform (14) is fixedly mounted on the side of the sealed box (11), and a movable rod (141) is provided at the lower part thereof, and the position of the movable rod (141) in the box can be adjusted; the visible light camera ( 15) is installed at the lower part of the movable rod (141) of the three-axis platform (14), and the lens faces the light opening (111) of the closed box (11), and the position of the visible light camera (15) can be adjusted; the computing unit (2) is configured with an implementation program corresponding to the target detection method; the display unit (3) is a display; external light passes through the light opening (111) and is divided into two paths through the beam splitter (12), one path is the infrared thermal band, which is reflected by the beam splitter and enters the infrared camera (13), and the other path is the visible light band, which is transmitted through the beam splitter (12) to the visible light camera (15), and the infrared camera (13) and the visible light camera (15) use global shutter synchronization control to collect images; the collected infrared image and visible light image are transmitted to the computing unit (2), and the computing unit (2) processes the input infrared image and visible light image according to the method described in claim 1, and finally outputs the result to the display unit (3).

5. The target detection device according to claim 4, wherein: The dimensions of the sealed box (11) are 80 cm in length, 40 cm in width, and 80 cm in height; the light opening (111) is circular with a diameter of 15 cm; the beam splitter (12) is at an angle of 40 degrees to the bottom surface; the position of the movable rod (141) in the box can be adjusted by a knob on the three-axis platform (14), thereby adjusting the position of the visible light camera (15); the collected infrared image and visible light image are transmitted to the computing unit (2) via a wired or wireless method, and the computing unit (2) outputs the result to the display unit (3) via a cable or wirelessly.

6. The target detection device according to claim 5, wherein: The beam splitter (12) is made of a zinc-coated silicon wafer; the infrared camera (13) is a FLIR-A35; the computing unit (2) is a computer or an image processor FPGA; and the display unit (3) is a warning light or a warning bell.

Images