Lightweight rgb-ir fusion small target detection method

Abstract

A lightweight RGB-IR fusion small target detection method, comprising the following steps: S100: preprocessing the image to be detected, wherein the preprocessing refers to aligning the R and RGB images by using homography transformation, obtaining a homography transformation matrix, taking the RGB image as the first three channels, taking the R image as the fourth channel, and composing a four-channel image tensor; S200: inputting the image tensor obtained after preprocessing into a lightweight RGB-IR fusion small target detection network for target detection to obtain the category, position, width and height of the image to be detected. The method has the characteristics of strong detection robustness, small calculation amount, real-time operation on the embedded end with limited computing power, and can be widely used in the field of monitoring and security.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision, pattern recognition and artificial intelligence, and in particular relates to a lightweight RGB-IR fusion method for small target detection. Background Technology

[0002] With the advent of the intelligent era, people have increasingly diverse requirements for the application scenarios of vision systems. Visible light cameras are particularly sensitive to lighting conditions, and have limitations in low-light or backlit environments. Infrared cameras have low resolution and blurry edges, and even under ideal lighting conditions, their imaging details are inferior to those of visible light cameras. Therefore, RGB-IR fusion detection demonstrates significant advantages and value, and has important applications in drones, smart homes, robotics, medical and defense fields.

[0003] The general steps of traditional video stream object detection are as follows: Image detection is performed frame-by-frame on the input video stream. First, a feature extraction module outputs a feature map of the image. Then, a feature fusion module fuses the extracted features to obtain a feature map that combines low-dimensional and high-dimensional information. Finally, regression prediction is performed on the feature map to obtain the coordinate parameters of the detection box and the confidence score of the object detection category. The result is then returned to the input image. Currently, mainstream object detection algorithms such as YOLO lack specific design for RGB-IR fusion detection, making it impossible to utilize both infrared and visible light information simultaneously. Furthermore, current fusion detection models generally have excessively large parameter counts and computational demands, making it impossible to achieve high frame rate real-time detection on embedded systems. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention discloses a lightweight RGB-IR fusion method for small target detection, comprising the following steps:

[0005] S100: Preprocess the image to be detected. The preprocessing refers to aligning the IR and RGB images using homography transformation to obtain a homography transformation matrix. The RGB image is used as the first three channels and the IR image is used as the fourth channel to form a four-channel image tensor.

[0006] S200: Input the preprocessed image tensor into a lightweight RGB-IR fusion small target detection network for target detection to obtain the category, position, width, and height of the image to be detected.

[0007] The above technical solution enables the detection of small targets in complex environments, featuring accurate detection and positioning, strong robustness, high precision, and support for real-time detection. This method is applicable not only to images but also to dynamic small target detection in RGB-IR video streams. It can be widely used in fields such as surveillance and security, where distant people and vehicles occupy relatively few pixels in the image, and are thus termed small targets. This method has low computational requirements, enabling real-time fusion and small target detection on embedded devices with limited computing power. Attached Figure Description

[0008] Figure 1 is a flowchart of a lightweight RGB-IR fusion small target detection method provided in an embodiment of the present invention;

[0009] Figure 2 This is a schematic diagram of the structure of a lightweight RGB-IR fusion small target detection network provided in one embodiment of this disclosure;

[0010] Figure 3 This is a schematic diagram of the Ghost convolution principle referenced in one embodiment of this disclosure;

[0011] Figure 4 This is a schematic diagram of the SE attention mechanism referenced in one embodiment of this disclosure;

[0012] Figure 5 This is a schematic diagram of the C3SE module referenced in one embodiment of this disclosure;

[0013] Figure 6 This is a schematic diagram of the GCR module referenced in one embodiment of this disclosure;

[0014] Figure 7 This is a schematic diagram of the SE-BottleNeck module referenced in one embodiment of this disclosure;

[0015] Figure 8 This is a schematic diagram of the SE attention mechanism referenced in one embodiment of this disclosure;

[0016] Figure 9 This is a schematic diagram of the SPPF module referenced in one embodiment of this disclosure;

[0017] Figure 10 This is a schematic diagram of an application device used in the field of surveillance and security according to an embodiment of this disclosure;

[0018] Figure 11 This is a comparison diagram of the effects of RGB-IR fusion detection and pure RGB detection on a practical application device according to an embodiment of this disclosure. Detailed Implementation

[0019] To enable those skilled in the art to understand the technical solutions disclosed in this invention, the following will describe them in conjunction with embodiments and related appendices. Figures 1 to 11 The technical solutions of various embodiments are described herein. The described embodiments are some, but not all, of the embodiments of the present invention.

[0020] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. Those skilled in the art will understand that the embodiments described herein can be combined with other embodiments.

[0021] See Figure 1 In one embodiment, the present invention discloses a lightweight RGB-IR fusion small target detection method, comprising the following steps:

[0022] S100: Preprocess the image to be detected. The preprocessing refers to aligning the IR and RGB images using homography transformation to obtain a homography transformation matrix. The RGB image is used as the first three channels and the IR image is used as the fourth channel to form a four-channel image tensor.

[0023] S200: Input the preprocessed image tensor into a lightweight RGB-IR fusion small target detection network for target detection to obtain the category, position, width, and height of the image to be detected.

[0024] In this embodiment, the method includes two steps: aligning RGB and IR images through homography transformation and performing target detection based on a lightweight RGB-IR fusion small target detection network. This embodiment can detect small targets in RGB-IR images in real time.

[0025] The image to be detected is preprocessed, which refers to aligning the IR and RGB images using homography transformation: First, the intrinsic and relative extrinsic parameters of the camera are obtained using Zhang's calibration method. Then, the homography transformation matrix at a fixed distance is inferred using the camera pinhole model. After obtaining the matrix, the infrared image is homography transformed to achieve alignment between the RGB and IR images. Finally, the RGB image is used as the first three channels and the IR image is used as the fourth channel to form a four-channel image tensor.

[0026] The RGB-IR small object detection algorithm described above belongs to a subclass of general object detection tasks and can utilize classic object detection models. General object detection algorithms mainly fall into two categories: single-stage algorithms, represented by YOLO and SSD, are simpler, faster, and more suitable for practical applications; two-stage algorithms, represented by Faster R-CNN, are more complex, offer higher accuracy, but are slower. Object detection algorithms can also be categorized based on whether they require prior anchor boxes: anchor-based and anchor-free. Among anchor-based algorithms, the YOLO series is the most representative.

[0027] This method, considering the need for lightweight deployment, designs and implements a lightweight RGB-IR fusion small target detection method, which belongs to a single-stage target detection network. The RGB-IR fusion-based small target detection network is structurally based on the YOLOv5 framework, with improvements and upgrades to key modules. Specifically, homography transformation preprocessing is performed at the input to align the image, and the fused 4-channel tensor is used as the input to the neural network. The connection structure is improved: a GCR module is added after the C3 module in layer 6 of the baseline network to adjust its channel count, and an Up-Sample module is used to upsample it to obtain a 32*160*160 output, which is then concatenated with the same-sized output of the GCR module in layer 2. Finally, it passes through a C3 module to obtain the small target feature layer. This feature layer has a size of 32*160*160, and only a 4x downsampling is performed on the 640*640 input image. Any target larger than 4 pixels will not suffer direct feature loss due to downsampling. Compared to the baseline network Yolov5n with a maximum feature map size of 64*80*80, this feature layer is more conducive to the detection of small targets. Ghost convolution is used to lighten the network, and SE attention mechanism is added to the C3 module to form the C3SE module, which realizes the weight allocation of the feature channel level.

[0028] In another embodiment, the preprocessing in step S100 is specifically as follows: using the RGB image as a reference, the IR image is subjected to homography transformation, and after the homography-transformed IR image is aligned with the RGB image, the IR image and the RGB image are combined into a four-channel tensor, wherein the first three channels of the RGB image are copied, and the homography-transformed IR image is copied to the fourth channel. The combined tensor will be used as the input of the lightweight RGB-IR fusion small target detection network.

[0029] In another embodiment, the lightweight RGB-IR fusion small target detection network in step S200 includes an Upsample module, a Concat module, a GCR convolution module, an SPPF module, a MaxPool max pooling module, a small target detection layer, and a C3SE module improved using the SE attention mechanism.

[0030] In this embodiment, as Figure 2 As shown, the Upsample and Concat modules are used to combine and connect the GCR convolution module, the C3SE attention mechanism convolution module, and the SPPF spatial pyramid pooling module. The Upsample module performs nearest-neighbor sampling on the feature map, expanding its size; the Concat module concatenates the feature maps at the channel level, merging features; the GCR module combines Ghost convolution and ReLU activation functions to achieve lightweight non-linear feature extraction; the C3SE module combines multiple GCR modules and the SE attention mechanism to achieve more complex feature extraction; and the SPPF module combines the GCR module and the MaxPool max pooling module, using cascaded max pooling modules to combine features from different receptive fields, enhancing the detection capability for targets at different scales.

[0031] In this embodiment, the size of the four-channel tensor fed into the network is m×n (for an image resolution of 640x640 in one instance). It then undergoes three stages of feature extraction, resulting in feature maps downsampled by 4x, 8x, and 16x, respectively. For an example, the image sizes are 160x160, 80x80, and 40x40, respectively. Each pixel in each feature map layer corresponds to a 4x4, 8x8, or 16x16 region in the original image, respectively. Next, K prediction boxes are generated for each pixel in these three feature maps based on pre-set prior box parameters. Each bounding box requires six parameters: the top-left corner coordinates (horizontal and vertical), width and height, and the target category (person, vehicle) and its confidence score. Therefore, after these three feature maps pass through the regression head module, for a single instance, we will obtain outputs of 160×160×K×(5+2), 80×80×K×(5+2), and 40×40×K×(5+2), which represent the positions of 40×40×K predicted bounding boxes and their probabilities of belonging to different categories. During training, these predictions and ground truth annotations are used to calculate the loss. During prediction, a non-maximum suppression algorithm is used to process these predicted bounding boxes to obtain the final prediction result.

[0032] In this embodiment, the small target detection layer is implemented by optimizing network connections and extracting shallow features, such as... Figure 2As shown, a GCR module was added after the C3 module in layer 6 of the baseline network to adjust its channel count. An Up-Sample module was used to upsample it, resulting in a 32*160*160 output. This output was then concatenated with the same-sized output from the GCR module in layer 2, and finally passed through a C3 module to obtain a small target feature layer. This feature layer, with a size of 32*160*160, only undersampled a 640*640 input image by a factor of 4. Any target larger than 4 pixels will not suffer direct feature loss due to downsampling. Compared to the baseline network Yolov5n's maximum feature map of 64*80*80, this feature layer is more advantageous for small target detection.

[0033] In another embodiment, Ghost convolutions are used to achieve model lightweighting.

[0034] In this embodiment, to achieve model lightweighting, Ghost convolutions are used instead of traditional convolutions in the model, such as... Figure 3 As shown, the input feature map is first subjected to pointwise convolution, i.e., a convolution with a kernel size of 1, to obtain its intrinsic feature map. The intrinsic feature map represents the core information in the input feature map and is difficult to generate using other feature maps. In this embodiment, the number of channels in the intrinsic feature map is half that of the input feature map. After extracting the intrinsic feature map, it is used to generate more feature maps through depthwise separable convolution. Finally, the feature maps generated by the depthwise separable convolution are concatenated with the intrinsic feature map before output.

[0035] The idea behind Ghost convolution is to use one set of features to generate another set of features. This method can significantly reduce computation and parameter count without changing the size of the input and output feature maps. After using Ghost convolution, the number of model parameters in one instance was reduced from 1.76M to 1.06M, and the computational cost in another instance was reduced from 4.2 GFLOPS to 2.8 GFLOPS, enabling real-time object detection on computationally limited embedded devices.

[0036] In another embodiment, on the one hand, due to the different imaging principles of infrared and visible light cameras, RGB and IR images may contradict each other under certain environments. On the other hand, Ghost convolution is used, where the depthwise separable convolution results in relatively independent output feature maps at the channel level. A SE attention mechanism is used to achieve weight allocation at the channel level. The SE attention mechanism consists of two linear layers. The first linear layer further compresses the vector and then uses the ReLU activation function to achieve non-linear feature extraction. The second linear layer restores the number of channels to be the same as the input. Finally, the Sigmoid activation function is used to transform the numerical range to 0-1, obtaining a weight vector, which is then multiplied with the original feature map to obtain a feature map with redistributed weights, thus implementing the attention mechanism at the feature channel level. The specific process is as follows: Figure 4 As shown in the figure: F sq The squeezing operation is implemented using global average pooling, which compresses the feature map into a vector, as shown in the following formula:

[0037]

[0038] In the formula: H and W represent the height and width of the input feature map, x c (i, j) represents the input feature located at (i, j).

[0039] F ex This represents the activation operation, which uses convolution to adjust the vector, as shown in the following formula:

[0040] F ex (z, W)=σ(g(z, W))=σ(W2δ(W1z)),

[0041] In the formula: z represents the vector obtained from the previous compression operation, W1 and W2 represent the weights of the first and second linear layers, which are obtained through learning, δ represents the ReLU activation function to achieve non-linear feature extraction, and σ represents the Sigmoid activation function to limit the range of the output vector to (0, 1).

[0042] F scale This means multiplying the adjusted vector as weights with the original feature map to achieve channel-level weight allocation, as shown in the following formula:

[0043] F scale (x c s c ) = s c ·x c

[0044] In another embodiment, the C3SE module includes a GCR module and an SE-BottleNeck module, wherein the SE-BottleNeck module is composed of a GCR module and an SE attention module in series, and the SE attention module includes a squeeze operation module, an excitation operation module and a scaling operation module.

[0045] In this embodiment, the C3SE module represents the merging of the SE attention mechanism with the C3 module to achieve weight allocation at the feature channel level, such as... Figure 5 As shown in the figure, the GCR module represents the combination of Ghost convolution and ReLU activation function, specifically as follows: Figure 6 As shown; the SE-BottleNeck module represents the combination of the SE module and the BottleNeck module, specifically as follows: Figure 7 As shown in the figure, there are two GCR modules for feature extraction. The extracted results are added to the input features to form a BottleNeck structure. The results of the BottleNeck are input to the SE module to implement the attention mechanism.

[0046] In another embodiment, the SE attention mechanism squeezing operation module transforms the input feature map from the C3 module into a vector of the same length as its number of channels through global average pooling; the activation operation module adjusts the obtained vector using convolution; and the scaling operation module multiplies the adjusted vector as channel weights with the input features at the channel level to complete the channel attention mechanism and realize the weight allocation at the channel level.

[0047] In this embodiment, the SE attention mechanism comprises two linear layers. The first linear layer further compresses the vector and then uses the ReLU activation function to achieve non-linear feature extraction. The second linear layer restores the number of channels to be the same as the input. Finally, the Sigmoid activation function is used to transform the numerical range to 0-1, obtaining a weight vector, which is then multiplied with the original feature map to obtain a feature map with redistributed weights, thus realizing the feature channel-level attention mechanism. The specific process is as follows: Figure 4 As shown in the figure: F sq The squeezing operation is implemented using global average pooling, which compresses the feature map into a vector, as shown in the following formula:

[0048]

[0049] In the formula: H and W represent the height and width of the input feature map, x c (i, j) represents the input feature located at (i, j).

[0050] F ex This represents the activation operation, which uses convolution to adjust the vector, as shown in the following formula:

[0051] F ex (z, W)=σ(g(z, W))=σ(W2δ(W1z)),

[0052] In the formula: z represents the vector obtained from the previous compression operation, W1 and W2 represent the weights of the first and second linear layers, which are obtained through learning, δ represents the ReLU activation function to achieve non-linear feature extraction, and σ represents the Sigmoid activation function to limit the range of the output vector to (0, 1).

[0053] F scale This means multiplying the adjusted vector as weights with the original feature map to achieve channel-level weight allocation, as shown in the following formula:

[0054] F scale (x c s c ) = s c ·x c

[0055] In another embodiment, the homography transformation matrix will generate a translational deviation as the detection distance changes, and the translational deviation will be manually adjusted in both the X and Y directions to correct the translational deviation.

[0056] In this embodiment, the homography transformation depends on the homography transformation matrix. To calculate this matrix, RGB and IR parameters must first be calibrated to obtain the camera's intrinsic and extrinsic parameters. Then, the homography transformation matrix is ​​calculated using a pinhole camera model. The derivation formula is as follows:

[0057]

[0058] In the formula: [X rgb Y rgb Z rgb ] represents the coordinates of a point in the RGB camera coordinate system;

[0059] [X ir Y ir Z ir The coordinates of this point in the IR camera coordinate system; [x rgb y rgb ] represents the projected coordinates of the spatial point on the RGB camera plane; [x ir y ir ] represents the projected coordinates of the point on the IR camera plane; R and t represent the rotation and translation matrices from the IR camera to the RGB camera; K rgb With K ir Z represents the intrinsic parameter matrix of the RGB and IR cameras; rgb With Z irThis represents the distance from the target point to the RGB camera and the IR camera. H is the homography transformation matrix we are looking for.

[0060] The above equation shows that the homography transformation matrix from RGB to IR camera will deviate in the translation part as the detection distance changes. Therefore, in practical applications, manual adjustment is required in both the X and Y directions to correct this translation deviation.

[0061] In another embodiment, the small target detection layer is formed by adding a GCR module after the C3 module of the 6th layer of the YOLOv5-n benchmark network to adjust its channel number, using an Up-Sample module to upsample it, and then concatenating it with the same-sized output of the GCR module of the 2nd layer. Finally, it is passed through a C3 module to obtain the small target feature layer.

[0062] In this embodiment, as Figure 2 As shown in the 21*160*160 feature map in the lower right corner, a GCR module was added after the C3 module in layer 6 of the baseline network to adjust its channel count. An Up-Sample module was used to upsample it, resulting in a 32*160*160 output. This output was then concatenated with the same-sized output from the GCR module in layer 2, and finally passed through a C3 module to obtain the small target feature layer. This feature layer, with a size of 32*160*160, only undersampled a 640*640 input image by a factor of 4. Any target larger than 4 pixels will not suffer direct feature loss due to downsampling. Compared to the baseline network Yolov5n's maximum feature map of 64*80*80, this feature layer is more advantageous for small target detection.

[0063] In another embodiment, the SPPF module includes a GCR module and a MaxPool maximum pooling module.

[0064] In this embodiment, the SPPF module combines the GCR module and the MaxPool max pooling module, achieving feature combinations of different receptive fields through cascaded max pooling modules, such as... Figure 9 As shown, this enhances the ability to extract features from targets at different scales.

[0065] In another embodiment, the training process of the lightweight RGB-IR fusion small target detection network in step S200 includes the following steps:

[0066] Step 1: Randomly initialize the weights of the fusion small object detection network, input the training set images into the fusion small object detection network, and obtain the prediction results of the input training set images;

[0067] Step 2: Calculate the loss using the prediction results and the labels in the training set, and backpropagate the calculated loss to the fused small object detection network using the SGD gradient descent method; Step 3: Repeat steps 1 and 2. When the fused small object detection network converges, the training is complete; if it does not converge, increase the number of training iterations until it converges.

[0068] In another embodiment, this method is applied to the field of surveillance and security, deployed in a device that includes infrared and visible light cameras and an embedded development board, achieving robust detection of small targets such as people and vehicles in the environment. A schematic diagram of the device is shown below. Figure 10 As shown. In this embodiment, the alignment step in S100 is first executed to align the infrared and visible light images. Then, the fused image tensor is sent to the NPU computing unit of the development board to run the RGB-IR fusion small target detection model described in step S200. In this application scenario, distant people and vehicles occupy fewer pixels in the image and are referred to as small targets. Traditional methods have low detection accuracy for small targets, and the model described in S200 effectively improves the detection accuracy of small targets. In addition, the RGB-IR fusion detection method described in this method improves robustness to the environment. The infrared camera improves the detection capability in scenes such as darkness and smoke; the visible light camera improves the detection accuracy under good lighting conditions and can detect through glass, avoiding the problem that the infrared camera cannot image under ordinary glass. The fusion detection method complements the advantages of both, improving detection robustness. Figure 11 As shown, in nighttime scenarios, the fusion detection method significantly improves detection accuracy.

[0069] Finally, it should be noted that those skilled in the art can make many variations based on the guidance of this specification and without departing from the scope of protection of the claims of this invention, and these variations are all within the scope of protection of this invention.

Claims

1. A lightweight RGB-IR fusion small target detection method, comprising the following steps: S100: preprocessing the image to be detected, wherein the preprocessing refers to aligning the IR and RGB images using homography transformation to obtain a homography transformation matrix, taking the RGB image as the first three channels, taking the IR image as the fourth channel, and composing a four-channel image tensor; S200: inputting the image tensor obtained after preprocessing into a lightweight RGB-IR fusion small target detection network to detect the target, and obtaining the class, position, width, and height of the image to be detected; The homography transformation depends on the homography transformation matrix, which will deviate in the translation part as the detection distance changes, and is manually adjusted in the X and Y directions to correct the translation deviation; wherein The lightweight RGB-IR fusion small target detection network in step S200 is improved on the basis of the YOLOv5 framework, and the network comprises an Upsample up-sampling module, a Concat module, a GCR module, a SPPF module, a MaxPool maximum pooling module, a small target detection layer, and a C3SE module improved using an SE attention mechanism; The C3SE module comprises a GCR module and an SE-BottleNeck module, wherein the SE-BottleNeck module is composed of a GCR module and an SE attention module in series, and the SE attention module comprises a squeezing operation module, an excitation operation module, and a scaling operation module; The small target detection layer is added with a GCR module after the C3 module of the 6th layer of the benchmark network YOLOv5-n to adjust the number of channels thereof, uses an Up-Sample module to perform up-sampling, splices the same size output of the 2nd layer GCR module, and finally obtains a small target feature layer through a C3 module. The SPPF module comprises a GCR module and a MaxPool maximum pooling module, and the GCR module represents the combination of a Ghost convolution and a Relu activation function.

2. The method of claim 1, wherein the pre-processing in step S100 is specifically: taking the RGB image as a reference, performing homography transformation on the IR image, aligning the IR image with the RGB image after the homography transformation, and then combining the IR image and the RGB image into a four-channel tensor, wherein, The RGB image is taken as the first three channels, the IR image after homography transformation is taken as the fourth channel, and the combined four-channel image tensor is taken as the input of the lightweight RGB-IR fusion small target detection network.

3. The method of claim 1, wherein the squeezing operation module converts the input feature map from the C3 module into a vector with the same length as the number of channels thereof through global average pooling; the excitation operation module adjusts the obtained vector using convolution; and the scaling operation module multiplies the adjusted vector with the input feature at the channel level as a channel weight to complete the channel attention mechanism and realize the weight distribution at the channel level.

4. The method of claim 1, wherein the training process of the lightweight RGB-IR fusion small target detection network in step S200 comprises the following steps: Step 1: randomly initializing the weights of the fusion small target detection network, inputting the training set picture into the fusion small target detection network, and obtaining the prediction result of the input training set picture; Step 2: Calculate the loss using the prediction results and the labels contained in the training set, and use the SGD gradient descent method to backpropagate the fused small target detection network based on the calculated loss; Step 3: Repeat steps 1 and 2. When the fused small object detection network converges, the training is complete; if it does not converge, increase the number of training iterations until it converges.

Images