Target detection model, training method thereof and target detection method

Abstract

The application discloses a target detection model and a training method and a target detection method thereof, wherein the training method comprises the following steps: acquiring bottom layer features of a visible light image and an infrared image containing a labeled box of a target to be detected by using a convolutional neural network model; inputting the bottom layer features into a Transform model for training to obtain middle layer features of the visible light image and the infrared image; fusing the bottom layer features and the middle layer features to respectively obtain full modal features of the visible light image and full modal features of the infrared image; and jointly feeding the full modal features of the visible light image and the full modal features of the infrared image into a feedforward neural network module of the convolutional neural network model to obtain prediction category and prediction box information of the target to be detected. The application proposes a local region collaborative enhancement module, realizes fusion of coding features and cross-modal convolution features, and can greatly improve the detection precision of small and large targets in a complex scene.

Description

Technical Field

[0001] This invention discloses a target detection model and its training method, as well as a target detection method, specifically a target detection model and its training method that fuses visible light and infrared light, belonging to the field of target detection technology. Background Technology

[0002] With the development of computer technology and the widespread application of computer vision principles, real-time target tracking using computer image processing technology is becoming increasingly popular. Dynamic real-time target tracking and positioning has broad application value in intelligent transportation systems, intelligent monitoring systems, military target detection, and surgical instrument positioning in medical navigation surgeries. However, the primary prerequisite for dynamic real-time target tracking is target detection.

[0003] Current target detection methods are mostly based on visible light images. However, in low-light conditions, there are problems such as difficulty in clearly distinguishing targets from noise and targets being easily lost in a short time, resulting in low target detection accuracy.

[0004] To address the low accuracy issue associated with target detection using visible light images, current technologies typically fuse visible light and infrared images for target detection. Compared to visible light images, infrared images primarily present the target's temperature information and are more resistant to occlusion. They can effectively highlight targets in low-light environments or concealed situations, such as pedestrians in nighttime surveillance scenarios and ships against complex sea and sky backgrounds. However, due to the low contrast and significant loss of detail in infrared images, the fusion of infrared and visible light images for target detection suffers from low accuracy. Summary of the Invention

[0005] The purpose of this application is to provide a target detection model and its training method, as well as a target detection method, to solve the technical problem of low detection accuracy in the prior art when performing target detection by fusing infrared images and visible light images.

[0006] A first aspect of the present invention provides a method for training an object detection model, comprising:

[0007] Using a convolutional neural network model, the low-level features of visible light and infrared images containing bounding boxes of the target to be detected are obtained;

[0008] The low-level features are input into the Transformer model for training to obtain the mid-level features of the visible light image and the infrared image;

[0009] By fusing the low-level features and the mid-level features, the full-modal features of the visible light image and the full-modal features of the infrared image are obtained respectively;

[0010] The full-modal features of the visible light image and the full-modal features of the infrared image are jointly fed into the feedforward neural network module of the convolutional neural network model to obtain the predicted category and prediction box information of the target to be detected.

[0011] Preferably, the low-level features and the mid-level features are fused to obtain the full-modal features of the visible light image and the full-modal features of the infrared image, respectively, specifically including:

[0012] The bottom-level features of the visible light image are stitched together with the mid-level features of the infrared image to obtain the full-modal features of the visible light image;

[0013] The low-level features of the infrared image are stitched together with the mid-level features of the visible light image to obtain the full-modal features of the infrared image.

[0014] Preferably, the low-level features and the mid-level features are fused to obtain the full-modal features of the visible light image and the full-modal features of the infrared image, respectively, specifically including:

[0015] The mid-layer features of the visible light image, the mid-layer features of the infrared image, and the bottom-layer features of the visible light image are fused together to obtain the full-modal features of the visible light image;

[0016] The mid-layer features of the visible light image, the mid-layer features of the infrared image, and the bottom-layer features of the infrared image are fused to obtain the full-modal features of the infrared image.

[0017] Preferably, the prediction box information includes standardized center coordinates, height, and width.

[0018] Preferably, the loss function in the convolutional neural network model is determined by matching the predicted bounding box and the real detection box using the Hungarian algorithm;

[0019] The actual detection frame is determined based on the visible light image and infrared image containing the target bounding box to be detected.

[0020] Preferably, the actual detection bounding box is determined based on the visible light image and infrared image containing the bounding box of the target to be detected, specifically as follows:

[0021] Construct the union of the bounding boxes of the target to be detected in the visible light image and the bounding boxes of the target to be detected in the infrared image;

[0022] In the union set, nonmaximum suppression is used to merge the target bounding boxes of the same target in the visible light image and the infrared image to obtain the true detection box.

[0023] Preferably, the convolutional neural network model includes three linear layers, and the activation function used in the convolutional neural network model is the ReLU activation function.

[0024] Preferably, a convolutional neural network model is used to obtain the low-level features of the visible light image and infrared image containing the bounding boxes of the target to be detected, specifically including:

[0025] The visible light image and infrared image containing the bounding box of the target to be detected are respectively input into the convolution module of the convolutional neural network model for feature extraction, so as to obtain the low-level features of the visible light image containing the bounding box of the target to be detected and the low-level features of the infrared image containing the bounding box of the target to be detected.

[0026] Preferably, before using a convolutional neural network model to obtain the low-level features of the visible light image and infrared image containing the bounding boxes of the target to be detected, the method further includes:

[0027] The visible light image and infrared image containing the bounding box of the target to be detected are registered.

[0028] A second aspect of the present invention provides an object detection model, which is trained using the object detection model training method described above.

[0029] A third aspect of the present invention provides a target detection method, comprising:

[0030] Acquire visible light and infrared images containing the target to be detected;

[0031] The visible light image and infrared image to be detected are input into the target detection model to obtain the predicted category and prediction box information of the target to be detected.

[0032] The target detection model and training method of the present invention, as well as the target detection method thereon, have the following advantages compared to the prior art:

[0033] This invention proposes a Local Region Collaborative Enhancement Module (LocalEn), which integrates encoded features with cross-modal convolutional features to further enhance the expressive power of detection features and significantly improve the high detection accuracy of targets of varying sizes in complex scenes. Attached Figure Description

[0034] Figure 1 This is a flowchart illustrating the target detection model training method in an embodiment of the present invention;

[0035] Figure 2This is a schematic diagram of the local region collaborative enhancement module in an embodiment of the present invention;

[0036] Figure 3 This is a schematic diagram of the structure corresponding to step 4 in this embodiment of the invention;

[0037] Figure 4 This is an overall structural diagram of the target detection model training method used in this embodiment of the invention. Detailed Implementation

[0038] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0039] like Figure 1 As shown, a first aspect of the present invention provides a method for training an object detection model, comprising:

[0040] Step 1: Use a convolutional neural network model to obtain the low-level features of visible light images and infrared images containing bounding boxes of the target to be detected in the training set.

[0041] Before step 1, the visible light images and infrared images in the training set need to be registered.

[0042] This invention uses vehicle-mounted cameras and target detection in all-weather scenarios as examples to illustrate the registration process.

[0043] In vehicle-mounted scenarios, cameras are typically fixed, and visible light and infrared cameras may have different viewing angles in the same scene, necessitating registration. This invention can employ methods such as cross-correlation for registration. Usually, only one registration is required; if the angles of the visible light and infrared cameras change, the registration relationship needs to be updated. Subsequent visible light and infrared images are both registered images.

[0044] After registration, step 1 specifically involves:

[0045] The registered visible light image and infrared image containing the bounding boxes of the target to be detected are input into the convolutional module of the convolutional neural network model for feature extraction, thereby obtaining the low-level features of the visible light image containing the bounding boxes of the target to be detected. and the low-level features of infrared images containing bounding boxes of the targets to be detected Where B is the number of images, C is the number of channels, and H and W are the width and height of the feature map.

[0046] Furthermore, to improve the accuracy of object detection, this invention proposes a Local Region Collaborative Enhancement Module (LocalEn), which specifically involves feeding back the mid-level features obtained in subsequent steps 2 and 3 to the front-end convolutional module, fusing the low-level and mid-level features to more effectively represent the low-level features, thereby improving the accuracy of object detection. The structure of the Local Region Collaborative Enhancement Module is as follows: Figure 2 As shown.

[0047] Step 2: Input the low-level features into the Transformer model for training to obtain the mid-level features of the visible light image and the infrared image, specifically:

[0048] After converting the feature tensor corresponding to the low-level features obtained in step 1 into a (H×W)×C feature tensor, it is input into the encoder and decoder of the Transformer model for training.

[0049] In the encoder output of the Transformer model, the output features layer by layer undergo reshaping, 1x1 convolutional layers, upsampling layers, and other operations to obtain mid-level features at the same scale as the corresponding low-level features. The mid-level features of the visible light image are as follows: The mid-layer features of infrared images are .

[0050] Step 3: Fuse the low-level features and mid-level features to obtain the full-modal features of the visible light image and the full-modal features of the infrared image, respectively.

[0051] In this embodiment of the invention, after the low-level features are obtained by the convolution module, the low-level features are combined with the mid-level features output by the Transformer encoder to enhance the expressive power of the low-level features and improve the detection accuracy of small targets.

[0052] Since the Transformer encoder is a multi-layered stacked module, the output of the last layer or the combined output of multiple layers can be selected. This output is then reshaped using 1x1 convolutional layers and upsampling layers to achieve the same scale as the corresponding convolutional feature layers. Finally, it is fused with the low-level features output from the convolutional feature layers through concatenation or point-by-point addition.

[0053] In this embodiment of the invention, the output of the visible light image after feature extraction by Convolutional Neural Networks (CNN) and Transformer is denoted as... The output of the infrared image after feature extraction by CNN and Transformer is denoted as These two features, after upsampling, are the same size as the feature map to be fused, with the number of channels being... .

[0054] When using a splicing method for fusion, the specific steps are as follows:

[0055] By stitching together the low-level features of a visible light image with the mid-level features of an infrared image, the full-modal features of the visible light image are obtained.

[0056] By stitching together the low-level features of the infrared image with the mid-level features of the visible light image, the full-modal features of the infrared image are obtained.

[0057] When this invention uses a splicing method for fusion, it directly combines... and By splicing together the underlying features of different modalities, the feature map is transformed from C channels to... One channel.

[0058] When merging is done by adding points one by one, the specific steps are as follows:

[0059] By fusing the mid-level features of the visible light image, the mid-level features of the infrared image, and the bottom-level features of the visible light image, the full-modal features of the visible light image are obtained, and the corresponding formula is shown in equation (1):

[0060] (1)

[0061] In equation (1), These are the underlying features of visible light images. These are mid-level features of a visible light image. These are mid-layer features of the infrared image. and These are all feature layer stacking coefficients, which can be determined based on experimental results. Specifically, they can be... Set it to 0.

[0062] By fusing the mid-level features of the visible light image, the mid-level features of the infrared image, and the bottom-level features of the infrared image, the full-modal features of the infrared image are obtained, and the corresponding formula is shown in equation (2):

[0063] (2)

[0064] In equation (1), These are the underlying features of infrared images. These are mid-level features of a visible light image. These are mid-layer features of the infrared image. and These are all feature layer stacking coefficients, which can be determined based on experimental results. Specifically, they can be... Set it to 0.

[0065] When this invention uses a point-by-point addition method for fusion, the feature map corresponding to the bottom layer features has the same number of channels as the feature map corresponding to the middle layer features, that is... .

[0066] In the training of the local region collaborative enhancement network of this invention, the gradient backpropagation direction needs to be restricted. In order not to affect the backbone convolutional neural network, the gradients of the 1x1 convolutional layer used to fuse the mid-level and low-level features of the Transformer encoder output are not backpropagated. That is, the 1x1 convolutional layer is updated using the network loss gradient, but the backpropagation of the gradient into the backbone network is cut off.

[0067] Step 4: Combine the full-modal features of the visible light image and the full-modal features of the infrared image and feed them into the feedforward neural network module of the convolutional neural network model to obtain the predicted category and bounding box information of the target to be detected. Its structure is as follows: Figure 3 As shown, it specifically includes:

[0068] The full-modal features of the visible light image and the full-modal features of the infrared image are jointly fed into the feedforward neural network module of the convolutional neural network model for training. The feedforward neural network module outputs the predicted category and prediction box information of the target to be detected. The prediction box information includes the normalized center coordinates, height and width of the prediction box.

[0069] In this embodiment of the invention, the visible light image is converted into a fixed number of tokens (fixed to N) through convolution and Transformer encoding / decoding, and the infrared image is converted into a fixed number of tokens (fixed to N) through convolution and Transformer encoding / decoding. Both sets of tokens are used as input to the feedforward neural network (FFN) module, totaling 2N tokens. The final FFN is computed by a three-layer linear layer with ReLU activation and hidden layers, and can be implemented using 1×1 convolutions. The FFN prediction box is standardized in its center coordinates, height, and width, and then activated using the softmax function to obtain the predicted class label.

[0070] To calculate the loss during network training, in this embodiment of the invention, the loss function in the feedforward neural network module is determined by matching the predicted bounding box and the real detection box using the Hungarian algorithm.

[0071] Among them, the predicted bounding boxes are the information of 2N detection boxes and their categories output by the feedforward neural network module;

[0072] The true detection bounding boxes are the union of visible light and infrared images containing the bounding boxes of the target to be detected in the training dataset.

[0073] In this embodiment of the invention, the bounding boxes of the target objects on the ground truth image are expanded into 2N detection boxes. A special class label is used to indicate that no object was detected, or that it is considered to be the background category. Thus, both the prediction and the ground truth are two sets of 2N elements each. Then, the Hungarian algorithm is used for bipartite graph matching, that is, to match the elements of the prediction set and the ground truth set one-to-one, so as to minimize the matching loss.

[0074] This invention utilizes the Hungarian algorithm, through bipartite graph matching, to force each ground truth (gt) to have a unique prediction result. This ensures that the encoder's initial weights are no longer uniformly distributed; that is, they no longer calculate similarity with all keys, but rather with more meaningful keys. This replaces the manual setting of anchor points and the non-maximum suppression process, transforming object detection into an ensemble prediction problem. This greatly simplifies the overall detection process, significantly reduces detection time, and improves both detection speed and accuracy.

[0075] Furthermore, in this embodiment of the invention, the process of determining the actual detection box is as follows:

[0076] Construct the union of the bounding boxes for the target object in the visible light image and the bounding boxes for the target object in the infrared image. Here, since the two images have already been registered, the bounding boxes for the target object in the two images can be directly merged.

[0077] In the merge set, nonmaximum suppression is used to merge the target bounding boxes of the same target in the visible light image and the infrared image to obtain the true detection box.

[0078] The target detection model training method of this invention extracts features from registered visible light and infrared images using a convolutional CNN module. The Transformer's encoding module feeds back mid-level feature information to the front-end convolutional part, achieving more effective representation of low-level features. In the back-end, feature mapping is performed on the two modal features to reduce feature differences. Then, the two sets of features are processed through a learnable weighted fusion module to obtain a full-modal feature representation. Finally, the detection results are matched and output using a Hungarian bipartite graph allocation algorithm. The overall structure of the training method of this invention is as follows: Figure 4 As shown.

[0079] This invention proposes a Local Region Collaborative Enhancement Module (LocalEn), which integrates encoded features with cross-modal convolutional features to further enhance the expressive power of detection features and significantly improve the high detection accuracy of targets of varying sizes in complex scenes.

[0080] A second aspect of the present invention provides an object detection model, which is trained using the object detection model training method described above.

[0081] A third aspect of the present invention provides a target detection method, comprising:

[0082] Acquire visible light and infrared images containing the target to be detected;

[0083] The visible light image and infrared image are input into the target detection model to obtain the predicted category and prediction box information of the target to be detected.

[0084] The target detection method of the present invention fuses visible light images and infrared images for detection, which can achieve information complementarity and thus improve the target detection accuracy. It also overcomes the problems of relying on manual design, complex operation and slow calculation speed that exist when using convolutional neural networks alone. The target detection method of the present invention has a fast detection speed and high detection accuracy and precision.

[0085] The above description is merely a few embodiments of this application and is not intended to limit this application in any way. Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any changes or modifications made by those skilled in the art without departing from the scope of the technical solution of this application using the disclosed technical content are equivalent to equivalent implementation cases and fall within the scope of the technical solution.

Claims

1. A method for training an object detection model, characterized in that, include: Using a convolutional neural network model, the low-level features of visible light and infrared images containing bounding boxes of the target to be detected are obtained; The low-level features are input into the Transformer model for training to obtain the mid-level features of the visible light image and the infrared image; By fusing the low-level features and the mid-level features, the full-modal features of the visible light image and the full-modal features of the infrared image are obtained respectively; The full-modal features of the visible light image and the full-modal features of the infrared image are jointly fed into the feedforward neural network module of the convolutional neural network model to obtain the predicted category and prediction box information of the target to be detected. By fusing the low-level features and the mid-level features, the full-modal features of the visible light image and the full-modal features of the infrared image are obtained, specifically including: The bottom-level features of the visible light image are stitched together with the mid-level features of the infrared image to obtain the full-modal features of the visible light image; the bottom-level features of the infrared image are stitched together with the mid-level features of the visible light image to obtain the full-modal features of the infrared image. Alternatively, the mid-layer features of the visible light image, the mid-layer features of the infrared image, and the bottom-layer features of the visible light image can be fused to obtain the full-modal features of the visible light image; and the mid-layer features of the visible light image, the mid-layer features of the infrared image, and the bottom-layer features of the infrared image can be fused to obtain the full-modal features of the infrared image.

2. The target detection model training method according to claim 1, characterized in that, The prediction box information includes standardized center coordinates, height, and width.

3. The target detection model training method according to claim 1, characterized in that, The loss function in the convolutional neural network model is determined by matching the predicted bounding box and the ground truth detection box using the Hungarian algorithm. The actual detection frame is determined based on the visible light image and infrared image containing the target bounding box to be detected.

4. The target detection model training method according to claim 3, characterized in that, The actual detection bounding box is determined based on the visible light image and infrared image containing the bounding box of the target to be detected, specifically as follows: Construct the union of the bounding boxes of the target to be detected in the visible light image and the bounding boxes of the target to be detected in the infrared image; In the union set, nonmaximum suppression is used to merge the target bounding boxes of the same target in the visible light image and the infrared image to obtain the true detection box.

5. The target detection model training method according to claim 1, characterized in that, The convolutional neural network model includes three linear layers, and the activation function used in the convolutional neural network model is the ReLU activation function.

6. The target detection model training method according to claim 1, characterized in that, Using a convolutional neural network model, the low-level features of visible light and infrared images containing bounding boxes of the target to be detected are obtained, specifically including: The visible light image and infrared image containing the bounding box of the target to be detected are respectively input into the convolution module of the convolutional neural network model for feature extraction, so as to obtain the low-level features of the visible light image containing the bounding box of the target to be detected and the low-level features of the infrared image containing the bounding box of the target to be detected.

7. A target detection model, characterized in that, The model is trained using the target detection model training method described in any one of claims 1-6.

8. A target detection method, characterized in that, include: Acquire visible light and infrared images containing the target to be detected; The visible light image and infrared image are input into the target detection model of claim 7 to obtain the predicted category and prediction box information of the target to be detected.

Images