A 3D target detection method fusing RGB and infrared information

Abstract

The application relates to a 3D target detection method fusing RGB and infrared information, comprising the following steps: acquiring pixel-aligned RGB and infrared image sample data; constructing a target detection network based on deep learning, training the sample data, and obtaining a target detection model; inputting simultaneously photographed and aligned RGB and infrared images in the actual world into the model to obtain 3D pose information of the target. The target detection network comprises a dual-modal fusion backbone network (extracting dual-modal features and performing feature-level fusion), a semantic feature extraction module, a depth perception feature enhancement module (implicitly learning depth perception features), a depth position encoding module (generating depth position encoding), a depth perception Transformer module (globally integrating feature maps), and a detection head (predicting object categories, 2D frames and 3D frames). Compared with the prior art, the application can effectively fuse the complementary information of the RGB and infrared dual modalities, can improve the target detection performance in a low-light scene, and can ensure the accuracy of the detection result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving visual perception, and particularly relates to a 3D target detection method fusing RGB and infrared information. BACKGROUND

[0002] With the rise of artificial intelligence represented by automatic driving, visual-based perception technology has become a research hotspot in the current academic and industrial circles, and target detection is a popular topic. The target detection task is mainly divided into 2D target detection and 3D target detection. 2D target detection is generally performed on images captured by a camera, which requires identifying the categories of each target object and generating a 2D bounding box on the image to accurately locate the object instance. 3D target detection is to extend the positioning task to the 3D space, and the output positioning result is the 3D bounding box and orientation of the object in the real world.

[0003] Since the automatic driving perception task needs to provide the pose information of the object in the real world, a 3D target detection method must be used. Most of the existing technologies use RGB cameras for 3D target detection. Although the RGB camera has the advantages of low cost, rich image semantic and texture information, etc., it has the disadvantage of being easily affected by natural environment such as light, and has the problem of detection difficulty in insufficient light conditions (such as night). In addition, the existing technology uses infrared cameras for 3D target detection. Since the infrared camera can capture the radiant heat of the object, it can reflect the clear outline of the object even in low light conditions, but the disadvantage is that the image texture features are not as rich as the RGB camera, resulting in low accuracy of the final detection result. SUMMARY

[0004] The purpose of the present application is to overcome the defects of the prior art and provide a 3D target detection method fusing RGB and infrared information, which can improve the target detection performance in low light conditions by effectively fusing the complementary information of RGB and infrared dual modal, and ensure the accuracy of the detection result.

[0005] The purpose of the present application can be achieved by the following technical scheme: a 3D target detection method fusing RGB and infrared information, comprising the following steps:

[0006] S1, acquiring pixel-aligned RGB and infrared image sample data;

[0007] S2, constructing a target detection network based on deep learning, and training the target detection network using the sample data obtained in step S1 to obtain a target detection model;

[0008] S3, inputting the simultaneously captured and aligned RGB and infrared images in the actual world into the target detection model to output the 3D pose information of the target.

[0009] Further, the target detection network comprises a dual-modal fusion backbone network, a semantic feature extraction module, a depth perception feature enhancement module, a depth position encoding module, a depth perception Transformer module and a detection head, the dual-modal fusion backbone network is used for extracting dual-modal features from pixel-aligned RGB and infrared images and fusing the dual-modal features to obtain fused features;

[0010] The semantic feature extraction module is used for extracting semantic features from the fused features;

[0011] The depth perception feature enhancement module is used for predicting depth information from the fused features and mining depth perception features;

[0012] The depth position encoding module is used for generating corresponding depth position encodings according to the depth information;

[0013] The depth perception Transformer module is based on the semantic features, the depth perception features and the depth position encodings, and establishes a global interaction relationship between the semantic features and the depth perception features to obtain integrated feature maps;

[0014] The detection head is based on the integrated feature maps and outputs corresponding detection results.

[0015] Further, the step S2 of training the target detection network comprises the following steps:

[0016] S21, for the input pixel-aligned RGB and infrared images, the dual-modal fusion backbone network extracts dual-modal features and uses a multi-layer feature-level fusion algorithm to fuse the dual-modal features to obtain fused features;

[0017] S22, for the fused features obtained in step S21, the semantic feature extraction module extracts semantic features;

[0018] S23, the depth perception feature enhancement module learns the depth information of objects by auxiliary depth supervision based on the fused features obtained in step S21, and further implicitly learns depth perception features based on the depth information;

[0019] S24, the depth position encoding module generates corresponding depth position encodings according to the depth information predicted in step S23;

[0020] S25, the depth perception Transformer module is based on the semantic features generated in step S22, the depth perception features generated in step S23 and the depth position encoding generated in step S24, uses the encoder and decoder architecture of the Transformer to establish the global interaction relationship between the semantic features and the depth perception features, and obtains an integrated feature map, thereby realizing the key guidance of the depth information to the 3D target detection.

[0021] S26, the integrated feature map is input into the anchor-based 2D-3D detection head to predict the object category, 2D box and 3D box respectively, and a loss function is designed for the training of the target detection network.

[0022] Further, the step S21 specifically comprises the following steps:

[0023] S211, for the input pixel-aligned RGB and infrared images, respective features are extracted by two convolutional neural networks respectively to obtain RGB modal features and infrared modal features at multiple scale levels;

[0024] S212, the corresponding RGB modal features and infrared modal features are spliced on the last three network layers of the two convolutional neural networks, and 1x1 convolution weighted fusion is performed to obtain three spliced features;

[0025] S213, according to the network layer high-low order of the three spliced features, the three spliced features are further fused by using an iterative deep aggregation (IDA) structure to output a single-scale fusion feature.

[0026] Further, the iterative deep aggregation structure in the step S213 specifically splices the features of the high-layer network through 3x3 convolution, deconvolution upsampling and the features of the low-layer network, extracts again through 3x3 convolution, and obtains a single-scale fusion feature through iterative tree aggregation.

[0027] Further, the step S23 specifically comprises the following steps:

[0028] S231, the depth range is discretized into D small intervals, and the fusion feature F e R C×H×W output in step S21 is used to obtain an initial depth perception feature X e R C×H×W through a first convolutional layer, and then a depth class probability D e R D×H×W of each pixel on the feature map is obtained through a second convolutional layer.

[0029] S232. For the depth category probability distribution D, use group convolution to merge adjacent depth categories. The number of categories is downsampled to D′=D / r with a sampling interval of r. After downsampling, the categories share similar depth cues, reducing the amount of computation.

[0030] S233. The initial depth-sensing feature X is further extracted through a convolutional layer to obtain X′.

[0031] S234. For each depth category in D′, all pixels of X′ are weighted according to the corresponding probability in D′ and then accumulated in the spatial dimension to obtain the depth prototype F of each depth category. d :

[0032]

[0033] Among them, X i ′ is the feature of the i-th pixel of X′. Let be the probability that the i-th pixel of D′ belongs to the d-th depth category;

[0034] S235. For each pixel position of D′, the depth prototype F of all depth categories. d The reconstructed depth-aware features F′ are obtained by weighting and summing the probabilities corresponding to D′, enabling each pixel to understand the representation of the depth category from a global perspective:

[0035]

[0036] S236. Concatenate the initial depth-sensing feature X with the reconstructed depth-sensing feature F′, and then extract the enhanced depth-sensing feature through a 1×1 convolution.

[0037] Furthermore, step S24 specifically includes the following steps:

[0038] S241. Construct a learnable embedding code E based on each depth category introduced in step S23. d =[e1,e2,...,e D ]∈R D×C ;

[0039] S242. From the depth probability distribution D predicted in step S23, find the depth category with the highest probability for each pixel, and then from E... d Find the embedding code e corresponding to the depth category in the middle. d Using the features of C channels at that pixel, the initial depth location code P∈R is obtained. H×W×C ;

[0040] S243. Local location cues are further aggregated through convolutional layers, and the final depth location code is obtained after residual connection.

[0041] Further, the step S25 specifically comprises the following steps:

[0042] S251, flattening the semantic feature generated in step S22 in spatial dimension into X c ∈R N×C , input into the Transformer encoder, and obtaining the encoded key matrix and value matrix through multi-head self-attention and feedforward neural network;

[0043] S252, inputting the depth perception feature with rich depth clues generated in step S23 after being flattened in spatial dimension into the Transformer decoder, and obtaining the depth perception query matrix through multi-head self-attention;

[0044] S253, the Transformer decoder further utilizes the self-attention mechanism to cross query the semantic feature encoded key matrix and value matrix with the depth perception query matrix, and obtain the output feature guided by the depth information.

[0045] Further, the Transformer encoder and the Transformer decoder are both based on the standard Transformer architecture, the depth position encoding generated in step S24 is imported as the position encoding in the Transformer architecture, and the operation expression of the Transformer self-attention layer is:

[0046]

[0047] Q∈R N×C

[0048] K∈R N×C

[0049] V∈R N×C

[0050] Wherein, Q is the query matrix, K is the key matrix, and V is the value matrix, Q, K, and V are collectively used as the given input of the self-attention layer.

[0051] Further, the step S26 specifically comprises the following steps:

[0052] S261, using the 2D bounding box [x 2d ,y 2d ,w 2d ,h 2d ] and the 3D bounding box [x p ,y p ,z,w 3d ,h 3d ,l 3d ,θ] to define the anchor point, wherein [x 2d,y 2d ] and [x p ,y p [w] represents the projections of the center of the 2D bounding box and the center of the 3D bounding box onto the image plane, respectively. 2d ,h 2d ] and [w 3d ,h 3d ,l 3d ] represents the pixel size of the 2D bounding box and the physical size of the 3D bounding box, respectively; z represents the depth of the center of the 3D bounding box; and θ represents the viewing angle indicating the orientation of the object.

[0053] S262, The target confidence c, classification score cls, and relative offset of the 2D and 3D bounding boxes are predicted for each anchor point by the 2D-3D detector head. x ,t y ,t w ,t h ] 2d and [t] x ,t y ,t w ,t h ,t l ,t z ,t θ ] 3d ;

[0054] S263. During the inference phase, the 2D and 3D bounding boxes are recovered using the following formula based on the relative offsets of the anchor points and the network predictions:

[0055]

[0056] in, This indicates the recovery parameter, and * indicates that the vectors are multiplied at corresponding positions.

[0057] S264. During the training phase, design a total loss function L consisting of four parts: target confidence loss, classification loss, bounding box regression loss, and auxiliary deep supervision loss, for training:

[0058] L = L conf +L cls +L reg +L dep

[0059] Among them, L conf For the target confidence loss, L cls For classification loss, L reg For bounding box regression loss, L dep To assist in deep monitoring of losses.

[0060] Compared with the prior art, the present invention has the following advantages:

[0061] I. The target detection network based on deep learning of the present application combines pixel-aligned RGB and infrared images to train a target detection model, which includes a dual-modal fusion backbone network, a semantic feature extraction module, a depth perception feature enhancement module, a depth position encoding module, a depth perception Transformer module, and a detection head. The dual-modal fusion backbone network is used to extract dual-modal features from the pixel-aligned RGB and infrared images and fuse them to obtain fusion features. The semantic feature extraction module is used to extract semantic features from the fusion features. The depth perception feature enhancement module is used to predict depth information from the fusion features and mine depth perception features. The depth position encoding module is used to generate corresponding depth position encodings based on the depth information. The depth perception Transformer module is used to establish a global interaction relationship between semantic features and depth perception features based on semantic features, depth perception features, and depth position encodings to obtain integrated feature maps. The detection head is used to output corresponding detection results based on the integrated feature maps. This can effectively fuse the complementary information of RGB and infrared dual modalities, improve the 3D target detection performance in complex scenes, especially in low-light scenes, and has important theoretical significance and practical application value for realizing all-weather automatic driving.

[0062] II. The present application uses a dual-modal fusion backbone network to extract dual-modal features from pixel-aligned RGB and infrared images and adopts a multi-layer feature-level fusion algorithm for fusion, which can help the network learn to adaptively fuse complementary information under different semantic levels in different light conditions during the day and at night, fully utilize the input information of RGB images and infrared images, and thus be robust to changes in light conditions during the day and at night, especially improving the target detection performance in low-light scenes.

[0063] III. The present application considers that the depth information of an object is crucial for the positioning of a 3D bounding box and is also naturally missing in monocular images. Therefore, in the depth perception feature enhancement module of the target detection model, auxiliary depth supervision is used to make the network learn the depth information of the object and further implicitly learn depth perception features based on the depth information, which can ensure accurate depth information and thus improve the accuracy of subsequent 3D target detection. Moreover, the feature-level implicit learning method can greatly alleviate the problem of a large amount of noise and additional computational burden caused by explicit depth map generation of traditional depth estimators.

[0064] IV. In the present application, the depth perception Transformer module of the target detection model is based on semantic features, depth perception features, and depth position encodings, and uses the encoder and decoder architecture of Transformer to establish a global interaction relationship between semantic features and depth perception features and obtain integrated feature maps, so as to realize the key guidance of depth information to 3D target detection. BRIEF DESCRIPTION OF DRAWINGS

[0065] Figure 1 A flowchart of the method of the present application;

[0066] Figure 2 A general architecture diagram of the 3D target detection method fusing RGB and infrared information;

[0067] Figure 3 A schematic diagram of a dual-modal fusion backbone network;

[0068] Figure 4 A schematic diagram of a deep perception feature enhancement module;

[0069] Figure 5 A schematic diagram of a depth position encoding module;

[0070] Figure 6 A schematic diagram of a deep perception Transformer module. DETAILED DESCRIPTION

[0071] The present application will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0072] EMBODIMENT

[0073] As shown in the accompanying drawings, Figure 1 a 3D target detection method fusing RGB and infrared information includes the following steps:

[0074] S1, acquiring pixel-aligned RGB and infrared image sample data;

[0075] S2, constructing a target detection network based on deep learning, and training the target detection network using the sample data obtained in step S1 to obtain a target detection model;

[0076] S3, inputting the simultaneously captured and aligned RGB and infrared images in actual application into the target detection model to output the 3D pose information of the target.

[0077] The present embodiment applies the above technical solution to construct a target detection network based on deep learning, as shown in the accompanying drawings, Figure 2 which includes:

[0078] I. The dual-modal fusion backbone network extracts dual-modal features from the pixel-aligned RGB and infrared images, and uses a multi-layer feature-level fusion algorithm for fusion.

[0079] II. For the output fusion features, a semantic feature extraction module (in the form of a convolution layer in the present embodiment) is used to extract semantic features.

[0080] Three, a depth perception feature enhancement module is introduced for the output fusion feature, auxiliary depth supervision is introduced to enable the network to learn the depth information of the object, and then the depth perception feature is further learned implicitly based on the depth information;

[0081] Four, a depth position encoding module generates depth position encoding according to the predicted depth information;

[0082] Five, a depth perception Transformer module uses the encoder and decoder architecture of the Transformer to establish the global interaction relationship between the semantic feature and the depth perception feature based on the generated semantic feature, the generated depth perception feature and the generated depth position encoding, so as to realize the key guidance of the depth information to the 3D target detection;

[0083] Six, an anchor-based 2D-3D detection head is used to predict the object category, 2D box and 3D box, and a loss function is designed for network training.

[0084] In the above process one, as shown in Figure 3

[0085] The input RGB and infrared dual-mode information streams are independently extracted by two deep aggregation networks DLA-102, and a hierarchical depth aggregation (HDA) structure with depths of 1, 3, 4 and 1 is used to obtain features at multiple scale levels;

[0086] In the last three network layers of the two deep aggregation networks, the corresponding dual-mode features with step sizes of 8, 16 and 32 are spliced respectively, and are fused by 1×1 convolution weighting;

[0087] Between different network layers, further fusion is performed by using an iterative depth aggregation (IDA) structure, and a single scale fusion feature with a step size of 8 is output. The iterative depth aggregation (IDA) structure splices the high-level network features with the low-level network features after 3×3 convolution and deconvolution upsampling, and then extracts the single scale fusion feature by 3×3 convolution and iterative tree aggregation.

[0088] In the above process three, as shown in Figure 4

[0089] First, the depth range is discretized into D=96 small intervals, and then the fusion feature F∈R C ×H×W is output by the first convolution layer to obtain the initial depth perception feature X∈R C×H×W , and then the second convolution layer is used to obtain the depth category probability D∈R D×H×W of each pixel on the feature map;

[0090] ​​Then, for the depth category probability distribution D, adjacent depth categories are merged using group convolution. The number of categories is downsampled to D′=D / r with a sampling interval of r=4. After downsampling, the categories can share similar depth cues and reduce the amount of computation.

[0091] The initial depth-sensing feature X is further extracted through a convolutional layer to obtain X′;

[0092] For each depth category in D′, all pixels of X′ are weighted according to the corresponding probability in D′ and then accumulated in the spatial dimension to obtain the depth prototype F for each depth category. d :

[0093]

[0094] Where X′ i Let X′ be the feature of the i-th pixel. Let be the probability that the i-th pixel of D′ belongs to the d-th depth category;

[0095] For each pixel position of D′, the depth prototype F for all depth categories d The reconstructed depth-aware features F′ are obtained by weighting and summing the probabilities corresponding to D′, enabling each pixel to understand the representation of the depth category from a global perspective:

[0096]

[0097] Finally, the initial depth-sensing features and the reconstructed depth-sensing features are concatenated and then extracted again through a 1×1 convolution to obtain enhanced depth-sensing features.

[0098] It should be noted that assisted depth supervision is only used during the network training phase. It requires prior acquisition of the ground truth depth class probability distribution D of the training set images, generated from synchronously acquired and calibrated laser point clouds. The specific operation is as follows: the laser point cloud is projected onto the image plane according to the calibrated extrinsic parameters and camera intrinsic parameters to generate a sparse ground truth depth map; then, the Linear Incremental Discretization (LID) algorithm is used to generate depth intervals for class classification, as shown in the following formula:

[0099]

[0100] Where i is the category index, D is the number of categories, [d min ,d max [] represents the depth range. Note that pixels outside the depth range are considered invalid and are not included in the optimization during training.

[0101] In process four above, such as Figure 5 As shown, it includes:

[0102] First, based on each depth category introduced in process three, a learnable embedding encoding is constructed:

[0103] E d =[e1,e2,...,e D ]∈R D×C ;

[0104] Next, from the depth probability distribution D predicted in process three, find the depth category with the highest probability for each pixel, and then from E... d Find the embedding code e corresponding to the depth category in the middle. d Using the features of the C channels at that pixel, the initial depth location code P∈R is obtained. H×W×C ;

[0105] Finally, local location cues are further aggregated through convolutional layers, and the residual connections yield the final depth location code.

[0106] In process five above, such as Figure 6 As shown, it includes:

[0107] First, the semantic features generated in process two are flattened into X in the spatial dimension. c ∈R N×C Where N = H × W, the input is fed into the Transformer encoder, and the encoded key matrix and value matrix are obtained through multi-head self-attention and feedforward neural network;

[0108] Then, the depth-aware features with rich depth cues generated in process three are flattened in the spatial dimension and input into the Transformer decoder to obtain the depth-aware query matrix through multi-head self-attention.

[0109] Subsequently, the Transformer decoder further utilizes a self-attention mechanism to perform cross-queries on the key and value matrices encoded with semantic features using a depth-aware query matrix, thereby obtaining output features guided by depth information.

[0110] The Transformer encoder and decoder are both based on the standard Transformer architecture, given the query matrix Q∈R. N×C Key matrix K∈R N×C Value matrix V∈R N×C The operating formula for the Transformer self-attention layer is:

[0111]

[0112] It should be noted that the depth position code generated in process four is imported as the position code in the Transformer architecture.

[0113] The above process six specifically includes the following:

[0114] Firstly, 2D bounding box [x 2d ,y 2d ,w 2d ,h 2d ] and 3D bounding box [x p ,y p ,z,w 3d ,h 3d ,l 3d ,theta] are used to define anchor points, where [x 2d ,y 2d ] and [x p ,y p ] are the center of 2D box and the projection of 3D box center on image plane respectively, [w 2d ,h 2d ] and [w 3d ,h 3d ,l 3d ] are the pixel size of 2D box and the physical size of 3D box respectively, z is the depth of 3D box center, and theta is the observation angle representing the object orientation;

[0115] Then, the 2D-3D detector refers to the classic single-stage detector Yolov3 to predict target confidence c, classification score cls and the relative offset of 2D and 3D bounding boxes [t x ,t y ,t w ,t h ] 2d and [t x ,t y ,t w ,t h ,t l ,t z ,t θ ] 3d for each anchor point;

[0116] In the inference stage, according to the anchor points and the relative offset predicted by the network, the 2D box and the 3D box are recovered by the following formula:

[0117]

[0118] where represents the recovery parameter, and * represents the multiplication of the corresponding positions of the vectors;

[0119] In the training stage, the target confidence loss L conf , the classification loss L cls , the bounding box regression loss L reg , and the auxiliary depth supervision loss L dep are designed to constitute the total loss L:

[0120] L=L conf +Lcls +L reg +L dep

[0121]

[0122] where L conf , L cls , L dep all adopt focal loss function FL, L reg adopts smooth L1 loss function smooth L1 , N represents the total number of samples, i represents the sample index, pos is the set of positive samples, T is the set of all regression parameters, P is the set of pixels with effective depth labels, and D is the depth category probability distribution.

[0123] This embodiment uses 48 anchor points on each pixel of the feature map, and the 2D box includes 16 pixel height according to 24x2 i / 4 , i={0, 1,..., 15} defines the scale and 3 kinds of width-height ratio {0.5, 1.0, 1.5}, and the 3D box parameters are generated according to the statistical prior of the mean and variance of the 3D box true value of the training set. During training, all 3D box true values are projected onto the image plane to calculate the intersection over union (IoU) with all 2D anchor points, and the anchor points with IoU greater than 0.5 are selected as positive samples.

[0124] In summary, the technical scheme proposes a 3D target detection method fusing RGB and infrared information, which can use pixel-aligned RGB images and infrared images for 3D target detection, and the designed target detection network based on deep learning includes a dual-modal fusion backbone network, a semantic feature extraction module, a depth perception feature enhancement module, a depth position encoding module, a depth perception Transformer module and a 2D-3D detection head. First, the pixel-aligned RGB image and infrared image are input into the dual-modal fusion backbone network, the dual-modal features are extracted and fused at the feature level. The depth perception feature enhancement module learns the uncertainty distribution of depth through auxiliary depth supervision, and further learns the depth perception feature implicitly, and extracts semantic features through a convolution layer in parallel. The depth perception Transformer module globally integrates the two features by using the Transformer encoder and decoder architecture. The depth position encoding module generates depth position encoding and imports the Transformer architecture. Finally, the anchor-based detection head is used to predict the object category, 2D box and 3D box. Compared with the existing target detection method, the present application can obtain complementary information under different light conditions including daytime and night through RGB and infrared dual-modal sensors, and the multi-layer fusion strategy can help the network to learn to adaptively fuse the complementary information at different semantic levels, so that the input information of the RGB image and the infrared image can be fully utilized in theory, so that the light change of the daytime and night scene has robustness, and especially the detection performance of the low light scene can be improved;

[0125] The 3D target detection completed by the present application can reflect the coordinates of the object in the real world compared with the traditional 2D target detection in the field of RGB and infrared fusion, and is suitable for tasks such as automatic driving and robots that require such information;

[0126] The present application makes the network learn the depth information which is essential for the positioning of the 3D box of the object and is naturally missing in the monocular image, and uses the learned depth information to guide the 3D target detection, and uses feature-level implicit learning to alleviate the problem of a large amount of noise and additional computational burden caused by the explicit generation of depth map by the traditional depth estimator;

[0127] The present application combines the advanced convolutional neural network and the Transformer architecture, has higher detection accuracy and robustness, especially can improve the detection performance of the low light scene, and provides a new solution for all-weather automatic driving perception tasks.

Claims

1. A 3D target detection method that integrates RGB and infrared information, characterized in that, Includes the following steps: S1. Obtain pixel-aligned RGB and infrared image sample data; S2. Construct a deep learning-based object detection network and train the object detection network using the sample data obtained in step S1 to obtain an object detection model. S3. Input the RGB and infrared images that are captured and aligned simultaneously in reality into the target detection model, and output the 3D pose information of the target. The target detection network includes a dual-modal fusion backbone network, a semantic feature extraction module, a depth-aware feature enhancement module, a depth-position encoding module, a depth-aware Transformer module, and a detection head. The dual-modal fusion backbone network is used to extract dual-modal features from pixel-aligned RGB and infrared images and fuse them to obtain fused features. The semantic feature extraction module is used to extract semantic features from the fused features; The depth-sensing feature enhancement module is used to predict depth information from the fused features and to extract depth-sensing features. The depth location encoding module is used to generate the corresponding depth location code based on the depth information; The depth-aware Transformer module, based on semantic features, depth-aware features, and depth position encoding, obtains an integrated feature map by establishing a global interaction relationship between semantic features and depth-aware features. The detection head outputs corresponding detection results based on an integrated feature map; Step S2, training the object detection network, specifically includes the following steps: S21. For the input pixel-aligned RGB and infrared images, dual-modal features are extracted by a dual-modal fusion backbone network and fused using a multi-layer feature-level fusion algorithm to obtain fused features; S22. Based on the fusion features obtained in step S21, semantic features are extracted using the semantic feature extraction module; S23. Using the fused features obtained in step S21, the depth perception feature enhancement module is used to learn the depth information of the object through assisted depth supervision, and then the depth perception features are further implicitly learned based on the depth information. S24. The depth location coding module generates the corresponding depth location code based on the depth information predicted in step S23. S25. Based on the semantic features generated in step S22, the depth-sensing features generated in step S23, and the depth position encoding generated in step S24, the depth-sensing Transformer module uses the encoder and decoder architecture of the Transformer to establish a global interaction relationship between the semantic features and the depth-sensing features, and obtains an integrated feature map, thereby realizing the key guidance of depth information for 3D target detection. S26. Input the integrated feature map into the anchor-based 2D-3D detection head to predict the object category, 2D bounding box and 3D bounding box respectively, and design a loss function for training the object detection network; Step S21 specifically includes the following steps: S211. For the input pixel-aligned RGB and infrared images, two convolutional neural networks independently extract the corresponding features to obtain RGB modal features and infrared modal features at multiple scale levels. S212. In the last three layers of the two convolutional neural networks, the corresponding RGB modal features and infrared modal features are concatenated and then weighted by 1×1 convolution to obtain three concatenated features. S213. Based on the network layer height order of the three spliced ​​features, the three spliced ​​features are further fused using an iterative deep aggregation structure to output a single-scale fused feature. Step S23 specifically includes the following steps: S231, Discretize the depth range as For each small interval, the fused features output in step S21 are... The initial depth-sensing features are obtained through the first convolutional layer. Then, the depth class probability of each pixel on the feature map is obtained through a second convolutional layer. ; S232, Probability distribution of depth categories Group convolution is used to merge adjacent depth categories, with the number of categories increasing by... Sampling interval Downsampled categories share similar depth cues, reducing computational cost; S233, Initial Depth-Aware Features Extracted through convolutional layers ; S234, to Each depth category in All pixels according to The corresponding probabilities are weighted and then summed along the spatial dimension to obtain the depth prototype for each depth category. : in, for No. Features of each pixel for No. The pixel belongs to the first The probability of each depth category; S235, to Depth prototype for each pixel location across all depth categories according to The corresponding probabilities are weighted and summed to obtain the reconstructed deep perception features. This allows each pixel to understand the representation of the depth category from a global perspective: S236. Combine the initial depth-sensing feature X with the reconstructed depth-sensing feature. The features are concatenated and then extracted again through 1×1 convolution to obtain enhanced depth perception features.

2. The 3D target detection method fusing RGB and infrared information according to claim 1, characterized in that, In step S213, the iterative deep aggregation structure specifically involves upsampling the features of the high-level network through 3×3 convolution and deconvolution, concatenating them with the features of the low-level network, extracting them again through 3×3 convolution, and obtaining a single-scale fused feature through iterative tree aggregation.

3. The 3D target detection method fusing RGB and infrared information according to claim 1, characterized in that, Step S24 specifically includes the following steps: S241. Construct a learnable embedding code based on each depth category introduced in step S23. ; S242, Depth probability distribution predicted from step S23 In the middle, find the depth category with the highest probability for each pixel, and then from... Find the embedding code corresponding to the depth category in the middle. , to be used as the pixel The features of each channel are used to obtain the initial depth position encoding. ; S243. Local location cues are further aggregated through convolutional layers, and the final depth location code is obtained after residual connection.

4. The 3D target detection method fusing RGB and infrared information according to claim 1, characterized in that, Step S25 specifically includes the following steps: S251. Flatten the semantic features generated in step S22 in the spatial dimension as follows: ,in, The input is fed into the Transformer encoder, and the encoded key matrix and value matrix are obtained through multi-head self-attention and feedforward neural network; S252. Flatten the depth-aware features with rich depth cues generated in step S23 in the spatial dimension and input them into the Transformer decoder to obtain the depth-aware query matrix through multi-head self-attention. The S253 and Transformer decoders further utilize a self-attention mechanism, using a depth-aware query matrix to perform cross-queries on the key and value matrices encoded with semantic features, thereby obtaining output features guided by depth information.

5. A 3D target detection method fusing RGB and infrared information according to claim 4, characterized in that, Both the Transformer encoder and the Transformer decoder are based on the standard Transformer architecture. The depth position code generated in step S24 is imported as the position code in the Transformer architecture. The operation expression of the Transformer self-attention layer is: in, For querying the matrix, The key matrix, The matrix Q, K, and V together serve as the given inputs to the self-attention layer.

6. A 3D target detection method fusing RGB and infrared information according to any one of claims 1 to 5, characterized in that, Step S26 specifically includes the following steps: S261, Using 2D bounding boxes and 3D bounding boxes Predefined anchor points, where, and These are the projections of the center of the 2D bounding box and the center of the 3D bounding box onto the image plane, respectively. and These are the pixel dimensions of the 2D bounding box and the physical dimensions of the 3D bounding box, respectively. The depth of the center of the 3D frame. The viewing angle used to indicate the orientation of an object; S262, The 2D-3D detection head predicts the target confidence level for each anchor point. Category Score And the relative offsets of 2D and 3D bounding boxes. and ; S263. During the inference phase, the 2D and 3D bounding boxes are recovered using the following formula based on the relative offsets of the anchor points and the network predictions: in, Indicates the recovery parameters. This indicates that the vectors are multiplied at corresponding positions; S264. During the training phase, design a total loss function consisting of four parts: target confidence loss, classification loss, bounding box regression loss, and auxiliary deep supervision loss. Conduct training: in, For target confidence loss, For classifying losses, For bounding box regression loss, To assist in deep monitoring of losses.

Images