Multi-modal target detection method based on depth information graph

Abstract

The invention discloses a multi-modal target detection method based on a depth information graph. The method comprises the following steps: 1) acquiring an RGB image and a depth image, and preprocessing the RGB image and the depth image; 2) respectively extracting RGB features and depth features through a double-flow feature extraction network; 3) the RGB features and the depth features are processed through a cross-modal feature interaction module, feature enhancement guided by depth information is realized, and enhanced RGB features and enhanced depth features are obtained; 4) dynamically fusing the enhanced RGB features and the enhanced depth features through an adaptive feature fusion module to obtain fused features; and 5) processing the fusion features by using a target detection head, and outputting a detection result. According to the invention, through an attention mechanism guided by depth information and bidirectional feature interaction, the target detection precision in a complex scene can be significantly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, in particular to a multi-modal target detection method fusing RGB images and depth maps, which is especially suitable for real-time target recognition tasks in complex scenarios such as autonomous driving, robot navigation and intelligent monitoring. BACKGROUND

[0002] As a core basic technology of computer vision, target detection has been deeply applied and rapidly developed in recent years in key fields such as intelligent security systems, autonomous driving platforms, and industrial inspection pipelines. With the breakthrough progress of deep learning technology, target detection methods based on RGB images have achieved remarkable results, especially in ideal lighting conditions, which can achieve high recognition accuracy. These methods mainly extract rich texture features and color information through convolutional neural networks, locate target positions using region proposal networks, and then determine target categories through classification networks, forming a relatively mature technical route.

[0003] However, in actual engineering applications, especially in complex environmental scenarios, traditional RGB image target detection faces severe challenges: under low illumination or extreme lighting conditions, key texture information is severely lost, resulting in a significant decline in feature extraction quality; under adverse weather conditions, such as environmental interference caused by rain and fog weather, target features and background noise are highly confused. The introduction of depth information maps provides a new technical path to solve the above problems. Depth images obtained through active infrared sensors or TOF technology can directly provide three-dimensional geometric information of the scene, establishing a space structure representation independent of lighting conditions. This type of data accurately describes the positional relationship of targets in three-dimensional space, providing physical geometric features of object surfaces, effectively compensating for the perceptual limitations of RGB images. Especially in weak texture target recognition, depth information shows unique advantages.

[0004] Current multi-modal fusion methods have three key technical bottlenecks in actual deployment: first, the feature fusion mechanism is simplified, most researches use channel concatenation or weighted average as the primary fusion strategy, failing to fully exploit the complementary characteristics between different modalities; second, there is a cross-modal feature gap, the texture features of RGB images and the geometric features of depth maps have significant distribution differences, direct fusion leads to information interference; finally, the dynamic environment adaptability is insufficient, when the depth map noise increases sharply in rain and fog weather, existing methods lack effective noise suppression mechanisms, leading to a sharp decline in fusion performance. These technical defects seriously restrict the practical application value of target detection systems in safety-sensitive scenarios such as autonomous driving and industrial detection. SUMMARY

[0005] To overcome the shortcomings of existing technologies, this invention provides a multimodal target detection method based on depth information maps, aiming to significantly improve the detection robustness in complex scenes through feature enhancement guided by depth information and adaptive fusion mechanism.

[0006] To achieve the above-mentioned objectives, the present invention adopts the following technical solution: The multimodal target detection method based on depth information maps of this invention is characterized by the following steps: Step 1: Obtain an RGB image and a depth image to be detected, and perform normalization and missing value imputation on each image to obtain the preprocessed RGB image. and depth images ; make and The true annotation information of the targets includes: the regression target set of the true detection boxes. and category tag set ,in, Let d represent the true regression target of the j-th true detection box at the i-th scale, including: center coordinates and scaling factor; This represents the category label of the j-th ground truth bounding box at the i-th scale. Indicates the total number of scales; This represents the total number of true bounding boxes contained in each scale; Where x is the horizontal position, y is the vertical position, w is the width scaling factor, and h is the height scaling factor. Step 2: Construct a multimodal target detection network, including: a two-stream feature extraction module, a cross-modal feature interaction module, an adaptive feature fusion module, and a target detection head; Step 2.1: The dual-stream feature extraction module extracts features through parallel RGB backbone networks and deep backbone networks respectively. Multiscale RGB features and Multiscale depth features ;in, This represents the RGB features at the i-th scale extracted by the RGB backbone network. This represents the depth feature at the i-th scale extracted by the deep backbone network; Step 2.2: The cross-modal feature interaction module utilizes the feature modulation unit and the bidirectional feature correction unit to sequentially... and Perform interactive processing and output the RGB enhanced features at the i-th scale accordingly. and depth features at the i-th scale ; Step 2.3: The adaptive feature fusion module... and fused to obtain the fusion feature of the i-th scale ; Step 2.4: the target detection head processes the multi-scale fusion feature , and outputs the network prediction result, including the target prediction category and the target prediction box coordinates; Step 3: based on the real label information and the network prediction result, a multi-task loss function is constructed ; Step 4: the gradient descent method is used to train the target detection network, and the task loss function is calculated to update the network parameters until the maximum training times or the task loss function converges, so as to obtain the optimal target detection model for target recognition of the input RGB image and depth image.

[0007] The multi-modal target detection method has the characteristics that the RGB backbone network in the dual-stream feature extraction module in step 2.1 is composed of k convolutional blocks in cascade, wherein each convolutional block is composed of a convolutional layer, a batch normalization layer and a ReLU activation layer in cascade; When i = 1, , the RGB feature of the i-th scale is obtained by inputting the i-th convolutional block ; When i = 2, 3, …, k, the RGB feature of the i-1-th scale is input into the i-th convolutional block , and the RGB feature of the i-th scale is obtained , so that the RGB feature of the k-th scale is output by the k-th convolutional block ; The depth backbone network in the dual-stream feature extraction module in step 2.1 comprises k ConvNeXt blocks, and each ConvNeXt block is composed of a depth separable convolutional layer, a layer normalization layer, a channel expansion convolutional layer, a GELU activation layer and a channel compression convolutional layer in cascade; When i = 1, , the depth feature of the i-th scale is obtained by inputting the i-th ConvNeXt block ; When i = 2, 3, …, k, the depth feature of the i-1-th scale is input into the i-th ConvNeXt block , and the depth feature of the i-th scale is obtained , so that the depth feature of the k-th scale is output by the k-th ConvNeXt block .

[0008] Further, the feature modulation unit in the cross-modal feature interaction module of step 2.2 comprises, in sequence, a channel compression layer, a feature alignment layer, a geometric modulation layer and an activation layer; the bidirectional feature modification unit comprises, in sequence, a first feature splicing layer, a residual generation layer and a feature enhancement layer; Step 2.2.1: the feature modulation unit processes the i-th scale of the compressed feature and the i-th scale of the compressed feature to obtain the i-th scale of the activated feature ; The channel compression layer compresses the channel number of the i-th scale of the compressed feature to obtain the i-th scale of the depth compressed feature ; the i-th scale of the depth compressed feature and the i-th scale of the compressed feature are input into the feature alignment layer to perform element-wise addition to obtain the i-th scale of the aligned feature ; the geometric modulation layer performs convolution operation on the i-th scale of the aligned feature to generate the i-th scale of the spatial offset field , so as to obtain the i-th scale of the geometric modulation feature through formula (1): (1) In formula (1), DeformConv() represents a deformable convolution operator, and offset represents an offset field parameter; The activation layer processes the i-th scale of the geometric modulation feature using a LeakyReLU activation function to output the i-th scale of the activated feature ; Step 2.2.2: the bidirectional feature modification unit processes the i-th scale of the compressed feature and the i-th scale of the compressed feature , and the i-th scale of the depth compressed feature and the i-th scale of the depth compressed feature , respectively, to obtain the i-th scale of the RGB enhanced feature and the i-th scale of the depth feature ; The first feature splicing layer splices the i-th scale of the RGB enhanced feature and the i-th scale of the depth compressed feature along the channel dimension to obtain the i-th scale of the first spliced feature ; The residual generation layer performs convolution operation on the i-th scale of the first spliced feature to generate the i-th scale of the residual feature ; The feature enhancement layer performs residual connection on the i-th scale of the residual feature and the i-th scale of the depth compressed feature to obtain the i-th scale of the enhanced depth feature ; meanwhile, the i-th scale of the RGB enhanced feature and the i-th scale of the enhanced depth feature After performing residual connection, the RGB enhancement feature at the i-th scale is obtained. .

[0009] Furthermore, the adaptive feature fusion module in step 2.3 includes: a second feature concatenation layer, a weight generation network, and a spatial weighted fusion layer; Step 2.3.1: The second feature splicing layer will and The second stitching feature is stitched together along the channel dimension to form the i-th scale. ; Step 2.3.2: The weight generation network pairs After expanding the number of channels, batch normalization and ReLU activation are performed. The resulting features are then compressed through convolutional layers to generate the weight map for the i-th scale. ; Step 2.3.3: The spatial weighted fusion layer will Split into RGB weight map of the i-th scale and the depth weight map at the i-th scale And respectively through the Softmax function to and After normalization, the RGB weight map normalized to the i-th scale is obtained. and the depth weight map after normalization at the i-th scale Therefore, the fusion feature of the i-th scale can be obtained using equation (2). : (2) In equation (2), This represents matrix multiplication.

[0010] Furthermore, the target detection head in step 2.4 includes: a feature pyramid network and a multi-scale prediction head; Step 2.4.1: The feature pyramid network pairs After multi-scale feature extraction and fusion, multi-scale pyramid features are obtained. ,in, Represents the pyramid feature at the i-th scale; When i=k, the fused features at the i-th scale After sequential upsampling and convolution, the resulting dimensionality-reduced features are used as the pyramid features at the k-th scale. ; When i = k-1, k-2, ..., 1, the fused features at the i-th scale After sequential upsampling and convolution, the resulting dimensionality-reduced features are fused with the features at the (i+1)th scale. The addition is performed to obtain the pyramid feature of the i-th scale ; thereby obtaining the pyramid feature of the 1st scale ; Step 2.4.2: the multi-scale prediction head is composed of a convolution branch, a classification branch and a regression branch; The convolution branch is used to perform convolution on the shared intermediate feature and then perform ReLU activation operation to obtain the multi-scale shared intermediate feature ; wherein, represents the i-th scale shared intermediate feature. The classification branch is used to perform convolution and Sigmoid operation on the shared intermediate feature in turn to obtain the prediction probability of the i-th scale prediction box ; wherein, represents the prediction probability of the j-th prediction box of the i-th scale. The regression branch is used to perform convolution and channel grouping on the shared intermediate feature to obtain the regression target of the i-th scale prediction box , wherein, represents the d-dimensional regression target of the j-th prediction box of the i-th scale, including: prediction box center coordinates and scaling factor.

[0011] Further, the multi-task loss function in step 3 includes a classification loss , a regression loss and a modal alignment loss , and is obtained by formula (3): (3) In formula (3), and are the balance coefficients of the regression loss and the modal alignment loss , and have: ] (4) (5) (6) In the formula, represents the loss ; represents the pooling RGB feature obtained by global average pooling; represents the pooling depth feature obtained by global average pooling; represents the square of the two-norm. ​​The electronic device comprises a memory and a processor, and is characterized in that the memory is used for storing a program supporting the processor to execute the multi-modal target detection method, and the processor is configured to execute the program stored in the memory.

[0012] The computer readable storage medium stores a computer program, and is characterized in that the computer program is executed by a processor to perform the steps of the multi-modal target detection method.

[0013] Compared with the prior art, the present application has the following advantages: 1. The present application uses a depth information guided geometric modulation mechanism to dynamically adjust the RGB feature sampling position by using a spatial offset field generated by a depth map, significantly enhances the adaptability of the model to object geometric deformation, and substantially improves the target boundary recognition accuracy in complex occlusion scenes. This innovative design effectively overcomes the recognition limitations of traditional methods in object deformation and occlusion scenes.

[0014] 2. In terms of cross-modal interaction, the bidirectional feature correction unit constructed by the present application establishes a synergistic enhancement mechanism of depth information and visual information. The boundary information of the depth map strengthens the object representation of the RGB feature, while the RGB texture information effectively corrects the noise area in the depth map, significantly improves the target detection capability under low light conditions, and greatly reduces the false detection rate in rain, fog and other harsh environments.

[0015] 3. The spatial adaptive fusion mechanism of the present application assigns a dominant weight to the depth feature in the object boundary area through a pixel-level weight distribution strategy, and enhances the contribution of the RGB feature in the texture-rich area. This fusion method solves the simple weighting problem in the integration of multi-modal features in traditional methods, and significantly improves the detection robustness in complex environments.

[0016] 4. The innovative introduction of the modal alignment loss function of the present application builds a cross-modal feature correlation bridge, and through gradient normalization technology and adaptive learning rate scheduling, the model training process is more stable and efficient. The entire system significantly reduces power consumption while maintaining real-time performance, providing reliable technical support for safety-sensitive scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 is a flowchart of the method of the present application; Figure 2 is a flowchart of the cross-modal feature interaction module of the method of the present application; Figure 3 is a flowchart of the adaptive feature fusion module of the method of the present application. DETAILED DESCRIPTION

[0018] In this embodiment, asFigure 1 As shown, a multimodal target detection method based on depth information maps includes the following steps: Step 1: Obtain an RGB image and a depth image to be detected, and perform normalization and missing value imputation on each image to obtain the preprocessed RGB image. and depth images ; During the normalization stage, the RGB image is linearly mapped to [0,255] to [0,1], and the depth map is normalized according to the camera ranging range. Missing depth values ​​are filled by interpolation based on guided filtering, and edge-preserving filtering is combined to eliminate noise, ensuring that the depth information extracted by subsequent features is continuous and accurate. make and The true annotations include: the target's true detection bounding box and the regression target set. and category tag set ,in, Let x represent the d-dimensional true regression target of the j-th predicted box at the i-th scale, including the center coordinates and scaling factors: x is the horizontal position, y is the vertical position, w is the width scaling factor, and h is the height scaling factor. Let represent the category label of the j-th predicted bounding box at the i-th scale. Indicates the total number of scales; This indicates the total number of prediction boxes contained at each scale.

[0019] Step 2: Construct a multimodal target detection network, including: a two-stream feature extraction module, a cross-modal feature interaction module, an adaptive feature fusion module, and a target detection head; Step 2.1: The dual-stream feature extraction module extracts features through parallel RGB backbone networks and deep backbone networks respectively. Multiscale RGB features and Multiscale depth features ;in, This represents the RGB features at the i-th scale extracted by the RGB backbone network. This represents the depth feature at the i-th scale extracted by the deep backbone network.

[0020] In this embodiment, the RGB backbone network in the dual-stream feature extraction module in step 2.1 consists of k cascaded convolutional blocks, wherein each convolutional block is composed of a convolutional layer, a batch normalization layer and a ReLU activation layer cascaded together. When i=1, The i-th convolutional block is processed to obtain the RGB features at the i-th scale. ; the RGB feature of the i-th scale is obtained by inputting the i-th convolutional block for processing the RGB feature of the i-th scale is obtained by inputting the i-th convolutional block for processing , and the RGB feature of the k-th scale is output by the k-th convolutional block .

[0021] The deep backbone network in the dual-flow feature extraction module in step 2.1 includes k ConvNeXt blocks, and each ConvNeXt block is sequentially formed by cascading a depth separable convolution layer, a layer normalization layer, a channel expansion convolution layer, a GELU activation layer, and a channel compression convolution layer; when i = 1, the depth feature of the i-th scale is obtained by inputting the i-th ConvNeXt block for processing ; when i = 2, 3, …, k, the depth feature of the i-1-th scale is obtained by inputting the i-1-th ConvNeXt block for processing the depth feature of the i-th scale is obtained by inputting the i-th ConvNeXt block for processing , and the depth feature of the k-th scale is output by the k-th ConvNeXt block ; The deep backbone network adopts a ConvNeXt structure, which can not only reduce the computational complexity by using a depth separable convolution, but also enhance the feature expression capability through a channel expansion and compression mechanism, so that the network is lightweight and has strong context perception capability.

[0022] Step 2.2: The cross-modal feature interaction module uses a feature modulation unit and a bidirectional feature correction unit to sequentially interact with and , and correspondingly outputs the RGB enhanced feature of the i-th scale and the depth feature of the i-th scale ; In this embodiment, as shown in Figure 2 , the feature modulation unit in the cross-modal feature interaction module of step 2.2 sequentially includes a channel compression layer, a feature alignment layer, a geometric modulation layer, and an activation layer; and the bidirectional feature correction unit sequentially includes a first feature splicing layer, a residual generation layer, and a feature enhancement layer.

[0023] Step 2.2.1: The feature modulation unit processes and to obtain the activated feature of the i-th scale ; The channel compression layer compresses the channel number of by convolution to obtain the depth compressed feature of the i-th scale ; and With After inputting into the feature alignment layer for element-by-element addition, the aligned feature of the i-th scale is obtained ; the geometric modulation layer performs convolution operation to generate the spatial offset field of the i-th scale , so that the geometric modulation feature of the i-th scale is obtained through formula (1) : (1) In formula (1), DeformConv() represents a deformable convolution operator, and offset represents an offset field parameter; the generation of the offset field is based on the RGB feature and the depth feature, so that the deformable convolution can adaptively adjust the sampling position for different modalities, and accurately align the cross-modality target edge; The activation layer adopts a LeakyReLU activation function to process , and outputs the activated feature of the i-th scale .

[0024] Step 2.2.2: The bidirectional feature correction unit respectively processes and , and and , and correspondingly obtains the RGB enhanced feature of the i-th scale and the depth feature of the i-th scale ; The first feature splicing layer splices and along the channel dimension to obtain the first spliced feature of the i-th scale ; The residual generation layer performs convolution operation on to generate the residual feature of the i-th scale ; The feature enhancement layer performs residual connection on and to obtain the enhanced depth feature of the i-th scale ; at the same time, residual connection is performed on and to obtain the RGB enhanced feature of the i-th scale .

[0025] Step 2.3: The adaptive feature fusion module fuses and to obtain the fused feature of the i-th scale ; In this embodiment, as Figure 3As shown, the adaptive feature fusion module in step 2.3 includes: a second feature splicing layer, a weight generation network, and a spatial weighted fusion layer; Step 2.3.1: The second feature splicing layer splices the second spliced features of the i-th scale along the channel dimension ; ; ; Step 2.3.2: The weight generation network expands the channel number of the second spliced features of the i-th scale, and then performs batch normalization and ReLU activation processing, and the obtained features are compressed in the channel number through a convolutional layer, thereby generating the weight map of the i-th scale ; ; The weight generation network learns the cross-modal attention distribution through the structure of expansion and compression, and takes into account the channel attention and spatial attention, so that the fusion process can adaptively allocate modal weights at the global and local levels.

[0026] Step 2.3.3: The spatial weighted fusion layer splits the second spliced features of the i-th scale into the RGB weight map of the i-th scale ; and the depth weight map of the i-th scale , and respectively performs normalization processing through the Softmax function, respectively obtaining the normalized RGB weight map of the i-th scale ; and the normalized depth weight map of the i-th scale ; thereby obtaining the fusion features of the i-th scale : (2) In equation (2), denotes matrix multiplication.

[0027] Step 2.4: The target detection head processes the multi-scale fusion features , and outputs the network prediction results, including target prediction categories and target prediction box coordinates; In this embodiment, the target detection head in step 2.4 includes: a feature pyramid network, and a multi-scale prediction head; The feature pyramid network is used for integrating and enhancing features of different scales, including top-down feature extraction and feature fusion. The multi-scale target detection head is mainly composed of a classification branch and a regression branch, which are respectively used for predicting target class probabilities and bounding box regression parameters.

[0028] Step 2.4.1: The feature pyramid network performs multi-scale feature extraction and fusion on the second spliced features of the i-th scale , and obtains the multi-scale pyramid features , wherein denotes the pyramid features of the i-th scale; the fusion feature of the i-th scale After upsampling and convolution are sequentially performed, the reduced dimension feature obtained is taken as the pyramid feature of the k-th scale ; the fusion feature of the i-th scale After upsampling and convolution are sequentially performed, the reduced dimension feature obtained is taken as the pyramid feature of the i-th scale After addition is performed, the pyramid feature of the i-th scale is obtained ; thereby obtaining the pyramid feature of the 1st scale .

[0029] Step 2.4.2: The multi-scale prediction head is composed of a convolution branch, a classification branch and a regression branch; After convolution is performed on , and a ReLU activation operation is performed, the multi-scale shared intermediate feature is obtained; wherein, represents the i-th scale shared intermediate feature; The classification branch sequentially performs convolution and Sigmoid operation on to obtain the prediction probability of the i-th scale prediction box ; wherein, represents the prediction probability of the j-th prediction box of the i-th scale; The regression branch performs convolution and channel grouping on to obtain the regression target of the i-th scale prediction box , wherein, represents the d-dimensional regression target of the j-th prediction box of the i-th scale, including the prediction box center coordinates and the scaling factor: x is the horizontal position, y is the vertical position, w is the width scaling factor, and h is the height scaling factor.

[0030] Step 3: Based on the real label and the network prediction result, a multi-task loss function is constructed , including a classification loss , a regression loss and a modal alignment loss , and is obtained by formula (3): (3) In formula (3), and are the balance coefficients of the regression loss and the modal alignment loss , and have: ] (4) (5) (6) In formula (6), represents loss, represents pooling RGB features obtained by global average pooling; represents pooling depth features obtained by global average pooling; represents the square of the two-norm. By extracting the global representation of the RGB features and the depth features respectively through global average pooling, and minimizing the difference of their two-norm, it is helpful to unify the feature space.

[0031] Step 4: training the target detection network by using gradient descent method, and calculating the task loss function to update the network parameters until the maximum training times or the task loss function converges, thereby obtaining an optimal target detection model for target recognition on input RGB images and depth images to be measured. In this embodiment, an electronic device includes a memory for storing a program supporting a processor to execute the above-mentioned multi-modal target detection method, and the processor is configured to execute the program stored in the memory.

[0032] In this embodiment, a computer readable storage medium has a computer program stored thereon, and the computer program is executed by a processor to perform the steps of the above-mentioned multi-modal target detection method.

Claims

1. A method for multi-modal object detection based on depth information map, characterized in that, The method comprises the following steps: Step 1: Obtain an RGB image and a depth image to be detected, and perform normalization and missing value filling on the two images respectively to obtain a preprocessed RGB image and a depth image ;​ Let and The real label information of the target in the image, including: the regression target set of the real detection frame and the category label set Wherein, represents the d-dimensional real regression target of the jth real detection frame under the ith scale, including: the center coordinates and the scaling factor; represents the category label of the jth real detection frame under the ith scale, represents the total number of scales; represents the total number of real detection frames contained in each scale; Wherein, x is the horizontal position, y is the vertical position, w is the width scaling factor, and h is the height scaling factor Step 2: constructing a multi-modal target detection network, comprising: a double-flow feature extraction module, a cross-modal feature interaction module, an adaptive feature fusion module and a target detection head; Step 2.1: The dual-stream feature extraction module extracts features through parallel RGB backbone networks and deep backbone networks respectively. Multiscale RGB features and Multiscale depth features ;in, This represents the RGB features at the i-th scale extracted by the RGB backbone network. This represents the depth feature at the i-th scale extracted by the deep backbone network; Step 2.2: the cross-modal feature interaction module sequentially utilizes a feature modulation unit and a bidirectional feature correction unit to interactively process the RGB feature and the depth feature , and correspondingly outputs the RGB enhanced feature of the i-th scale and the depth feature of the i-th scale ; Step 2.3: The adaptive feature fusion module fuses the features of the i-th scale and to obtain the fused features of the i-th scale ; Step 2.4: the target detection head fuses the multi-scale features The processing is performed to output a network prediction result, including a target prediction category and target prediction box coordinates. Step 3: Based on the real label information and network prediction results, a multi-task loss function is constructed ; Step 4: training the target detection network by using gradient descent method and calculating the task loss function to update the network parameters until the maximum training number or the task loss function converges, thereby obtaining an optimal target detection model for target recognition on input RGB images and depth images.

2. The multi-modal object detection method of claim 1, wherein, The RGB backbone network in the double-flow feature extraction module in step 2.1 is composed of k convolutional blocks in cascade, wherein each convolutional block is composed of a convolutional layer, a batch normalization layer and a ReLU activation layer in cascade; When i = 1, Input into the i-th convolutional block for processing to obtain the RGB feature of the i-th scale ; RGB features of the i-th scale are output by the k-th convolutional block RGB features of the i-th scale are output by the k-th convolutional block RGB features of the i-th scale are output by the k-th convolutional block ; The depth backbone network in the double-flow feature extraction module in step 2.1 comprises k ConvNeXt blocks, and each ConvNeXt block is composed of a depth separable convolutional layer, a layer normalization layer, a channel expansion convolutional layer, a GELU activation layer and a channel compression convolutional layer in cascade; When i = 1, The input is processed in the i-th ConvNeXt block to obtain the deep feature of the i-th scale ; when i = 2, 3, …, k, the depth feature of the i-1th scale is input into the i-th ConvNeXt block for processing, and the depth feature of the i-th scale is obtained , so that the depth feature of the k-th scale is output by the k-th ConvNeXt block .

3. The multi-modal object detection method of claim 1, wherein, The feature modulation unit in the cross-modal feature interaction module of step 2.2 comprises a channel compression layer, a feature alignment layer, a geometric modulation layer and an activation layer in cascade; the bidirectional feature correction unit comprises a first feature concatenation layer, a residual generation layer and a feature enhancement layer in cascade; Step 2.2.1: The feature modulation unit processes the input feature map and to obtain the activated feature of the i-th scale ; The channel compression layer performs channel number compression on the feature map by convolution to obtain a deep compression feature of the i-th scale ; the feature alignment layer is input with and , and after element-by-element addition, an alignment feature of the i-th scale is obtained ; the geometric modulation layer performs convolution operation on to generate a spatial offset field of the i-th scale , so that a geometric modulation feature of the i-th scale is obtained by formula (1) : (1) In formula (1), DeformConv() represents a deformable convolution operator, and offset represents an offset field parameter; The activation layer adopts a LeakyReLU activation function to process and outputs an activation feature of the i-th scale ; Step 2.2.2: The bidirectional feature correction unit respectively... and ,as well as and After processing, the RGB enhancement features at the i-th scale are obtained. and depth features at the i-th scale ; The first feature splicing layer will and After stitching along the channel dimension, the first stitched feature at the i-th scale is obtained. ; The residual generation layer pair performs a convolution operation to generate residual features of the i-th scale ; The feature enhancement layer will After residual connection with , the enhanced deep feature of the i-th scale is obtained ; at the same time, after residual connection with and , the RGB enhanced feature of the i-th scale is obtained .

4. The multi-modal object detection method of claim 3, wherein, The adaptive feature fusion module in step 2.3 comprises a second feature concatenation layer, a weight generation network and a spatial weighted fusion layer; Step 2.3.1 : The second feature stitching layer stitches and along the channel dimension into the second stitched feature of the i-th scale ; Step 2.3.2: The weight generation network expands the number of channels of the feature map, and then performs batch normalization and ReLU activation processing, and the obtained feature is compressed in the number of channels through a convolutional layer to generate the weight map of the i-th scale . ​ Step 2.3.3: The spatially weighted fusion layer will split the RGB weight map of the i-th scale and the depth weight map of the i-th scale into and respectively, and normalize them by Softmax function respectively, to get the normalized RGB weight map of the i-th scale and the normalized depth weight map of the i-th scale ; thus the fusion feature of the i-th scale is obtained by formula (2) : (2) In formula (2), represents matrix multiplication.

5. The multi-modal object detection method of claim 1, wherein, The target detection head in step 2.4 comprises a feature pyramid network and a multi-scale prediction head; Step 2.4.2: the multi-scale prediction head is composed of a convolutional branch, a classification branch and a regression branch; Step 2.4.1: the feature pyramid network extracts and fuses multi-scale features to obtain multi-scale pyramid features wherein, represents the pyramid feature of the i-th scale.​ the fusion feature of the i-th scale when i=k after sequentially performing upsampling and convolution, the obtained dimension-reduced feature is taken as the pyramid feature of the k-th scale ; When i=k-1, k-2, …, 1, the fusion feature of the i-th scale is obtained After sequentially performing upsampling and convolution, the obtained dimension-reduced feature is added to the fusion feature of the i+1-th scale to obtain the pyramid feature of the i-th scale ; thereby obtaining the pyramid feature of the first scale ; The memory is used to store a program supporting the processor to execute the multi-modal target detection method of any one of claims 1-6, and the processor is configured to execute the program stored in the memory. The convolution branch is used for convolution operation and ReLU activation operation to obtain multi-scale shared intermediate features ; wherein, denotes the i-th scale shared intermediate feature The classification branch sequentially performs convolution and Sigmoid operation on to obtain the prediction probability of the prediction box of the i-th scale ; wherein, represents the prediction probability of the j-th prediction box of the i-th scale. The regression branch pair performs convolution and channel grouping to obtain the prediction box regression target of the i-th scale wherein, represents the d-dimensional regression target of the j-th prediction box of the i-th scale, including: prediction box center coordinates and scaling factors.

6. The multi-modal target detection method of claim 1, wherein, The multitask loss function in step 3 comprises a classification loss , a regression loss and a modality alignment loss and is given by equation (3): (3) In formula (3), and are the balancing coefficients of the regression loss and the modal alignment loss respectively, and have: ] (4) (5) (6) wherein denotes loss; denotes pooling RGB features obtained by global average pooling; denotes pooling depth features obtained by global average pooling; denotes the square of the two-norm.

7. An electronic device comprising a memory and a processor, characterized in that The computer program is executed by the processor to perform the steps of the multi-modal target detection method of any one of claims 1-6.

8. A computer-readable storage medium storing a computer program thereon, wherein ​

Images