Depth information fused difficult target detection method and device

Abstract

The invention provides a difficult target detection method and device fused with depth information, relates to the technical field of computer vision, and aims to solve the technical problems that an existing target detection method is difficult to detect a difficult target and is too high in training cost and calculation overhead. The method comprises the following steps: inputting a to-be-detected image data set into a pre-trained target detection model, and extracting multi-scale features in the to-be-detected image data set; performing characteristic enhancement processing on the multi-scale characteristics; searching a target in the image through a cross attention mechanism based on the multi-scale features after the feature enhancement processing to obtain frame coordinates and type information of the target; distance information of the target is obtained according to the frame coordinates and the type information of the target; wherein the target detection model is trained by using a depth map pseudo label as a supervision signal, and in the training process, the distance information of each target is obtained by fusing the frame coordinate and type information of each target with the depth information of each target.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision technology and artificial intelligence technology, and more specifically, to a method and device for detecting difficult targets by integrating depth information. Background Art

[0002] Object detection, one of the core tasks in computer vision, plays a fundamental role in modern industrial applications. This task aims to identify the category of a specific object in an image or video and accurately locate its spatial position. It is essentially a complex problem that integrates classification and regression.

[0003] While detection performance in conventional scenarios has matured, detection accuracy in certain complex scenarios still faces significant challenges. Researchers have focused on improving difficult target detection performance through data augmentation, model architecture improvements, advanced post-processing techniques, feature fusion, context modeling, and multimodal information fusion. However, these methods are common approaches in computer vision and, while they have improved difficult target detection performance to a certain extent, they are limited in scope and do not address the fundamental nature of difficult target detection.

[0004] In current research on difficult target detection, a common approach is to increase the number and diversity of training samples and perform data augmentation. Another approach is to extract multi-scale features, but these contain a lot of redundant and task-irrelevant information. Failure to efficiently extract key information from these features can impair model performance. Yet another approach is multimodal data fusion, where RGB images, infrared images, and LiDAR images are fed into the model for prediction. However, these methods all suffer from high training costs and computational overhead. Therefore, overall, existing target detection methods struggle to detect difficult targets and suffer from excessive training costs and computational overhead. Summary of the Invention

[0005] In view of this, the present invention provides a difficult target detection method and device that integrates depth information, aiming to solve the technical problems that existing target detection methods have difficulty in detecting difficult targets and have excessive training costs and computational overheads.

[0006] One aspect of the present invention provides a difficult target detection method that integrates depth information, including: obtaining image data to be detected, and preprocessing the image data to generate an image dataset to be detected; inputting the image dataset to be detected into a pre-trained target detection model, and performing the following operations: extracting multi-scale features from the image dataset to be detected; performing feature enhancement processing on the multi-scale features; based on the multi-scale features after feature enhancement processing, searching for targets in the image through a cross-attention mechanism to obtain the target's bounding box coordinates and type information; obtaining the target's distance information based on the target's bounding box coordinates and type information; wherein the target detection model is trained using depth map pseudo-labels as supervisory signals, and during the training process, the distance information of each target is obtained by fusing the bounding box coordinates and type information of each target with the depth information of each target.

[0007] According to an embodiment of the present invention, the target detection model is configured as a Detect Transformer model, wherein the structure of the Detect Transformer model includes: a backbone network for extracting multi-scale features from the image data set to be detected; a variable attention module for acting as an encoder to perform feature enhancement processing on the multi-scale features, and for acting as a decoder to search for targets in the image through a cross-attention mechanism based on the multi-scale features after feature enhancement processing to obtain the bounding box coordinates and type information of the target; a projector for outputting the bounding box coordinates and type information of the target; and for outputting the depth information of the target during training; a linear fusion module for obtaining the distance information of the target based on the bounding box coordinates and type information of the target; and for fusing the bounding box coordinates and type information of each target with the depth information of each target during training to obtain the distance information of each target.

[0008] According to an embodiment of the present invention, the variable attention module is configured with a first cross-attention mechanism and a second cross-attention mechanism; the first cross-attention mechanism is configured to be able to act as a first decoder during training to search for targets in the image according to the target query vector, and obtain the bounding box coordinates and type information of each target; the second cross-attention mechanism is configured to be able to act as a second decoder during training to predict the depth of the target according to the depth query vector, and obtain the depth information of each target; wherein the first decoder and the second decoder share weights.

[0009] According to an embodiment of the present invention, the training process of the target detection model includes: obtaining a training sample set, and performing data enhancement processing on the sample set to obtain an enhanced sample set; inputting the enhanced sample set into a pre-trained monocular depth estimation model to generate pseudo labels to obtain depth map pseudo labels; inputting the depth map pseudo labels into the target detection model, and performing the following operations: using the backbone network to extract multi-scale features in the image data set to be detected; using the variable attention module as an encoder to perform feature enhancement processing on the multi-scale features; based on the multi-scale features after feature enhancement processing, searching for targets in the image through a first cross-attention mechanism, and outputting the target's bounding box coordinates and type information through a projector; based on the multi-scale features after feature enhancement processing, predicting the depth of the target through a second cross-attention mechanism, and outputting the target's depth information through a projector; using the linear fusion module to fuse the bounding box coordinates and type information of each target with the depth information of each target to obtain the distance information of each target.

[0010] According to an embodiment of the present invention, performing data enhancement processing on a sample set to obtain an enhanced sample set includes: performing geometric transformation processing on the sample set; performing color transformation processing on the sample set after the geometric transformation processing; and performing noise injection processing on the sample set after the color transformation processing.

[0011] According to an embodiment of the present invention, the distance information of each target is obtained by fusing the bounding box coordinates and type information of each target with the depth information of each target, including: performing matrix multiplication operations on the bounding box coordinates and type information of each target and the depth information of each target to obtain a multiplication matrix; normalizing the multiplication matrix to obtain a weight matrix; and obtaining the distance information of each target based on the weight matrix.

[0012] According to an embodiment of the present invention, the target detection model is configured with a first loss function and a second loss function; the first loss function is used to achieve a one-to-one matching of each target query with the ground truth value through a two-way matching method during the target detection process, wherein the matching cost simultaneously considers the classification prediction and the bounding box prediction; the second loss function is used to select the corresponding area from the pseudo-label of the depth map according to the real bounding box of the target query after two-way matching during the depth estimation process, and calculate the average depth value of the area.

[0013] Another aspect of the present invention provides a difficult target detection device that integrates depth information, including: an acquisition module for acquiring image data to be detected, and preprocessing the image data to generate an image data set to be detected; an execution module for inputting the image data set to be detected into a pre-trained target detection model, and performing the following operations: extracting multi-scale features from the image data set to be detected; performing feature enhancement processing on the multi-scale features; based on the multi-scale features after feature enhancement processing, searching for targets in the image through a cross-attention mechanism to obtain the target's bounding box coordinates and type information; obtaining the target's distance information based on the target's bounding box coordinates and type information; wherein the target detection model is trained using depth map pseudo-labels as supervisory signals. During the training process, the distance information of each target is obtained by fusing the bounding box coordinates and type information of each target with the depth information of each target via the bounding box coordinates and type information of each target.

[0014] Another aspect of the present invention provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the above method.

[0015] Another aspect of the present invention provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the above method when executed.

[0016] Another aspect of the present invention provides a computer program product, which includes computer executable instructions. When the instructions are executed, they are used to implement the above method.

[0017] Compared with the prior art, the method and device for detecting hidden objects by integrating depth information provided by the embodiments of the present invention have at least the following beneficial effects:

[0018] (1) The method and device for difficult target detection that integrates depth information provided by the embodiments of the present invention innovatively optimize the monocular depth estimation task and the target detection task in a coordinated manner based on the inherent characteristics of the target detection task. This cross-task fusion mechanism significantly improves the target detection performance of the model in complex scenes by enhancing the model's ability to understand the three-dimensional structure of the scene, especially for difficult targets that are difficult to detect with traditional methods (such as occluded targets and small-scale targets). The target detection model only uses the pseudo-label of the depth map as a supervisory signal to guide the model to learn depth information, without the need to collect real depth data or introduce auxiliary data from other modalities. This lightweight design based on a single RGB image input not only effectively reduces the training cost of the model, but also maintains the real-time advantage of the model in practical applications.

[0019] (2) The method and apparatus for difficult target detection that integrates depth information provided by the embodiments of the present invention apply an existing depth estimation model to create a pseudo-label for the depth map of an image, and combine the monocular depth estimation task with the target detection task, thereby guiding the model to learn more valuable information and improving its performance in difficult target detection.

[0020] (3) The difficult target detection method and device provided by the embodiment of the present invention that integrates depth information uses a depth query vector to search for depth-related information from features, and interacts with the target query vector to predict the depth of the target, thereby greatly improving the accuracy of target detection.

[0021] (4) The difficult target detection method and device for integrating depth information provided by the embodiments of the present invention only performs the depth estimation task during the training process, and the execution of the target detection task does not depend on the depth estimation task. Therefore, the depth estimation task branch can be directly removed during reasoning (application) without adding additional computational overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0023] Figure 1 A flowchart of a method for detecting difficult targets by integrating depth information according to an embodiment of the present invention is schematically shown;

[0024] Figure 2 The following schematically shows a flow chart of a method for training a target detection model according to an embodiment of the present invention;

[0025] Figure 3 Schematically illustrates a comparison diagram of the principles of the target detection model in the training phase and the inference phase according to an embodiment of the present invention;

[0026] Figure 4 A block diagram schematically illustrates a structure of a difficult target detection device integrating depth information according to an embodiment of the present invention;

[0027] Figure 5 The structure block diagram of an electronic device suitable for implementing a difficult target detection method integrating depth information according to an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0028] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concept of the present invention.

[0029] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the presence of the features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0030] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0031] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0032] In the embodiments of the present invention, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of all data involved (including, but not limited to, user personal information) comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures are taken to prevent unauthorized access to user personal information data and maintain the security of user personal information and network security.

[0033] Object detection, one of the core tasks in computer vision, plays a fundamental role in modern industrial applications. This task aims to identify the category of a specific object in an image or video and accurately locate its spatial position. It is essentially a complex problem that integrates classification and regression.

[0034] From an application perspective, object detection technology has been widely used in a number of key areas, such as vehicle and pedestrian detection in intelligent transportation systems, abnormal behavior identification in security monitoring, lesion detection in medical diagnosis, and environmental monitoring and precision agriculture management in the low-altitude economy. These application scenarios fully demonstrate the practical value and social benefits of object detection technology.

[0035] Although detection performance in conventional scenarios has become mature, detection accuracy still faces significant challenges in some complex scenarios. Specifically, these challenges are mainly reflected in three typical problems: (1) small target detection, because the target occupies a relatively small proportion in the image, its feature information is easily submerged by background noise; (2) occluded target detection, the target is partially or completely occluded, resulting in a significant reduction in recognizable features; (3) blurred target detection, motion blur causes the target boundary to be blurred and texture features to be lost. These factors jointly restrict the further improvement of detection performance.

[0036] Current mainstream object detection algorithms can be divided into three major technical routes. Specifically, the R-CNN (Region-based Convolutional Neural Network) series adopts a two-stage detection framework, first generating region proposals and then performing fine classification and regression; the YOLO (You Only Look Once) series implements end-to-end single-stage detection, directly predicting the target category and location; the DETR (Detect Transformer) series innovatively transforms the detection task into a set prediction problem, achieving fully end-to-end training through the Transformer architecture, eliminating post-processing steps such as non-maximum suppression in traditional methods.

[0037] Researchers have been working to improve the performance of difficult object detection through data augmentation, model architecture improvements, advanced post-processing techniques, feature fusion, context modeling, and multimodal information fusion. These methods are common approaches in computer vision and, while they have improved difficult object detection performance to some extent, they are generally task-specific and do not address the fundamental nature of difficult object detection.

[0038] In current research on difficult object detection, a common approach is to increase the number and diversity of training samples and simulate difficult scenes through data augmentation, such as masking and interference. However, collecting and labeling large amounts of data requires significant time and labor. Furthermore, the scenes simulated through data augmentation still differ from real-world scenes, providing insufficient effective knowledge for the model. Another approach is to extract multi-scale features, such as by constructing a feature pyramid to fuse features from different layers of the model to achieve a comprehensive understanding of information from local to global perspectives. This approach is commonly used in intensive prediction tasks, but multi-scale features contain a lot of redundant and task-irrelevant information. Failure to efficiently extract key information from these features can impair model performance. Another approach is multimodal data fusion, which involves integrating RGB, infrared, and LiDAR images into the model for prediction. This allows for a description of the scene from different perspectives and provides complementary information. However, collecting paired multimodal data is costly, increasing computational overhead and reducing the model's real-time performance. Overall, existing object detection methods struggle to detect difficult objects and are prohibitively expensive to train and compute.

[0039] It's worth noting that humans perceive the world in three dimensions, but RGB images only depict a two-dimensional world. The lack of depth information impairs object detection performance. When performing object detection, depth information can be used to hierarchically separate scenes from near to far, deconstructing them and accurately identifying and locating small and occluded objects within them. Furthermore, depth estimation facilitates a comprehensive understanding of a scene, from global information to local details. Recent studies have also demonstrated a significant correlation between depth estimation and object detection. Therefore, in computer vision, incorporating additional depth information can improve object detection accuracy, especially for difficult objects.

[0040] Based on this, an embodiment of the present invention provides a difficult target detection method and device that integrates depth information, aiming to solve the technical problems that existing target detection methods have difficulty in detecting difficult targets and have excessive training costs and computational overheads.

[0041] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0042] Figure 1 The flowchart of the difficult target detection method integrating depth information according to an embodiment of the present invention is schematically shown.

[0043] like Figure 1As shown, the difficult target detection method integrating depth information in this embodiment may include operations S1 to S5.

[0044] In operation S1 , image data to be detected is acquired, and the image data is preprocessed to generate an image data set to be detected.

[0045] Input the image dataset to be detected into the pre-trained object detection model and perform the following operations S2 to S5;

[0046] In operation S2, multi-scale features are extracted from the image dataset to be detected.

[0047] In operation S3 , feature enhancement processing is performed on the multi-scale features.

[0048] In operation S4, based on the multi-scale features after feature enhancement processing, the target in the image is searched through the cross-attention mechanism to obtain the bounding box coordinates and type information of the target.

[0049] In operation S5 , distance information of the target is obtained according to the frame coordinates and type information of the target.

[0050] Among them, the target detection model is trained using depth map pseudo labels as supervision signals. During the training process, the distance information of each target is obtained by fusing the bounding box coordinates and type information of each target with the depth information of each target.

[0051] In this embodiment, the image data to be detected can be, for example, an RGB image. The purpose of target detection is to detect the bounding box coordinates and type information of the target from the RGB image, and then obtain the target distance information. It is important to note that this target detection model performs both target detection and depth estimation during training. These two tasks are relatively independent. When applied, the depth estimation module can be removed, and only the target query can be input for target detection.

[0052] The method for detecting difficult targets that incorporates depth information, provided by an embodiment of the present invention, draws on the inherent characteristics of target detection tasks and innovatively collaboratively optimizes the monocular depth estimation task with the target detection task. This cross-task fusion mechanism significantly improves the model's target detection performance in complex scenarios by enhancing the model's understanding of the scene's three-dimensional structure. This is particularly true for difficult targets that are difficult to detect with traditional methods, such as occluded and small-scale targets. This target detection model utilizes only depth map pseudo-labels as supervisory signals to guide the model's learning of depth information, without the need to collect real depth data or introduce auxiliary data from other modalities. This lightweight design, based on a single RGB image input, not only effectively reduces the model's training cost but also maintains the model's real-time advantage in practical applications.

[0053] According to an embodiment of the present invention, the target detection model is configured as a DETR (Detect Transformer) model. The structure of the DETR model may specifically include: a backbone network, a variable attention module, a projector and a linear fusion module.

[0054] Among them, the backbone network is used to extract multi-scale features from the image dataset to be detected.

[0055] The variable attention module is used as an encoder to perform feature enhancement processing on multi-scale features, and as a decoder to search for targets in the image through the cross-attention mechanism based on the multi-scale features after feature enhancement processing to obtain the bounding box coordinates and type information of the target.

[0056] The projector is used to output the bounding box coordinates and type information of the target; and to output the depth information of the target during training.

[0057] The linear fusion module is used to obtain the distance information of the target based on the target's bounding box coordinates and type information; and is used to fuse the bounding box coordinates and type information of each target with the depth information of each target during training to obtain the distance information of each target.

[0058] According to an embodiment of the present invention, the variable attention module is configured with a first cross-attention mechanism and a second cross-attention mechanism.

[0059] Among them, the first cross-attention mechanism is configured to be able to act as a first decoder during the training process to search for targets in the image according to the target query vector to obtain the bounding box coordinates and type information of each target.

[0060] The second cross-attention mechanism is configured to act as a second decoder during training to predict the depth of the target based on the depth query vector to obtain the depth information of each target.

[0061] The first decoder and the second decoder share weights.

[0062] The following will detail the efficient training strategy for the target detection model DETR according to an embodiment of the present invention.

[0063] Figure 2 The following schematically shows a flow chart of a method for training a target detection model according to an embodiment of the present invention.

[0064] like Figure 2 As shown, the target detection model of this embodiment performs target detection and depth estimation tasks simultaneously, thereby improving the model's target detection performance. The training steps are as follows:

[0065] (1) Obtain a training sample set and perform data enhancement on the sample set to obtain an enhanced sample set.

[0066] In this embodiment, for example, data enhancement technology can be used to diversify the training samples, specifically including performing geometric transformation processing (rotation, translation, scaling) on the sample set; performing color transformation processing (adjusting image brightness, contrast, saturation, etc.) on the sample set after geometric transformation processing; and performing noise injection processing on the sample set after color transformation processing.

[0067] (2) The enhanced sample set is input into the pre-trained monocular depth estimation model to generate pseudo labels and obtain the depth map pseudo labels.

[0068] In this embodiment, the enhanced sample set is input into a pre-trained monocular depth estimation model (eg, DepthAnything V2 model), and a depth map (Depth Map) having the same size as the input image is output.

[0069] The Depth Anything V2 model is currently the most advanced monocular depth estimation model. Its network architecture is simple, using Vision Transformation (Vision Transformation) as the backbone for feature extraction, Depth Transformation (DPT) as the decoder for depth prediction, and initializing the backbone parameters using the DINO V2 model (an advanced visual representation learning model). It constructs a large amount of high-quality synthetic data and collects a large amount of unlabeled data from the internet as training samples. The model is trained using a teacher-student training approach. Depth Anything V2 exhibits strong generalization and stability, generating high-precision depth maps for various scenarios.

[0070] The target detection model training method provided by the embodiment of the present invention applies the existing depth estimation model to create pseudo-labels for the depth map of the image, and combines the monocular depth estimation task with the target detection task to guide the model to learn more valuable information and improve its performance in difficult target detection.

[0071] Input the depth map pseudo-label into the target detection model DETR and perform the following steps (3) to (7).

[0072] (3) Use the backbone network to extract multi-scale features from the image dataset to be detected.

[0073] In this embodiment, ResNet or Swin Transformer is used as the backbone network to extract multi-scale features (including features at four scales, with the sizes of the challenge images being 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original images, respectively).

[0074] (4) Use the variable attention module as an editor to perform feature enhancement on multi-scale features.

[0075] In this example, a variable attention module is used as an encoder to enhance the multi-scale features extracted by the backbone network. This mechanism combines the sparse spatial sampling properties of variable convolution with the relational modeling capabilities of the Transformer. Compared to the conventional attention mechanism in the Transformer, it alleviates the problems of slow convergence and high computational cost.

[0076] (5) Based on the multi-scale features after feature enhancement, the target in the image is searched through the first cross-attention mechanism, and the bounding box coordinates and type information of the target are output through the projector.

[0077] In this embodiment, a set of object query vectors is first set. , for example, it can be represented by a set of learnable parameters, where n is the number of vectors, c is the length of each vector, and each target query represents a target. Then, the variable attention module is used as the first decoder to perform cross-attention analysis, that is, the first cross-attention mechanism is used to search for targets in the image, where q o As the query in the cross attention, the features output by the backbone network are used as the key and value, and the first decoder finally outputs , which gathers important features related to the target. Finally, the target query output by the first decoder is The linear projection layer (also known as the projector) outputs the bounding box coordinates and type information of each target.

[0078] (6) Based on the multi-scale features after feature enhancement, the depth of the target is predicted through the second cross-attention mechanism, and the depth information of the target is output through the projector.

[0079] In this embodiment, an additional set of depth query vectors is set , is also represented by a set of learnable parameters, where m is the number of vectors, and each depth query vector represents a distance range. Similarly, the depth query vector is also input into the second decoder composed of a variable attention module (the second decoder shares weights with the previous first decoder) to interact with the enhanced multi-scale features, that is, the second cross-attention mechanism is used to predict the distance information in the image, and finally the projector outputs , which gathers important features related to the target depth.

[0080] (7) Using the linear fusion module, the distance information of each target is obtained by fusing the bounding box coordinates and type information of each target with the depth information of each target. Specifically, it may include:

[0081] Perform matrix multiplication on the bounding box coordinates and type information of each target and the depth information of each target to obtain a multiplication matrix;

[0082] Normalize the multiplication matrix to obtain the weight matrix;

[0083] According to the weight matrix, the distance information of each target is obtained.

[0084] In this embodiment, the target query output by the decoder is With deep query Perform matrix multiplication, and then perform softmax calculation on the matrix to obtain a weight matrix W∈R n×m , each row in the weight matrix represents the probability value of each target corresponding to each distance range (the sum of the probabilities in each row is 1). The weight matrix W is calculated as follows:

[0085]

[0086] Then the weight matrix W is multiplied by the center value Cent of the matrix range element-wise, and then summed up to get the distance D∈R between each target and the camera. n×1 , the distance D is calculated as follows:

[0087]

[0088] According to an embodiment of the present invention, the target detection model belongs to multi-task joint learning, performing target detection and depth estimation simultaneously, and is therefore configured with two types of loss functions, namely a first loss function and a second loss function.

[0089] The first loss function is used during object detection to achieve a one-to-one match between each object query and the ground truth using a bidirectional matching method (the Xenya matching algorithm). The matching cost considers both the classification prediction and the bounding box prediction. After matching, the classification loss and bounding box loss are calculated between the predicted and true values. The classification loss uses the probabilities loss, while the bounding box loss combines the L1 loss and the IoU loss.

[0090] The second loss function is used in the depth estimation process to select the corresponding area from the pseudo-label of the depth map according to the real bounding box of the target query after bidirectional matching, and calculate the average depth value of the area as the predicted target. The depth loss uses MSE.

[0091] The target detection model training method provided by the embodiment of the present invention uses a depth query vector to search for depth-related information from features, and interacts with the target query vector to predict the depth of the target, greatly improving the accuracy of target detection.

[0092] Figure 3 The following diagram schematically illustrates a comparison of the principles of the target detection model in the training phase and the inference phase according to an embodiment of the present invention.

[0093] like Figure 3 As shown, the target detection model of the embodiment of the present invention only performs the depth estimation task during the training process, and the execution of the target detection task does not depend on the depth estimation task. Therefore, the depth estimation task branch can be directly removed during reasoning (application) without adding additional computational overhead.

[0094] Figure 4 The structure of a difficult target detection device integrating depth information according to an embodiment of the present invention is schematically shown.

[0095] like Figure 4 As shown, the difficult target detection device 400 for fusing depth information according to the embodiment of the present invention includes: an acquisition module 410 and an execution module 420 .

[0096] The acquisition module 410 is used to acquire image data to be detected, and pre-process the image data to generate an image data set to be detected.

[0097] The execution module 420 is used to input the image dataset to be detected into the pre-trained object detection model and perform the following operations:

[0098] Extract multi-scale features from the image dataset to be detected;

[0099] Perform feature enhancement on multi-scale features; based on the enhanced multi-scale features, search for targets in the image through a cross-attention mechanism to obtain the target's bounding box coordinates and type information;

[0100] According to the target's frame coordinates and type information, the target's distance information is obtained;

[0101] Among them, the target detection model is trained using the depth map pseudo-label as the supervision signal. During the training process, the distance information of each target is obtained by fusing the bounding box coordinates and type information of each target with the depth information of each target.

[0102] Any number of the modules, submodules, units, and subunits according to embodiments of the present invention, or at least part of the functionality of any number of these units, can be implemented in a single module. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present invention can be split into multiple modules for implementation. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present invention can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented in hardware or firmware using any other reasonable method of circuit integration or packaging, or can be implemented in any one of the three implementation methods of software, hardware, and firmware, or any appropriate combination of any of these. Alternatively, one or more of the modules, submodules, units, and subunits according to embodiments of the present invention can be at least partially implemented as a computer program module that, when executed, can perform the corresponding functionality.

[0103] For example, any number of acquisition module 410 and execution module 420 can be combined into a single module / unit / sub-unit, or any one of these modules / units / sub-units can be split into multiple modules / units / sub-units. Alternatively, at least part of the functionality of one or more of these modules / units / sub-units can be combined with at least part of the functionality of other modules / units / sub-units and implemented in a single module / unit / sub-unit. According to an embodiment of the present invention, at least one of acquisition module 410 and execution module 420 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented in hardware or firmware through any other reasonable means of circuit integration or packaging, or can be implemented in any one of software, hardware, and firmware, or any suitable combination of these. Alternatively, at least one of acquisition module 410 and execution module 420 can be at least partially implemented as a computer program module that, when executed, performs the corresponding functionality.

[0104] It should be noted that the part of the difficult target detection device that integrates depth information in the embodiment of the present invention corresponds to the part of the difficult target detection method that integrates depth information in the embodiment of the present invention. The description of the part of the difficult target detection device that integrates depth information specifically refers to the part of the difficult target detection method that integrates depth information, which will not be repeated here.

[0105] Figure 5The structure block diagram of an electronic device suitable for implementing a difficult target detection method integrating depth information according to an embodiment of the present invention is schematically shown. Figure 5 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0106] like Figure 5 As shown, an electronic device 500 according to an embodiment of the present invention includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage unit 508 into a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0107] The storage unit 508 stores various programs and data required for the operation of the electronic device 500. The processor 501, ROM 502, and storage unit 508 are interconnected via a bus 504. The processor 501 executes the programs stored in the ROM 502 and / or storage unit 508 to perform various operations according to the method flow of the embodiment of the present invention. It should be noted that the programs may also be stored in one or more memories other than the ROM 502 and storage unit 508. The processor 501 may also execute the programs stored in the one or more memories to perform various operations according to the method flow of the embodiment of the present invention.

[0108] According to an embodiment of the present invention, electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to bus 504. Electronic device 500 may also include one or more of the following components connected to I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section 508 including a hard disk; and a communication section 509 including a network interface card such as a LAN card or modem. Communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to I / O interface 505 as needed. Removable media 511, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 510 as needed, so that computer programs read from the removable media can be installed into storage section 508 as needed.

[0109] According to an embodiment of the present invention, the method flow according to an embodiment of the present invention can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the processor 501, the above-mentioned functions defined in the system of the embodiment of the present invention are executed. According to an embodiment of the present invention, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0110] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0111] According to embodiments of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium. Examples include, but are not limited to, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0112] For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 502 and / or the storage portion 508 described above and / or one or more memories other than the ROM 502 and the storage portion 508 .

[0113] An embodiment of the present invention also includes a computer program product, which includes a computer program, which contains program code for executing the method provided by the embodiment of the present invention. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the method provided by the embodiment of the present invention.

[0114] When the computer program is executed by the processor 501, the above functions defined in the system / device of the embodiment of the present invention are performed. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0115] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 509, and / or installed from a removable medium 511. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0116] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages include, but are not limited to, languages such as Java, C++, Python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0117] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of the systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the boxes may occur in an order different from that marked in the accompanying drawings. For example, two boxes shown in succession may actually be executed substantially in parallel, or they may sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, as well as the combination of boxes in the block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or may be implemented using a combination of dedicated hardware and computer instructions. It will be understood by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention may be combined and / or coupled in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or combinations fall within the scope of the present invention.

[0118] The above describes embodiments of the present invention. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A difficult target detection method integrating depth information, characterized in that: The method comprises: Acquire image data to be detected, and preprocess the image data to generate an image data set to be detected; Input the image dataset to be detected into the pre-trained object detection model and perform the following operations: Extracting multi-scale features from the image dataset to be detected; Performing feature enhancement processing on the multi-scale features; Based on the multi-scale features after feature enhancement, the cross-attention mechanism is used to search for objects in the image and obtain the bounding box coordinates and type information of the object; According to the target's bounding box coordinates and type information, the target's distance information is obtained; The target detection model is trained using the depth map pseudo-label as a supervisory signal. During the training process, the distance information of each target is obtained by fusing the bounding box coordinates and type information of each target with the depth information of each target.

2. The method according to claim 1, characterized in that The target detection model is configured as a DetectTransformer model, wherein the structure of the DetectTransformer model includes: A backbone network, used to extract multi-scale features from the image dataset to be detected; a variable attention module, configured to act as an encoder to perform feature enhancement processing on the multi-scale features; and to act as a decoder to search for objects in the image through a cross-attention mechanism based on the multi-scale features after feature enhancement processing to obtain bounding box coordinates and type information of the objects; The projector is used to output the bounding box coordinates and type information of the target; and is used to output the depth information of the target during training; The linear fusion module is used to obtain the distance information of the target based on the target's bounding box coordinates and type information; and is used to fuse the bounding box coordinates and type information of each target with the depth information of each target during training to obtain the distance information of each target.

3. The method according to claim 2, characterized in that The variable attention module is configured with a first cross-attention mechanism and a second cross-attention mechanism; The first crisscross attention mechanism is configured to act as a first decoder during training to search for objects in an image according to an object query vector and obtain bounding box coordinates and type information of each object; The second cross attention mechanism is configured to act as a second decoder during training to predict the depth of the object based on the depth query vector to obtain depth information of each object; The first decoder and the second decoder share weights.

4. The method according to claim 3, characterized in that The training process of the target detection model includes: Obtaining a training sample set, and performing data enhancement processing on the sample set to obtain an enhanced sample set; The enhanced sample set is input into the pre-trained monocular depth estimation model to generate pseudo labels and obtain the depth map pseudo labels; The depth map pseudo-label is fed into the object detection model and the following operations are performed: Extracting multi-scale features from the image dataset to be detected using the backbone network; Using the variable attention module as an encoder to perform feature enhancement processing on the multi-scale features; Based on the multi-scale features after feature enhancement processing, searching for the target in the image through the first cross attention mechanism, and outputting the bounding box coordinates and type information of the target through the projector; Based on the multi-scale features after feature enhancement processing, predict the depth of the target through the second cross attention mechanism, and output the depth information of the target through the projector; The linear fusion module is used to fuse the bounding box coordinates and type information of each target with the depth information of each target to obtain the distance information of each target.

5. The method according to claim 4, characterized in that The data enhancement processing is performed on the sample set to obtain an enhanced sample set, comprising: Performing geometric transformation processing on the sample set; Performing color transformation on the sample set after geometric transformation; Perform noise injection processing on the sample set after color transformation.

6. The method according to claim 4, characterized in that The distance information of each target is obtained by fusing the bounding box coordinates and type information of each target with the depth information of each target, including: Perform matrix multiplication on the bounding box coordinates and type information of each target and the depth information of each target to obtain a multiplication matrix; Performing normalization calculation on the multiplication matrix to obtain a weight matrix; According to the weight matrix, the distance information of each target is obtained.

7. The method according to claim 4, characterized in that The target detection model is configured with a first loss function and a second loss function; The first loss function is used to achieve a one-to-one matching between each target query and the ground truth value through a two-way matching method during the target detection process, wherein the matching cost considers both the classification prediction and the bounding box prediction; The second loss function is used to select the corresponding area from the pseudo-label of the depth map according to the real bounding box of the target query after bidirectional matching during the depth estimation process, and calculate the average depth value of the area.

8. A difficult target detection device integrating depth information, characterized in that: The device comprises: An acquisition module is used to acquire image data to be detected, and preprocess the image data to generate an image data set to be detected; An execution module, configured to input the image dataset to be detected into a pre-trained object detection model and perform the following operations; Extracting multi-scale features from the image dataset to be detected; Performing feature enhancement processing on the multi-scale features; Based on the multi-scale features after feature enhancement, the cross-attention mechanism is used to search for objects in the image and obtain the bounding box coordinates and type information of the object; According to the target's bounding box coordinates and type information, the target's distance information is obtained; The target detection model is trained using the depth map pseudo-label as a supervisory signal. During the training process, the distance information of each target is obtained by fusing the bounding box coordinates and type information of each target with the depth information of each target.

9. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors are enabled to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having executable instructions stored thereon, characterized in that: When the instruction is executed by a processor, the processor executes the method according to any one of claims 1 to 7.

Images