A method, apparatus, electronic device, and storage medium for target detection.

Abstract

This disclosure provides a method, apparatus, electronic device, and storage medium for target detection. The method includes: extracting features from an image to be detected to obtain a feature map of the image; generating a depth map corresponding to a projection area of ​​a target object in the image onto the ground based on the feature map; and determining three-dimensional detection information of the target object based on the depth map and the feature map. In this disclosure, the projection area is associated with the target object to some extent. Thus, the depth map corresponding to a local ground surface can specifically guide the three-dimensional detection of the feature map of the target object on that local ground surface, improving detection accuracy.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and more specifically, to a method, apparatus, electronic device, and storage medium for target detection. Background Technology

[0002] Compared to two-dimensional (2D) object detection tasks, three-dimensional (3D) object detection tasks are more difficult and complex, often requiring the extraction of 3D geometric and semantic information of objects from 3D scenes. This primarily includes the object's length, width, height, center point, and orientation angle. Among these, 3D object detection using monocular images is widely used in various fields (such as autonomous driving) due to its cost-effectiveness and practicality.

[0003] However, 3D object detection techniques based on monocular images mainly rely on external subtasks that perform tasks such as 2D object detection and depth map estimation. Since these subtasks are trained independently, they suffer from accuracy loss, limiting the performance ceiling of the network model and failing to meet the accuracy requirements of 3D detection. Summary of the Invention

[0004] This disclosure provides at least one method, apparatus, electronic device, and storage medium for target detection to improve the accuracy of 3D target detection.

[0005] In a first aspect, embodiments of this disclosure provide a method for target detection, the method comprising:

[0006] Feature extraction is performed on the image to be detected to obtain the feature map of the image to be detected;

[0007] Based on the feature map, a depth map is generated corresponding to the projection area of ​​the target object in the image to be detected onto the ground.

[0008] Based on the depth map and the feature map, the three-dimensional detection information of the target object is determined.

[0009] The aforementioned target detection method not only extracts features from the image to be detected but also generates a depth map based on the extracted feature map, corresponding to the projection area of ​​the target object in the image onto the ground. Furthermore, the 3D detection information of the target object is determined based on the depth map and the feature map. Since the generated depth map points to the target object in the image to be detected and corresponds to the projection area of ​​the target object onto the ground, this projection area is associated with the target object to some extent. Therefore, when performing 3D detection using the feature map of the target object on a local surface, the depth map corresponding to that local surface can be used as a guide, thereby improving the detection accuracy.

[0010] In one possible implementation, after obtaining the feature map of the image to be detected, the method further includes:

[0011] Two-dimensional detection is performed on the feature map to obtain two-dimensional detection information for the target object;

[0012] The step of determining the 3D detection information of the target object based on the depth map and the feature map includes:

[0013] Based on the two-dimensional detection information, determine the three-dimensional prior bounding box information for the target object;

[0014] Based on the 3D prior bounding box information, the depth map, and the feature map, the 3D detection bounding box information of the target object is determined.

[0015] The 3D detection here can be combined with 3D prior bounding box information. The 3D prior bounding box can constrain the starting position of 3D detection to a certain extent, so as to search for 3D detection box information near the starting position, thereby further improving the accuracy of 3D detection.

[0016] In one possible implementation, the two-dimensional detection information includes the two-dimensional detection bounding box information of the target object and the category information of the target object; determining the three-dimensional prior bounding box information for the target object based on the two-dimensional detection information includes:

[0017] Based on the category information of the target object, determine the clustering information of each subcategory included in the category to which the target object belongs;

[0018] Based on the clustering information of each sub-category and the two-dimensional detection box information of the target object, the three-dimensional prior box information for the target object is determined.

[0019] The 3D prior bounding box here can be determined by combining the category information of the target object. Different categories of target objects may result in different sizes, positions, and other characteristics of the corresponding 3D prior bounding boxes. Using category information can help determine the position of the 3D prior bounding box with high accuracy.

[0020] In one possible implementation, determining the 3D prior bounding box information for the target object based on the clustering information of each sub-category and the 2D detection box information of the target object includes:

[0021] For each of the subcategories, a depth value corresponding to that subcategory is determined based on the clustering height value included in the clustering information of that subcategory and the width value included in the two-dimensional detection box information of the target object.

[0022] Based on the clustering information of the sub-category and the depth value corresponding to the sub-category, a three-dimensional prior bounding box is determined for the target object.

[0023] In one possible implementation, determining the 3D detection bounding box information of the target object based on the 3D prior bounding box information, the depth map, and the feature map includes:

[0024] The offset of the 3D detection box is determined based on the depth map and the feature map;

[0025] Based on the 3D prior bounding box information and the 3D detection box offset, the 3D detection box information of the target object is determined.

[0026] This can be a prediction of the offset. Combining the offset with the 3D prior box can yield a more accurate 3D detection box.

[0027] In one possible implementation, determining the 3D detection box offset based on the depth map and the feature map includes:

[0028] Based on the location range included in the two-dimensional detection box information of the target object, a depth map and a feature map matching the location range are extracted from the depth map and the feature map, respectively.

[0029] The offset of the 3D detection box is determined based on the depth map and feature map that match the location range.

[0030] Here, by utilizing the location range of the two-dimensional detection box information of the target object, the depth map and feature map can be cropped at the corresponding positions. This will ensure that the predicted offset is specific to the target object and does not contain information related to other interfering regions, thus improving prediction accuracy.

[0031] In one possible implementation, the 3D prior bounding box information is multiple; determining the 3D detection box information of the target object based on the 3D prior bounding box information and the 3D detection box offset includes:

[0032] Determine the weights corresponding to each of the three-dimensional prior box information;

[0033] Based on each of the three-dimensional prior bounding box information, the weight corresponding to each of the three-dimensional prior bounding box information, and the offset of the three-dimensional detection box, the three-dimensional detection box information of the target object is determined.

[0034] In one possible implementation, the method further includes:

[0035] The predicted probabilities of each sub-category included in the category information of the target object are determined based on the depth map and the feature map.

[0036] The determination of the weights corresponding to each of the three-dimensional prior bounding boxes includes:

[0037] Based on the predicted probabilities of each sub-category, the weights of the 3D prior bounding box information corresponding to each sub-category are determined.

[0038] Considering that the prediction probabilities for different subcategories are not the same, the higher the probability, the higher the likelihood that the target object points to the corresponding subcategory. This allows us to assign higher weights to the corresponding 3D prior box information, which will further improve the prediction accuracy of the final 3D detection box.

[0039] In one possible implementation, the step of detecting the feature map to obtain two-dimensional detection information for the target object includes:

[0040] The offset of the two-dimensional detection box is determined based on the feature map;

[0041] Based on the preset two-dimensional prior bounding box information and the offset of the two-dimensional detection box, the two-dimensional detection information of the target object is determined.

[0042] In one possible implementation, the depth map is determined by a trained depth map generation network; the depth map generation network is trained from image samples and labeled depth maps determined based on the 3D bounding box information of the target objects labeled in the image samples.

[0043] In one possible implementation, the 3D annotation box information of the target object includes the position coordinates and depth value of the center point of the bottom surface of the annotation box; the annotation depth map is obtained according to the following steps:

[0044] Based on the correspondence between the three-dimensional coordinate system where the three-dimensional annotation box is located and the ground coordinate system where the center point of the bottom surface of the annotation box is located, the three-dimensional annotation box information of the target object is projected onto the ground to obtain the projection area of ​​the target object on the ground and the extended area where the projection area is located.

[0045] The depth value of each three-dimensional annotation point on the extended area is determined based on the position coordinates and depth value of the center point of the bottom surface of the annotation frame, which are included in the three-dimensional annotation frame information.

[0046] Based on the correspondence between the camera coordinate system and the image coordinate system, each three-dimensional annotation point on the extended region in the camera coordinate system is projected onto the pixel plane in the image coordinate system to obtain the projection points in the pixel plane.

[0047] The annotation depth map is obtained based on the depth values ​​of each three-dimensional annotation point on the extended region and the projection points in the pixel plane.

[0048] The depth map here can be achieved by combining ground projection operations and coordinate system transformation operations. By constructing an extended region, a local ground area, including the target object, can be completely covered. The corresponding depth map can be obtained using the 3D projection result of the extended region. This depth map reflects the depth information of the extended region, including the local ground area. This depth information can specifically assist in the 3D detection of target objects on the corresponding local ground area.

[0049] In one possible implementation, determining the depth value of each 3D annotation point on the extended region based on the position coordinates and depth value of the center point of the bottom surface of the annotation frame, which are included in the 3D annotation frame information, includes:

[0050] The depth value and position coordinates of the center point of the bottom surface of the annotation box are respectively determined as the depth value and position coordinates of the center point of the extended region;

[0051] Given the location coordinates of the center point of the extended region, the depth value of the center point of the extended region is used as the starting depth value, and the depth value of each three-dimensional annotation point in the extended region is determined at a preset depth interval.

[0052] Secondly, embodiments of this disclosure also provide a target detection apparatus, the apparatus comprising:

[0053] The extraction module is used to extract features from the image to be detected, thereby obtaining a feature map of the image to be detected.

[0054] The generation module is used to generate a depth map corresponding to the projection area of ​​the target object in the image to be detected onto the ground, based on the feature map.

[0055] The first detection module is used to determine the three-dimensional detection information of the target object based on the depth map and the feature map.

[0056] Thirdly, embodiments of this disclosure also provide an electronic device, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, the steps of the target detection method as described in the first aspect and any of its various embodiments are performed.

[0057] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the target detection method as described in the first aspect and its various embodiments.

[0058] For a description of the effects of the aforementioned target detection apparatus, electronic equipment, and computer-readable storage medium, please refer to the description of the target detection method above; it will not be repeated here.

[0059] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0060] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. These drawings are incorporated in and constitute a part of this specification. They illustrate embodiments conforming to this disclosure and, together with the specification, serve to explain the technical solutions of this disclosure. It should be understood that the following drawings only show some embodiments of this disclosure and should not be considered as limiting the scope. Those skilled in the art can obtain other related drawings based on these drawings without creative effort.

[0061] Figure 1 A flowchart of a target detection method provided by an embodiment of this disclosure is shown;

[0062] Figure 2 This diagram illustrates the application of the target detection method provided in the embodiments of this disclosure.

[0063] Figure 3 A schematic diagram of a target detection apparatus provided in an embodiment of this disclosure is shown;

[0064] Figure 4 A schematic diagram of an electronic device provided in an embodiment of the present disclosure is shown. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. The components of the embodiments of this disclosure described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0066] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0067] In this document, the term "and / or" merely describes a relationship, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0068] Research has shown that with the successful application of deep learning in object detection, especially in 3D object detection, detection accuracy has reached a very high level. Common 3D object detection methods are based on LiDAR data, but the high cost of data acquisition equipment makes large-scale applications and deployments difficult. Monocular image-based 3D object detection, however, can utilize automotive cameras, making it economical and readily available. For images viewed from a single perspective, the task of monocular 3D detection is to detect the 3D geometric and semantic information of the target object in a 3D scene, primarily including the object's length, width, height, center point, and orientation angle.

[0069] In related technologies, 3D object detection techniques based on monocular images mainly rely on external subtasks, which are responsible for performing tasks such as 2D object detection and depth map estimation. Since these subtasks are trained independently, they suffer from accuracy loss, limiting the performance ceiling of the network model and failing to meet the accuracy requirements of 3D detection, making them difficult to use in practical applications.

[0070] The current challenge in 3D object detection methods lies in the depth prediction of 3D bounding boxes. The labels in 3D object detection only provide depth information for the center or corner points of the bounding box, making it difficult for the network to learn and generate more abundant and accurate depth information. This is mainly because related 3D object detection methods primarily guide the learning of 3D bounding boxes through the prediction results of their subtasks, such as depth estimation, pseudo-point cloud generation, and semantic segmentation. However, these subtasks require a large number of accurate depth labels, making them difficult to use in practical applications. Furthermore, the accuracy of these subtasks limits the performance ceiling of 3D object detection, making it unreliable in 3D object detection.

[0071] Based on the above research, this disclosure provides a method, apparatus, electronic device, and storage medium for target detection to improve the accuracy of 3D target detection.

[0072] To facilitate understanding of this embodiment, a method for object detection disclosed in this disclosure will first be described in detail. The object detection method provided in this disclosure is generally executed by a computer device with certain computing capabilities. This computer device may include, for example, a terminal device, a server, or other processing devices. The terminal device may be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. In some possible implementations, the object detection method can be implemented by a processor calling computer-readable instructions stored in memory.

[0073] See Figure 1 The diagram shows a flowchart of a target detection method provided in an embodiment of this disclosure. The method includes steps S101 to S103, wherein:

[0074] S101: Extract features from the image to be detected to obtain the feature map of the image to be detected;

[0075] S102: Based on the feature map, generate a depth map corresponding to the projection area of ​​the target object in the image to be detected onto the ground;

[0076] S103: Based on the depth map and feature map, determine the 3D detection information of the target object.

[0077] To facilitate understanding of the object detection method provided in this disclosure, the application scenarios of the method will be described in detail below. The object detection method described above can be mainly applied in the field of computer vision, such as vehicle detection in autonomous driving and drone detection. Considering the widespread application of autonomous driving, vehicle detection will be used as an example in the following illustrations.

[0078] 3D object detection techniques in related technologies mainly rely on external subtasks that perform tasks such as 2D object detection and depth map estimation. Because these subtasks are trained independently, they suffer from accuracy loss, resulting in low final 3D detection accuracy.

[0079] To address the aforementioned issues, this disclosure provides a scheme for 3D detection that combines local depth maps and feature maps, resulting in higher detection accuracy.

[0080] The image to be detected in this embodiment can be an image captured in the target scene, and the images captured will vary depending on the application scenario. Taking autonomous driving as an example, the image to be detected can be an image captured by a camera device installed on the autonomous vehicle during the vehicle's movement. This image can include all target objects within the field of view of the camera device. The target objects can be vehicles in front or pedestrians in front, without specific limitations.

[0081] Before performing 3D detection, embodiments of this disclosure can extract feature maps from the image to be detected using various feature extraction methods. For example, feature maps can be extracted from the image to be detected through image processing, or a trained feature extraction network can be used to extract feature maps.

[0082] Considering that feature extraction networks can uncover deeper image features, this embodiment of the disclosure can employ a feature extraction network to extract feature maps. The feature extraction network here can be a Convolutional Neural Network (CNN). In specific applications, a CNN model including convolutional blocks, dense blocks, and transition blocks can be used. Here, a convolutional block can consist of convolutional layers, batch normalization layers, and rectified linear units (ReLU). A dense block can consist of multiple convolutional blocks and multiple skip connections. A transition block generally consists of convolutional blocks and average pooling layers. The specific composition of the convolutional blocks, dense blocks, and transition blocks, such as the number of convolutional layers and average pooling layers, can be determined based on the specific application scenario and is not specifically limited here.

[0083] To perform 3D detection, embodiments of this disclosure can generate a local depth map based on the extracted feature map. This local depth map corresponds to the projection area of ​​the target object in the image to be detected onto the ground, pointing to the depth information of the local ground associated with the target object. Since the local ground is bound to the target object to a certain extent, combining this with the extracted feature map can more accurately detect the target object.

[0084] The aforementioned local depth map can be determined using a trained depth map generation network. This network is trained to establish the correspondence between the features and depth of corresponding pixels in the image samples and the labeled depth map. Thus, when the extracted feature map is input into the trained network, it can output a depth map pointing to the projection area of ​​the target object on the ground.

[0085] In practical applications, the feature map and depth map can be cropped using the ROI-align method, so that the 3D detection of the target object can be achieved based on the cropped depth map and feature map corresponding to the target object.

[0086] The 3D detection in this embodiment can be based on residual prediction of 3D prior boxes. This is mainly because the information of the original 3D prior boxes can be used to guide subsequent 3D detection in residual prediction. For example, the original 3D prior boxes can be used as the initial position, and the 3D detection box can be searched in the vicinity of the initial position. Especially when the accuracy of the 3D prior boxes is relatively high, this will significantly improve the detection efficiency compared to directly performing 3D detection.

[0087] The aforementioned 3D prior bounding box can be determined based on 2D detection information, thus enabling 3D detection based on 3D prior bounding box information, depth map, and feature map.

[0088] The embodiments of this disclosure can determine the two-dimensional detection information of the target object according to the following steps:

[0089] Step 1: Determine the offset of the 2D detection box based on the feature map;

[0090] Step 2: Based on the preset two-dimensional prior box information and the two-dimensional detection box offset, determine the two-dimensional detection information of the target object.

[0091] Here, the two-dimensional detection information can be determined based on the calculation result between the two-dimensional detection box offset and the preset two-dimensional prior box information.

[0092] In this embodiment, the two-dimensional detection information can be obtained by performing two-dimensional detection on the feature map using a trained first object detection network. The first object detection network can be trained to establish a correspondence between the feature map of an image sample and two-dimensional annotation information, or it can be trained to establish a correspondence between the feature map of an image sample and the offset (corresponding to the difference between the two-dimensional annotation box and the two-dimensional prior box). Using the former correspondence, the two-dimensional detection information of the target object in the image to be detected can be directly determined. Using the latter correspondence, the offset can be determined first, and then the offset can be summed with the two-dimensional prior box to determine the two-dimensional detection information of the target object.

[0093] Regardless of which correspondence is used, the determined two-dimensional detection information can include the position information (x, y) of the two-dimensional detection box. 2d y 2d w 2d h 2d ), center point location information (x) p y p ), facing angle (α) 3dThe target object's category information (cls) can also include other information related to 2D detection, without specific restrictions.

[0094] Considering the superior characteristics of residual prediction, the first object detection network here can implement two-dimensional residual prediction. In practical applications, this first object detection network can first perform dimensionality reduction through a convolutional layer and a linear rectified layer, and then perform residual prediction of the two-dimensional detection boxes through multiple convolutional layers, resulting in high prediction accuracy.

[0095] In this embodiment of the disclosure, the three-dimensional prior box information can be determined based on the two-dimensional detection information determined by the first target detection network described above. Specifically, this can be achieved through the following steps:

[0096] Step 1: Based on the category information of the target object, determine the clustering information of each subcategory included in the category to which the target object belongs;

[0097] Step 2: Based on the clustering information of each subcategory and the two-dimensional detection box information of the target object, determine the three-dimensional prior box information for the target object.

[0098] Here, the 3D prior bounding box information can be determined by combining the clustering information of the subcategories within the target object's category and the 2D detection box information of the target object. This is mainly because for target objects belonging to the same category, the 3D detection results corresponding to different subcategories will have certain differences. For example, for targets belonging to the same vehicle category, the size of the 3D detection box for the car subcategory is significantly different from the size of the 3D detection box for the truck subcategory. To take into account the probability of each subcategory being predicted in 3D, the subcategories can be pre-divided, and the corresponding 3D prior bounding box information can be determined based on the clustering information of each subcategory.

[0099] In this embodiment of the disclosure, when the category information of the target object is determined, the clustering result corresponding to this category information can be used. Taking a vehicle as the target object as an example, vehicle image samples including various subcategories can be collected in advance. These vehicle image samples contain information such as the length, width, and height of the vehicle. For the vehicle image samples, clustering can be performed based on height values. Thus, vehicle image samples belonging to the same height range can be classified into a corresponding subcategory, thereby determining the clustering information of this subcategory. In practical applications, clustering methods such as K-means can be used to implement the above clustering process, which will not be elaborated upon here.

[0100] The process of determining the 3D prior bounding box information by combining clustering information and the 2D detection box information of the target object may include the following steps:

[0101] Step 1: For each subcategory within each subcategory, determine the depth value corresponding to that subcategory based on the cluster height value included in the clustering information of that subcategory and the width value included in the 2D detection box information of the target object.

[0102] Step 2: Based on the clustering information of the sub-category and the depth value corresponding to the sub-category, determine a 3D prior bounding box for the target object.

[0103] Here, each subcategory can correspond to a 3D prior bounding box. Information such as the size of the 3D prior bounding box can be determined by the clustering information of the corresponding subcategory. The depth information can be determined by the cluster height value and the width value included in the 2D detection box information. In specific applications, the ratio between the cluster height value and the width value can be calculated first, and then the focal length of the camera device can be multiplied.

[0104] Given the 3D prior bounding box information, embodiments of this disclosure can combine this information with the depth map and feature map to determine the 3D detection box information, specifically including the following steps:

[0105] Step 1: Determine the offset of the 3D detection box based on the depth map and feature map;

[0106] Step 2: Determine the 3D detection box information of the target object based on the 3D prior bounding box information and the 3D detection box offset.

[0107] Here, a second object detection network can be used to achieve 3D detection, and the offset of the 3D detection box output by the second object detection network can be obtained. Then, based on the 3D prior box information and the 3D detection box offset, the 3D detection box information of the target object can be determined.

[0108] The aforementioned 3D detection box information mainly includes the shape information (w) of the detection box. 3d h 3d , l 3d ) and depth information (z 3d ).

[0109] It should be noted that, compared to two-dimensional prediction, three-dimensional prediction can determine more dimensions of information about the target object. For example, it can also determine the various subcategories included in the category information of the target object. For instance, it can determine whether a target object belonging to the vehicle category is a car or a truck.

[0110] Considering that there can be multiple 3D prior boxes in this embodiment, and that each 3D prior box can predict a 3D detection box offset, and considering that different 3D prior boxes correspond to different subcategories, and that different subcategories have different prediction probabilities, we can first assign corresponding weights to the 3D prior box information corresponding to each subcategory based on the prediction probability of each subcategory, and then determine the 3D detection box information of the target object based on each 3D prior box information, the weight corresponding to each 3D prior box information, and the 3D detection box offset.

[0111] Here, subcategories with higher prediction probabilities can be assigned higher weights to highlight the role of the corresponding 3D prior boxes in subsequent 3D detection. Similarly, subcategories with lower prediction probabilities can be assigned lower weights to weaken the role of the corresponding 3D prior boxes in subsequent 3D detection, thereby making the determined 3D detection box information more accurate.

[0112] To further improve the accuracy of 3D detection, we can first crop the depth map and feature map, and then perform 3D detection. This can be achieved through the following steps:

[0113] Step 1: Based on the location range included in the 2D detection box information of the target object, extract the depth map and feature map that match the location range from the depth map and feature map respectively;

[0114] Step 2: Determine the offset of the 3D detection box based on the depth map and feature map that match the location range.

[0115] Here, the depth map and feature map corresponding to the location range included in the 2D detection box information can be cropped, that is, a local depth map and local feature map pointing to the target object can be obtained. Based on the local depth map and local feature map, the corresponding 3D detection box offset can be determined, and the 3D detection box offset can also be determined using the second object detection network.

[0116] In the process of predicting the offset of a 3D detection box, the local depth map and local feature map used can avoid the influence of other irrelevant features, resulting in higher prediction accuracy.

[0117] To implement the object detection method provided in this embodiment, it is necessary to train a first object detection network and a second object detection network. For different object detection networks, corresponding supervision signals (i.e., prior box information) can be set, thereby determining the corresponding loss function values. Based on these loss function values, backpropagation can guide network training; no specific limitations are imposed here.

[0118] Considering the crucial role of the depth map corresponding to the projection area of ​​the target object on the ground in the aforementioned target detection process, this embodiment of the disclosure also sets a corresponding supervision signal (i.e., labeled depth map) for the depth map, which can be implemented through a depth map generation network. The training process of this depth map generation network specifically includes the following steps:

[0119] Step 1: Obtain image samples and a depth map determined based on the 3D bounding box information of the target objects in the image samples;

[0120] Step 2: Extract features from the image samples to obtain feature maps of the image samples;

[0121] Step 3: Input the feature map of the image sample into the depth map generation network to be trained to obtain the depth map output by the depth map generation network, and determine the loss function value based on the similarity between the output depth map and the labeled depth map.

[0122] Step 4: If the loss function value is greater than the preset threshold, adjust the network parameter values ​​of the depth map generation network and input the feature map of the image sample into the adjusted depth map generation network until the loss function value is less than or equal to the preset threshold.

[0123] The image samples obtained here are acquired in a similar manner to the images to be detected, and will not be repeated here. Furthermore, the extraction of feature maps from the image samples can be found in the feature map extraction process for the images to be detected described above, and will not be repeated here either.

[0124] This embodiment of the disclosure can determine the loss function value based on the similarity between the depth map output by the depth map generation network and the labeled depth map, and adjust the network parameter values ​​of the depth map generation network according to the loss function value, so that the network input result and the labeled result tend to be consistent or closer.

[0125] The aforementioned labeled depth map can be obtained by following these steps:

[0126] Step 1: Based on the correspondence between the 3D coordinate system where the 3D annotation box is located and the ground coordinate system where the center point of the bottom surface of the annotation box is located, project the 3D annotation box information of the target object onto the ground to obtain the projection area of ​​the target object on the ground and the extended area where the projection area is located.

[0127] Step 2: Determine the depth value of each 3D annotation point on the extended area based on the position coordinates and depth value of the center point of the bottom surface of the 3D annotation frame, which are included in the 3D annotation frame information;

[0128] Step 3: Based on the correspondence between the camera coordinate system and the image coordinate system, project each 3D annotation point on the extended area in the camera coordinate system onto the pixel plane in the image coordinate system to obtain the projection points in the pixel plane.

[0129] Step 4: Based on the depth values ​​of each 3D annotation point on the extended region and the projection points in the pixel plane, obtain the annotation depth map.

[0130] In practical implementation, the depth values ​​of each 3D annotation point on the extended region are determined based on the position coordinates and depth values ​​of the center point of the bottom surface of the 3D annotation frame, including:

[0131] The depth value and position coordinates of the center point of the bottom surface of the annotation box are respectively determined as the depth value and position coordinates of the center point of the extended region;

[0132] Given the location coordinates of the center point of the extended region, the depth value of the center point of the extended region is used as the starting depth value, and the depth value of each three-dimensional annotation point in the extended region is determined at a preset depth interval.

[0133] This disclosure provides a method for generating local ground depth labels. Here, the depth information of the surrounding ground (corresponding to the extended area) can be obtained by utilizing the position of the center point of the bottom surface of the three-dimensional annotation box information of the target object (which falls on the ground).

[0134] Here, the center point of the bottom surface of the annotation box is at the same height as the surrounding ground. A large number of three-dimensional annotation points can be generated in an extended area around the center point. These three-dimensional points include the center point, as well as each three-dimensional annotation point on the extended area with a preset depth interval, starting from the depth value of the center point of the extended area.

[0135] In this way, by using the projection relationship, the three-dimensional annotation points can be projected onto the pixel plane, and the correspondence between the depth value of the three-dimensional annotation points and their projection points can be recorded. By calculating the average depth value of at least one corresponding three-dimensional annotation point for each projection point, the annotation depth map can be obtained.

[0136] The above projection relationship can be realized using the following formula:

[0137]

[0138] Among them, (x 3d y 3d , z 3d (x) represents the camera coordinates of the 3D labeled points. p y p P represents the projection point of the 3D annotation point projection. rect and R rect These represent the rotation correction matrix and the projection matrix, respectively.

[0139] In this way, by inputting the feature map of the image to be detected into the trained depth map generation network, the depth map corresponding to the projection area of ​​the target object in the image onto the ground can be determined. Then, the feature map and 3D prior box information can be combined to achieve 3D prediction of the target object.

[0140] To better understand the process of the three-dimensional prediction described above, we can now combine it with... Figure 2 Please provide a detailed explanation.

[0141] like Figure 2 As shown, for an image to be detected that contains a vehicle as a target object, a feature map of the image can first be extracted using a feature extraction network. Then, on the one hand, a two-dimensional detection is performed using a first object detection network to obtain the two-dimensional detection information of the target object. On the other hand, a depth map generation network is used to generate a depth map corresponding to the projection area of ​​the target object in the image onto the ground.

[0142] In this embodiment of the disclosure, based on the aforementioned two-dimensional detection information, three-dimensional prior bounding box information for the target object can be determined. For example... Figure 2 The image shows an exemplary display of the three 3D prior bounding box information determined by the corresponding three subcategories.

[0143] Here, before inputting the depth map and feature map into the trained second object detection network, cropping can be performed based on the 2D detection information using ROI-align. Then, the cropped depth map and feature map are input into the second object detection network to obtain the corresponding 3D detection box offset, such as... Figure 2 The Δ(w,h,l) shown 3d ,Δz 3d Information such as...

[0144] By combining the aforementioned 3D detection bounding box offset and 3D prior bounding box information, the 3D detection information can be determined. In practical applications, this 3D detection information can be displayed on the image to be detected.

[0145] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0146] Based on the same inventive concept, this disclosure also provides a target detection apparatus corresponding to the target detection method. Since the principle of the apparatus in this disclosure for solving the problem is similar to the target detection method described above in this disclosure, the implementation of the apparatus can refer to the implementation of the method, and the repeated parts will not be described again.

[0147] Reference Figure 3The diagram shown is a schematic representation of a target detection device according to an embodiment of this disclosure. The device includes: an extraction module 301, a generation module 302, and a first detection module 303; wherein,

[0148] The extraction module 301 is used to extract features from the image to be detected to obtain a feature map of the image to be detected;

[0149] The generation module 302 is used to generate a depth map based on the feature map, which is the projection area of ​​the target object in the image to be detected onto the ground.

[0150] The first detection module 303 is used to determine the three-dimensional detection information of the target object based on the depth map and the feature map.

[0151] The aforementioned target detection device can not only extract features from the image to be detected, but also generate a depth map corresponding to the projection area of ​​the target object in the image onto the ground based on the extracted feature map. Then, based on the depth map and the feature map, the 3D detection information of the target object can be determined. Since the generated depth map points to the target object in the image to be detected and corresponds to the projection area of ​​the target object onto the ground, this projection area is associated with the target object to a certain extent. Therefore, when performing 3D detection using the feature map of the target object on a local ground surface, the depth map corresponding to that surface can be used as a guide, improving the detection accuracy.

[0152] In one possible implementation, the above-described apparatus further includes:

[0153] The second detection module 304 is used to perform two-dimensional detection on the feature map of the image to be detected after obtaining the feature map, so as to obtain two-dimensional detection information for the target object.

[0154] The first detection module 303 is used to determine the three-dimensional detection information of the target object based on the depth map and the feature map according to the following steps:

[0155] Based on the two-dimensional detection information, determine the three-dimensional prior bounding box information for the target object;

[0156] Based on 3D prior bounding box information, depth map, and feature map, the 3D detection bounding box information of the target object is determined.

[0157] In one possible implementation, the two-dimensional detection information includes the two-dimensional detection bounding box information of the target object and the category information of the target object; the first detection module 303 is used to determine the three-dimensional prior bounding box information for the target object based on the two-dimensional detection information according to the following steps:

[0158] Based on the category information of the target object, determine the clustering information of each subcategory included in the category to which the target object belongs;

[0159] Based on the clustering information of each subcategory and the two-dimensional detection box information of the target object, the three-dimensional prior box information of the target object is determined.

[0160] In one possible implementation, the first detection module 303 is configured to determine the three-dimensional prior bounding box information for the target object based on the clustering information of each sub-category and the two-dimensional detection box information of the target object, according to the following steps:

[0161] For each subcategory, the depth value corresponding to that subcategory is determined based on the cluster height value included in the clustering information of that subcategory and the width value included in the two-dimensional detection box information of the target object.

[0162] Based on the clustering information of the sub-category and the depth value corresponding to the sub-category, a three-dimensional prior bounding box is determined for the target object.

[0163] In one possible implementation, the first detection module 303 is configured to determine the 3D detection bounding box information of the target object based on the 3D prior bounding box information, the depth map, and the feature map according to the following steps:

[0164] The offset of the 3D detection box is determined based on the depth map and feature map;

[0165] Based on the 3D prior bounding box information and the 3D detection box offset, the 3D detection box information of the target object is determined.

[0166] In one possible implementation, the first detection module 303 is configured to determine the offset of the three-dimensional detection box based on the depth map and the feature map according to the following steps:

[0167] Based on the location range included in the two-dimensional detection box information of the target object, the depth map and feature map that match the location range are extracted from the depth map and feature map respectively;

[0168] The offset of the 3D detection box is determined based on the depth map and feature map that match the location range.

[0169] In one possible implementation, there are multiple 3D prior bounding box information; the first detection module 303 is used to determine the 3D detection box information of the target object based on the 3D prior bounding box information and the 3D detection box offset according to the following steps:

[0170] Determine the weights corresponding to each 3D prior bounding box;

[0171] Based on the information of each 3D prior box, the weight corresponding to each 3D prior box, and the offset of the 3D detection box, the 3D detection box information of the target object is determined.

[0172] In one possible implementation, the first detection module 303 is configured to determine the weight corresponding to each 3D prior box information according to the following steps:

[0173] The predicted probabilities of each subcategory included in the category information of the target object are determined based on the depth map and feature map.

[0174] Based on the predicted probabilities of each sub-category, the weights of the 3D prior bounding box information corresponding to each sub-category are determined.

[0175] In one possible implementation, the second detection module 304 is used to detect the feature map according to the following steps to obtain two-dimensional detection information for the target object:

[0176] Determine the offset of the two-dimensional detection box based on the feature map;

[0177] Based on the preset two-dimensional prior bounding box information and the two-dimensional detection box offset, the two-dimensional detection information of the target object is determined.

[0178] In one possible implementation, the depth map is determined by a trained depth map generation network; the depth map generation network is trained from image samples and labeled depth maps determined based on the 3D bounding box information of the target objects labeled in the image samples.

[0179] In one possible implementation, the 3D annotation box information of the target object includes the position coordinates and depth value of the center point of the bottom surface of the annotation box; the generation module 302 is used to obtain the annotation depth map according to the following steps:

[0180] Based on the correspondence between the three-dimensional coordinate system where the three-dimensional annotation box is located and the ground coordinate system where the center point of the bottom surface of the annotation box is located, the three-dimensional annotation box information of the target object is projected onto the ground to obtain the projection area of ​​the target object on the ground and the extended area where the projection area is located.

[0181] The depth value of each 3D annotation point on the extended area is determined based on the position coordinates and depth value of the center point of the bottom surface of the 3D annotation frame, which are included in the 3D annotation frame information.

[0182] Based on the correspondence between the camera coordinate system and the image coordinate system, each three-dimensional annotation point on the extended area in the camera coordinate system is projected onto the pixel plane in the image coordinate system to obtain the projection points in the pixel plane.

[0183] An annotation depth map is obtained based on the depth values ​​of each 3D annotation point on the extended region and the projection points in the pixel plane.

[0184] In one possible implementation, the generation module 302 is configured to determine the depth value of each 3D annotation point on the extended region based on the position coordinates and depth value of the center point of the bottom surface of the annotation frame, which is included in the 3D annotation frame information, according to the following steps:

[0185] The depth value and position coordinates of the center point of the bottom surface of the annotation box are respectively determined as the depth value and position coordinates of the center point of the extended area;

[0186] Given the location coordinates of the center point of the extended region, the depth value of the center point of the extended region is used as the starting depth value, and the depth value of each 3D annotation point in the extended region is determined by a preset depth interval.

[0187] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0188] This disclosure also provides an electronic device, such as... Figure 4 The diagram shown is a schematic representation of an electronic device structure provided in this embodiment of the present disclosure, including: a processor 401, a memory 402, and a bus 403. The memory 402 stores machine-readable instructions executable by the processor 401 (e.g., ...). Figure 3 The device extracts the execution instructions corresponding to module 301, generation module 302, and first detection module 303. When the electronic device is running, processor 401 and memory 402 communicate via bus 403. When machine-readable instructions are executed by processor 401, the following processing is performed:

[0189] Feature extraction is performed on the image to be detected to obtain the feature map of the image to be detected;

[0190] Based on the feature map, a depth map is generated corresponding to the projection area of ​​the target object in the image to be detected onto the ground.

[0191] Based on depth maps and feature maps, the 3D detection information of the target object is determined.

[0192] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the target detection method described in the above-described method embodiments. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0193] This disclosure also provides a computer program product carrying program code. The program code includes instructions that can be used to execute the steps of the target detection method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0194] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0195] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0196] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0197] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0198] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0199] Finally, it should be noted that the above-described embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this disclosure. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A method for target detection, characterized in that, The method includes: Feature extraction is performed on the image to be detected to obtain the feature map of the image to be detected; Based on the feature map, a depth map is generated corresponding to the projection area of ​​the target object in the image to be detected onto the ground. Based on the depth map and the feature map, the three-dimensional detection information of the target object is determined; The depth map is determined by a trained depth map generation network; the depth map generation network is trained from image samples and labeled depth maps determined based on the 3D bounding box information of the target objects labeled in the image samples. The 3D annotation frame information of the target object includes the position coordinates and depth value of the center point of the bottom surface of the annotation frame; the annotation depth map is obtained according to the following steps: Based on the correspondence between the three-dimensional coordinate system where the three-dimensional annotation box is located and the ground coordinate system where the center point of the bottom surface of the annotation box is located, the three-dimensional annotation box information of the target object is projected onto the ground to obtain the projection area of ​​the target object on the ground and the extended area where the projection area is located. The depth value of each three-dimensional annotation point on the extended area is determined based on the position coordinates and depth value of the center point of the bottom surface of the annotation frame, which are included in the three-dimensional annotation frame information. Based on the correspondence between the camera coordinate system and the image coordinate system, each three-dimensional annotation point on the extended region in the camera coordinate system is projected onto the pixel plane in the image coordinate system to obtain the projection points in the pixel plane. The annotation depth map is obtained based on the depth values ​​of each three-dimensional annotation point on the extended region and the projection points in the pixel plane.

2. The method according to claim 1, characterized in that, After obtaining the feature map of the image to be detected, the method further includes: The feature map is then inspected to obtain two-dimensional detection information for the target object. The step of determining the 3D detection information of the target object based on the depth map and the feature map includes: Based on the two-dimensional detection information, determine the three-dimensional prior bounding box information for the target object; Based on the 3D prior bounding box information, the depth map, and the feature map, the 3D detection bounding box information of the target object is determined.

3. The method according to claim 2, characterized in that, The two-dimensional detection information includes the two-dimensional detection bounding box information of the target object and the category information of the target object; the step of determining the three-dimensional prior bounding box information for the target object based on the two-dimensional detection information includes: Based on the category information of the target object, determine the clustering information of each subcategory included in the category to which the target object belongs; Based on the clustering information of each sub-category and the two-dimensional detection box information of the target object, the three-dimensional prior box information for the target object is determined.

4. The method according to claim 3, characterized in that, The step of determining the 3D prior bounding box information for the target object based on the clustering information of each sub-category and the 2D detection box information of the target object includes: For each of the subcategories, a depth value corresponding to that subcategory is determined based on the clustering height value included in the clustering information of that subcategory and the width value included in the two-dimensional detection box information of the target object. Based on the clustering information of the sub-category and the depth value corresponding to the sub-category, a three-dimensional prior bounding box is determined for the target object.

5. The method according to claim 3, characterized in that, The step of determining the 3D detection bounding box information of the target object based on the 3D prior bounding box information, the depth map, and the feature map includes: The offset of the 3D detection box is determined based on the depth map and the feature map; Based on the 3D prior bounding box information and the 3D detection box offset, the 3D detection box information of the target object is determined.

6. The method according to claim 5, characterized in that, Determining the 3D detection box offset based on the depth map and the feature map includes: Based on the location range included in the two-dimensional detection box information of the target object, a depth map and a feature map matching the location range are extracted from the depth map and the feature map, respectively. The offset of the 3D detection box is determined based on the depth map and feature map that match the location range.

7. The method according to claim 5, characterized in that, The 3D prior bounding box information is multiple; determining the 3D detection box information of the target object based on the 3D prior bounding box information and the 3D detection box offset includes: Determine the weights corresponding to each of the three-dimensional prior box information; Based on each of the three-dimensional prior bounding box information, the weight corresponding to each of the three-dimensional prior bounding box information, and the offset of the three-dimensional detection box, the three-dimensional detection box information of the target object is determined.

8. The method according to claim 7, characterized in that, The method further includes: The predicted probabilities of each sub-category included in the category information of the target object are determined based on the depth map and the feature map. The determination of the weights corresponding to each of the three-dimensional prior bounding boxes includes: Based on the predicted probabilities of each sub-category, the weights of the 3D prior bounding box information corresponding to each sub-category are determined.

9. The method according to any one of claims 2-8, characterized in that, The step of detecting the feature map to obtain two-dimensional detection information for the target object includes: The offset of the two-dimensional detection box is determined based on the feature map; Based on the preset two-dimensional prior bounding box information and the offset of the two-dimensional detection box, the two-dimensional detection information of the target object is determined.

10. The method according to claim 1, characterized in that, The step of determining the depth value of each 3D annotation point on the extended region based on the position coordinates and depth value of the center point of the bottom surface of the annotation frame, including the 3D annotation frame information, includes: The depth value and position coordinates of the center point of the bottom surface of the annotation box are respectively determined as the depth value and position coordinates of the center point of the extended region; Given the location coordinates of the center point of the extended region, the depth value of the center point of the extended region is used as the starting depth value, and the depth value of each three-dimensional annotation point in the extended region is determined at a preset depth interval.

11. A target detection device, characterized in that, The device comprises: The extraction module is used to extract features from the image to be detected, thereby obtaining a feature map of the image to be detected. The generation module is used to generate a depth map corresponding to the projection area of ​​the target object in the image to be detected onto the ground, based on the feature map. The first detection module is used to determine the three-dimensional detection information of the target object based on the depth map and the feature map; The depth map is determined by a trained depth map generation network; the depth map generation network is trained from image samples and labeled depth maps determined based on the 3D bounding box information of the target objects labeled in the image samples. The 3D annotation box information of the target object includes the position coordinates and depth value of the center point of the bottom surface of the annotation box; The generation module is used to project the 3D annotation box information of the target object onto the ground based on the correspondence between the 3D coordinate system where the 3D annotation box is located and the ground coordinate system where the center point of the bottom surface of the annotation box is located, to obtain the projection area of ​​the target object on the ground and the extended area where the projection area is located; determine the depth value of each 3D annotation point on the extended area based on the position coordinates and depth value of the center point of the bottom surface of the annotation box included in the 3D annotation box information; project each 3D annotation point on the extended area in the camera coordinate system onto the pixel plane in the image coordinate system based on the correspondence between the camera coordinate system and the image coordinate system, to obtain the projection point in the pixel plane; and obtain the annotation depth map based on the depth value of each 3D annotation point on the extended area and the projection point in the pixel plane.

12. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is in operation, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the target detection method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the target detection method as described in any one of claims 1 to 10.

Images