METHOD, DEVICE, CONTROL UNIT, VEHICLE AND MEDIUM FOR THREE-DIMENSIONAL (3D) OBJECT DETECTION

Abstract

The embodiments of the present disclosure relate to a method, an apparatus, a controller, a vehicle, and a medium for three-dimensional (3D) object detection. The method comprises capturing a two-dimensional (2D) image of a 3D target scene. The method further comprises determining depth data corresponding to the 2D image. The method further comprises detecting a 3D object in the 3D target scene using a predetermined neural network model based on the 2D image and the depth data. In the method according to the embodiments of the present disclosure, the 3D object detection can be performed using the predetermined neural network model according to the embodiments of the present disclosure based solely on the 2D image and the depth data, thereby increasing the efficiency and accuracy of the 3D object detection.

Description

FIELD OF THE INVENTION

[0001] The embodiments of the present disclosure relate to the field of data processing and, more particularly, to a method, apparatus, controller, vehicle, and medium for three-dimensional (3D) object sensing. STATE OF THE ART

[0002] With the development of intelligent devices, the need for object detection in scenes is increasing, and more and more applications require the detection of objects in three-dimensional (3D) scenes (such as streets, office areas, or amusement parks). For example, during autonomous or assisted driving of vehicles, objects (such as other vehicles or pedestrians) in 3D driving scenes must be detected to determine the driving maneuvers to be performed. In such applications, the efficiency and accuracy of object detection can directly impact user safety, making the efficiency and accuracy of object detection crucial. DISCLOSURE OF THE INVENTION

[0003] Embodiments of the present disclosure provide a method, apparatus, controller, vehicle, and medium for three-dimensional (3D) object sensing.

[0004] According to a first aspect of the present disclosure, a method for three-dimensional 3D object detection is provided. The method comprises capturing a two-dimensional (2D) image of a 3D target scene. The method further comprises determining depth data corresponding to the 2D image. The method further comprises detecting a 3D object in the 3D target scene using a predetermined neural network model based on the 2D image and the depth data.

[0005] According to a second aspect of the present disclosure, a device for 3D object detection is provided. The device comprises a detection unit configured to detect a two-dimensional 2D image of a 3D target scene. The device further comprises a generation unit configured to determine depth data corresponding to the 2D image. The device further comprises a recognition unit configured to detect a 3D object in the 3D target scene using a predetermined neural network model based on the 2D image and the depth data.

[0006] According to a third aspect of the present disclosure, a control device is provided. The control device comprises at least one processor; and a memory coupled to the at least one processor and containing instructions stored thereon, wherein the instructions, when executed by the at least one processor, cause the control device to implement the method according to the first aspect of the present disclosure.

[0007] In a fourth aspect of the present disclosure, a vehicle is provided. The vehicle includes a control device according to the third aspect of the present disclosure and a predetermined neural network model according to the first aspect of the present disclosure.

[0008] In a fifth aspect of the present disclosure, a computer-readable storage medium is provided. Computer-executable instructions are stored on this computer-readable storage medium, wherein the computer-executable instructions are executed by a processor to implement the method according to the first aspect of the present disclosure. DESCRIPTION OF THE CHARACTERS

[0009] The above and other objects, features, and advantages of the present disclosure will become more apparent by describing the exemplary embodiments of the present disclosure in more detail in conjunction with the figures, wherein the same reference numerals generally refer to the same components throughout the exemplary embodiments. Fig. 1 shows a schematic diagram of an exemplary environment in which the apparatus and / or method according to embodiments of the present disclosure may be implemented. Fig. 2 shows a flowchart of a method for three-dimensional (3D) object detection according to embodiments of the present disclosure. Fig. 3 shows a schematic block diagram of a predetermined neural network model according to embodiments of the present disclosure. Fig. 4A shows a schematic diagram of an example of the process for creating a first neural network model according to embodiments of the present disclosure. Fig. 4B shows a schematic diagram of an example of the process for obtaining model anchors of the first neural network model according to embodiments of the present disclosure. Fig. 5 shows a schematic diagram of an example of the process for detecting a 3D object in a 3D target scene according to embodiments of the present disclosure. Fig. 6 shows a schematic diagram of an example of the process for obtaining a first model detection result by the first neural network model according to embodiments of the present disclosure. Fig. 7 shows a schematic diagram of an example of the process for determining a 3D object by a second neural network model according to embodiments of the present disclosure. Fig. 8 shows a schematic block diagram of an apparatus for 3D object detection according to embodiments of the present disclosure. Fig. 9 shows a schematic block diagram of an example of a device suitable for implementing embodiments of the content of the present disclosure.

[0010] In the figures, the same or corresponding reference numerals designate the same or corresponding parts. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0011] In the following, the embodiments of the present disclosure are described in more detail with reference to the figures. While some embodiments of the present disclosure are illustrated in the figures, it is to be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a thorough and complete understanding of the present disclosure. It should be understood that the figures and embodiments of the present disclosure are provided as examples only and are not intended to limit the scope of the disclosure.

[0012] In describing the embodiments of the present disclosure, the term "comprising" and similar terms should be understood as openly encompassing, that is, "including, but not limited to." The term "based on" should be understood as "based at least in part on." The terms "an embodiment" or "this embodiment" should be understood as "at least one embodiment." The terms "first," "second," etc., may refer to different or the same objects. The following may contain additional explicit and implicit definitions.

[0013] With the development of intelligent devices, the need for object detection in scenes is increasing. For example, in autonomous or assisted driving, the vehicle's perception module is crucial for the safety of the vehicle and its users. It acts as the vehicle's "eyes," perceiving the environment around the vehicle (e.g., detecting 3D objects in three-dimensional (3D) road traffic scenes) to determine the driving maneuvers to be performed. For example, the perception module can detect objects on the road (such as other vehicles, traffic lights, pedestrians, sidewalks, lane markings, buildings, and obstacles) based on the point clouds captured by LiDAR or the images captured by cameras.

[0014] LiDAR contributes to precise object detection due to its physical properties. However, LiDAR is susceptible to weather conditions such as rain or snow, making it unsuitable for diverse driving environments. Furthermore, the cost of LiDAR is relatively high, making it difficult to integrate into vehicles where cost savings are required. Therefore, the question of how to perform 3D object detection based on two-dimensional (2D) images captured by a monocular camera of a 3D target scene (for example, a street scene) is receiving increasing attention.

[0015] 3D object detection based on 2D images can typically be performed using neural network models. The neural network models used for such 3D object detection can be classified into anchor-free and anchor-based types. For example, SMOKE (Single-Stage Monocular 3D Object Detection via Keypoint Estimation) belongs to the anchor-free type of neural network models. For example, M3D-RPN (Monocular 3D Region Proposal Network) belongs to the anchor-based type of neural network models. Anchors of the anchor-based type of neural network models can generally be divided into 2D anchors and 3D anchors. For neural network models with 2D anchors, the inputs include only 2D information, such as 2D images.For neural network models with 3D anchors, the inputs must include not only 2D information but also 3D information about the objects, such as their position, size, and orientation information in the camera coordinate system.

[0016] However, while the above anchor-free neural network models are easy to develop and train because they do not require many preset parameters, the limitations of these neural network models result in lower detection accuracy and make it difficult to comprehensively detect objects in 3D scenes (i.e., the recall rate is low). This could pose safety risks in, for example, autonomous or assisted driving of vehicles, such as a vehicle colliding with an obstacle on the roadway due to the failure to detect it. Furthermore, this 3D object detection reconstructs the 3D information of the objects from the 2D images. This reconstruction itself leads to relatively large errors.

[0017] Furthermore, the above neural network models with 2D anchors reconstruct the 3D information of objects based only on 2D images and, in particular, cannot accurately predict the depths of neighboring objects. Furthermore, the above neural network models with 3D anchors require extensive 3D information (e.g., position, size, and orientation information in the camera coordinate system), resulting in high computational overhead. Therefore, the neural network models require a large amount of computational resources and are not suitable for vehicles with limited computational resources, nor can they perform efficient 3D object detection.

[0018] To solve at least the above and other potential problems, embodiments of the present disclosure provide a method for 3D object detection. The method includes acquiring a 2D image of a 3D target scene. The method further includes determining depth data corresponding to the 2D image. The method further includes detecting a 3D object in the 3D target scene using a predetermined neural network model based on the 2D image and the depth data. In the method according to embodiments of the present disclosure, 3D object detection can be performed using the predetermined neural network model according to embodiments of the present disclosure based solely on the 2D image and the depth data, thereby increasing the efficiency and accuracy of 3D object detection.

[0019] In the following, the embodiments of the present disclosure are described in detail in conjunction with the figures. Fig. 1 shows a schematic diagram of an exemplary environment 100 in which the apparatus and / or method according to embodiments of the present disclosure may be implemented. As in Fig. 1, vehicles 120 and 130 are present in the environment 100, traveling on a road 110. In some embodiments, the vehicles 120 and 130 may be vehicles capable of autonomous or assisted driving. In the following, the vehicle 120 is taken as an example to explain this disclosure. In some embodiments, the vehicle 120 may include a camera 121 (e.g., a mono camera), where the camera 121 has, for example, a large field of view F1 (e.g., 120°) to capture a comprehensive 2D image of a 3D target scene (i.e., the scene of the road 110).

[0020] In some embodiments, the vehicle 120 may also include a controller 122, wherein the controller 122 may be configured to communicate with the camera 121 to receive the 2D image captured by the camera 121. Furthermore, in some embodiments, the vehicle 120 may include a unit (e.g., radar) configured to acquire depth data corresponding to the 2D image captured by the camera 121. The controller 122 may also communicate with the unit to receive the depth data from the unit. Furthermore, the vehicle 120 may include a predetermined neural network model, described in more detail in the following examples, included in the controller 122 or coupled to the controller 122, such that the controller 122 may perform 3D object detection using the predetermined neural network model based on the 2D image and the depth data.For example, the controller 122 may detect another vehicle 130, a traffic light 140, a pedestrian 150, a sidewalk 160, and a lane marking 170 in the scene shown in FIG. Fig. 1. It is understood that these detected objects are only examples and, depending on the 3D target scene, any other objects such as other obstacles or buildings can be detected. In the following, in conjunction with the Fig. Examples of 3D object detection are described in the embodiments shown in Figures 2 to 7.

[0021] Fig. 2 shows a flowchart of a method 200 for 3D object detection according to embodiments of the present disclosure. The method 200 may be executed by any control device (e.g., the one described in Fig. 1) or an electronic device or server using a predetermined neural network model according to the embodiments of the present disclosure. As shown in Fig. 2, a 2D image of a 3D target scene is acquired in block 202. The 3D target scene may, for example, be the Fig. 1, and the captured 2D image may be a scene of the road 110 taken by the camera 121 on the vehicle 120 in Fig. 1 captured 2D image of the scene of road 110. It should be understood that the 3D target scene can also be any scene, such as an office area or an amusement park. In block 204, depth data corresponding to the 2D image is determined. In some embodiments, depth data corresponding to a 2D object can be determined via a radar on the vehicle. For example, the radar can be any type of radar.

[0022] In block 206, a 3D object is detected in the 3D target scene using a predetermined neural network model based on the 2D image and depth data. For example, the other vehicle 130, the traffic light 140, the pedestrian 150, the sidewalk 160, and the pavement marking 170, as shown in Fig. 1, or other obstacles or buildings may also be detected. In the method according to the embodiments of the present disclosure, the 3D object detection can be performed using the predetermined neural network model according to the embodiments of the present disclosure based solely on the 2D image and the depth data, thereby increasing the efficiency and accuracy of the 3D object detection. In the following, in connection with Fig. 3 and Fig. 4 describes a predetermined neural network model according to embodiments of the present disclosure.

[0023] Fig. 3 shows a schematic block diagram of a predefined neural network model 300 according to embodiments of the present disclosure. As in Fig. As shown in Figure 3, the predetermined neural network model 300 may include a first neural network model 302 and a second neural network model 304. In some embodiments, the first neural network model 302 may be created based on a neural network model for 2D object detection. In some embodiments, the second neural network model 304 may be a head neural network model for the 3D object detection task.

[0024] In some embodiments, the 2D object detection neural network model provided for creating the first neural network model 302 may be a 2D anchor neural network model for single-stage 2D object detection (e.g., YOLOv5 or any other neural network model). Fig. 4A shows a schematic diagram of an example of the process 400 for creating a first neural network model 302 according to embodiments of the present disclosure. As in Fig. 4A, predetermined depth data may be obtained in block 402. For example, this predetermined depth data may be training depth data from training data intended for training the first neural network model 302 to be created, or it may be other depth data that is different from the training depth data. In block 404, the predetermined depth data may be clustered to group it into multiple clusters. For example, a K-Means algorithm may be used to cluster the predetermined depth data.

[0025] In block 406, the 2D anchors may be placed in each of the plurality of clusters to obtain a model anchor for each of the clusters and thus create the first neural network model 302. In some embodiments, the above setting of the 2D anchors for each of the clusters may be performed such that the 2D anchors are linked to the mean of the predetermined depth data in the cluster to obtain the model anchor for the cluster. In some embodiments, the number of 2D anchors of the neural network model for 2D object detection provided for creating the first neural network model 302 may be a first number. In some embodiments, the number of clusters into which the predetermined depth data is grouped may be a second number. In some embodiments, the number of all model anchors in the created first neural network model 302 may be the product of the first number and the second number. This may be determined by reference to the Fig. 4B, the process for obtaining model anchors can be better understood.

[0026] Fig. 4B shows a schematic diagram of an example of the process 406A for obtaining model anchors of the first neural network model 302 according to embodiments of the present disclosure. Fig. Process 406A shown in Figure 4B corresponds to block 406 in Fig. 4A. The Fig. 4B may be 2D anchors of the neural network model for 2D object detection provided for creating the first neural network model 302. The 2D anchors 410 may include M 2D anchors 410-1 to 410-M, where M is an integer greater than or equal to 1. The number of clusters into which the depth data is divided in block 406 in Fig. 4A can be N, where N is an integer greater than or equal to 1. The mean values of the given depth data in the N clusters can be D1, D2, ..., DN, respectively, as in Fig. 4B shown.

[0027] As in Fig. 4B, the 2D anchors 410 are respectively set in each cluster and concatenated with the mean D1, D2, ..., DN of each cluster to obtain a corresponding model anchor 420-1, 420-2, ..., 420-N. By generating the model anchors 420-1, 420-2, ..., 420-N as described above, a first neural network model 302 may be created according to embodiments of the present disclosure. Thus, the first neural network model 302 may detect a 2D object in a 2D image and also predict a predicted depth for the detected 2D object. In some embodiments, the first neural network model 302 may be trained to perform the desired 2D object detection as well as depth prediction.

[0028] In some embodiments, the first neural network model 302 may be trained using training depth data (which may, for example, be the predetermined depth data above), corresponding training 2D images, and ground truth data. In some embodiments, the first neural network model 302 may receive training 2D images during the training process and extract features from these training 2D images to obtain a feature map. The feature map is then divided into multiple sub-images (e.g., into multiple grids). Subsequently, each of the model anchors 420-1, 420-2, ..., 420-N of the first neural network model 302 may output a detected 2D object for that sub-image (e.g., the 2D bounding box of a detected 2D object) and a corresponding predicted depth based on each sub-image and the corresponding training depth data from the training depth data.Subsequently, based on the intersection-over-union (IoU) of the 2D bounding box output by each model anchor with the corresponding bounding box of the ground truth data, as well as the difference between the predicted depth output by each model anchor and the corresponding depth of the ground truth data, the output of a model anchor that has an output closest to the ground truth data is selected as the final output of the first neural network model for that sub-image. In some embodiments, particularly with regard to training depth prediction, based on the input depth data, depth prediction can be performed using the linear prediction method shown in the following equation (1). Z=pred[0]∗depth_center_x_std+depth_center_x_mean

[0029] In equation (1), Z represents the predicted depth value, pred[0] represents the normalized depth value output by the first neural network model, depth_center_x_std represents the standard deviation of the training depth data for the x-th cluster, where x is an integer greater than or equal to 1 and less than or equal to N, and depth_center_x_mean represents the average of the training depth data for the x-th cluster. In some embodiments, depth prediction training may be terminated when the change in the depth predicted by the first neural network compared to the ground truth data is less than a predetermined change threshold, or when the difference between the predicted depth and the ground truth data is less than a predetermined difference threshold.It is understood that training can also be terminated based on other conditions depending on actual requirements, such as when the number of training runs reaches a certain threshold. It is understood that the training end conditions for 2D object detection training may be similar, or that depth prediction training and 2D object detection training may have related training end conditions.

[0030] In addition, in some embodiments, the Fig. 3. Training data of the second neural network model 304 may include, for example, training data corresponding to the output of the first neural network model 302, anchor parameters of the model anchors of the first neural network model 302 (e.g., mean values), and camera parameters (which serve as input for the second neural network model 304). This training data may also include the corresponding 3D object ground truth data. The training process of the second neural network model 304 is similar to that of the first neural network model 302 and will not be explained further. In the following, the process of 3D object detection by the trained first neural network model 302 according to the embodiments of the present disclosure and the trained second neural network model 304 will be described using the examples of Fig. 5 to 7 in conjunction with the Fig. 2 according to embodiments of the present disclosure.

[0031] Fig. 5 shows a schematic diagram of an example of the process 500 for detecting a 3D object in the 3D target scene according to embodiments of the present disclosure. Fig. Process 500 shown in Figure 5 corresponds to block 206 in Fig. 2. As in Fig. 5, in block 502, a first model detection result may be obtained using the first neural network model 302 based on the 2D image obtained in block 202 of the Fig. 2 and the depth data obtained in block 204 of the Fig. 2. In some embodiments, the first model detection result may include a detected 2D object for the 2D image (e.g., the bounding box of a detected 2D object) and a predicted depth for the detected 2D object. In block 504, a 3D object corresponding to the detected 2D object may be determined using the second neural network model 304 based on the first model detection result and corresponding parameters. In some embodiments, the corresponding parameters may include anchor parameters of the model anchors in the first neural network model. In some embodiments, the corresponding parameters may include camera parameters of the camera (e.g., the camera in Fig. 1 shown 121) which captures the 2D image. In the following, blocks 502 and 504 are described in conjunction with the Fig. 6 and Fig. 7 examples shown.

[0032] Fig. 6 shows a schematic diagram of an example of the process 600 for obtaining a first model detection result by the first neural network model 302 according to embodiments of the present disclosure. Fig. 6 corresponds to block 502 in Fig. 5. As in Fig. 6, in block 602, features are extracted from the 2D image using the first neural network model 302 to obtain a feature map 610 of the 2D image. In block 604, the feature map 610 may be divided into multiple sub-images. As shown in Fig. 6, the feature map 610 is divided into a plurality of sub-images having k rows and h columns, where k and h are integers greater than or equal to 1. In block 606, a corresponding anchor detection result may be calculated for each sub-image of the plurality of sub-images (for illustration, the one shown in Fig. 6 is used as an example, where i is an integer greater than or equal to 1 and less than or equal to k × h) are obtained using all model anchors 420-1, 420-2, ..., 420-N, respectively.

[0033] In some embodiments, the corresponding anchor detection result may include a corresponding detected 2D object for the sub-image 610-i. In some embodiments, the corresponding anchor detection result may also include a corresponding predicted depth for the corresponding detected 2D object. In some embodiments, for each model anchor of the all model anchors 420-1, 420-2, ..., 420-N of the first neural network model 302, the corresponding predicted depth is determined based on the average and standard deviation of the corresponding depth data from the depth data corresponding to the sub-image 610-i.For example, the predicted depth can be obtained in a similar way to equation (1) above, with one difference that depth_centerx_std in equation (1) here represents the standard deviation of the above depth data for the x-th cluster, and depth_center_x_mean here represents the average of the above depth data for the x-th cluster.

[0034] In block 608, a target anchor detection result is selected from all anchor detection results based on the non-maximum suppression (NMS) algorithm as the sub-image detection result for this sub-image 610-i. This allows the output of the first neural network model 302 to be obtained, namely the above first model detection result. In some embodiments, the above first model detection result includes the sub-image detection results for each of the sub-images. The first model detection result can be used by the second neural network model 304 to determine a 3D object.

[0035] Fig. 7 shows a schematic diagram of an example of the process 700 for determining a 3D object by the second neural network model 304 according to embodiments of the present disclosure. Fig. Process 700 shown in Figure 7 corresponds to block 504 in Fig. 5. In some embodiments, the second neural network model 304 may be an eight-channel 3D object detection head neural network model that implements the Fig. 7. As shown in Fig. 7, in block 702, the second neural network model 304 may determine, for each sub-image 610-i of the plurality of sub-images (k × h sub-images), the corresponding predicted depth included in the sub-image detection result of the sub-image 610-i as the predicted depth of the corresponding 3D object corresponding to the corresponding detected 2D object included in the sub-image detection result. In some embodiments, the first channel of the second neural network model 304 is configured to determine the predicted depth of the corresponding 3D object.

[0036] In block 704, the orientation of the corresponding 3D object in the top view can be determined based on the predicted depth and the center of the model anchor provided for obtaining the partial image acquisition result (for clarity, the Fig. 4 is used as an example; it should be understood that such a model anchor 410-2 may also be any other model anchor). In some embodiments, the second channel and the third channel of the second neural network model 304 are configured to determine the orientations of the corresponding 3D object in the top view. In some embodiments, the second neural network model 304 may determine the orientation of the corresponding 3D object in the top view using the following equations (2), (3), and (4). θ=αz+arctan(X / Z') sin(αz)=pred[1] cos(αz)=pred[2]

[0037] In the above equations (2) to (4), X represents the horizontal coordinate of the center of the above-defined model anchor 420-2 in the camera coordinate system in the top view, Z' represents the vertical coordinate of the center of the model anchor 420-2 in the camera coordinate system in the top view. pred[1] represents the output of the second channel of the second neural network model 304, pred[2] represents the output of the third channel of the second neural network model 304, ϑ represents the monitoring signal of pred[1] and pred[2]. α z represents the predicted orientation of the corresponding 3D object in plan view.

[0038] In block 706, the predicted length, width, and height of the corresponding 3D object may be determined based on a preset average length, width, and height corresponding to the type of the above-detected 2D object. In some embodiments, the fourth channel through the sixth channel of the second neural network model 304 may be provided to determine the predicted length, width, and height of the corresponding 3D object. In some embodiments, the second neural network model 304 may determine the predicted length, width, and height of the corresponding 3D object through the following equations (5), (6), and (7). dim_h=h¯∗exp(pred[3]) dim_w=w¯∗exp(pred[4]) dim_l=l¯∗exp(pred[5])

[0039] In the above equations (5) to (7), l̅, h̅, and w̅ respectively represent the preset average length, width, and height corresponding to the type of the above detected 2D object. For example, l̅, h̅, and w̅ can be adjusted according to actual needs. pred[3] represents the output of the fourth channel of the second neural network model 304, pred[4] represents the output of the fifth channel of the second neural network model 304, and pred[5] represents the output of the second neural network model 304. dim_l, dim_w, and dim_h respectively represent the predicted length, width, and height of the corresponding 3D object.

[0040] In block 708, the position of the corresponding 3D object in 3D space may be determined based on the above predicted depth, length, width, height, and camera parameters (e.g., the camera parameters of camera 121). In some embodiments, the seventh and eighth channels of the second neural network model 304 may be provided to determine the position of the corresponding 3D object in 3D space. In some embodiments, the second neural network model 304 may determine the position of the corresponding 3D object in 3D space using equations (8), (9), and (10). proj_center_u=stride∗(grid_u+pred[6]) proj_center_v=stride∗(grid_v+pred[7]) Z[proj_center_uproj_center_v1]=K3×3[XYZ']

[0041] In the above equations (8) to (10), grid_u and grid_v represent the centers of the sub-image 610-i of the feature map 610, proj_center_u and proj_center_v represent the predicted centers of the detected 2D object for the sub-image 610-i. pred[6] represents the output of the seventh channel of the second neural network model 304, pred[7] represents the output of the eighth channel of the second neural network model 304. K 3×3 represents the camera parameter matrix of the camera. (X, Y, Z') represents the coordinates of the corresponding 3D object in the above camera coordinate system, i.e., the position of the corresponding 3D object in 3D space.

[0042] By the above approach, the above predetermined neural network model 300 of the present disclosure can perform one-step 3D object detection based solely on 2D images and corresponding depth data, without requiring a large amount of 3D data for extensive calculations, thereby improving the efficiency of 3D object detection. Furthermore, the above predetermined neural network model according to embodiments of the present disclosure uses a linear depth prediction method as described in the present disclosure without performing complex calculations, thereby further improving the efficiency of 3D object detection.Furthermore, according to embodiments of the present disclosure, model anchors suitable for the above predefined neural network model of the present disclosure are created, whereby the above predefined neural network model achieves a higher recall rate in 3D object detection in the 3D target space, that is, the probability of actually existing 3D objects being missed is significantly reduced. Furthermore, the accuracy of 3D object detection is increased because the above predefined neural network model uses real depth data instead of 3D information inferred from 2D information.

[0043] Fig. 8 shows a schematic block diagram of an apparatus 800 for 3D object detection according to embodiments of the present disclosure. As in Fig. As shown in Figure 8, the device 800 comprises a capture unit 802 configured to capture a two-dimensional 2D image of a 3D target scene. The device 800 further comprises a generation unit 804 configured to determine depth data corresponding to the 2D image. The device 800 further comprises a recognition unit 806 configured to recognize a 3D object in the 3D target scene using a predetermined neural network model based on the 2D image and the depth data.

[0044] In some embodiments, the predetermined neural network model may include a first neural network model and a second neural network model. In some embodiments, the first neural network model may be created based on a neural network model for 2D object detection. In some embodiments, the neural network model for 2D object detection may be a 2D anchor neural network model for single-stage 2D object detection. In some embodiments, the second neural network model may be a head neural network model for the 3D object detection task.

[0045] In some embodiments, the device 800 may also include a model creation unit, wherein the model creation unit is configured to create the first neural network model based on the neural network model for 2D object detection. In some embodiments, the model creation unit may be configured to obtain predetermined depth data. In some embodiments, the model creation unit may be configured to cluster the predetermined depth data to group it into a plurality of clusters. In some embodiments, the model creation unit may be configured to place the 2D anchors in each of the plurality of clusters to obtain a model anchor for each of the clusters and thus create the first neural network model.In some embodiments, the seeding for each of the clusters may be performed such that the 2D anchors are concatenated with the mean of the given depth data in the cluster to obtain a model anchor for the cluster. In some embodiments, the number of 2D anchors may be a first number. In some embodiments, the number of clusters may be a second number. In some embodiments, the number of all model anchors in the first neural network model may be the product of the first number and the second number.

[0046] In some embodiments, the recognition unit 806 may be configured to obtain a first model detection result using the first neural network model based on the 2D image and the depth data. In some embodiments, the first model detection result may include a detected 2D object for the 2D image. In some embodiments, the first model detection result may include a predicted depth for the detected 2D object.

[0047] In some embodiments, the recognition unit 806 may be configured to extract features from the 2D image using the first neural network model to obtain a feature map of the 2D image. In some embodiments, the recognition unit 806 may be configured to divide the feature map into multiple sub-images. In some embodiments, the recognition unit 806 may be configured to obtain a corresponding anchor detection result for each of the multiple sub-images, respectively using all model anchors. In some embodiments, the corresponding anchor detection result may include a corresponding detected 2D object for the sub-image. In some embodiments, the corresponding anchor detection result may include a corresponding predicted depth for the corresponding detected 2D object.In some embodiments, for each model anchor of the first neural network model, the corresponding predicted depth may be determined based on the average and standard deviation of the corresponding depth data from the depth data corresponding to the sub-image. In some embodiments, the detection unit 806 may be configured to select a target anchor detection result from all anchor detection results based on the non-maximum suppression (NMS) algorithm as the sub-image detection result for that sub-image. In some embodiments, the first model detection result includes the sub-image detection results for each of the sub-images.

[0048] In some embodiments, the recognition unit 806 may be configured to determine a 3D object corresponding to the recognized 2D object using the second neural network model based on the first model detection result and corresponding parameters. In some embodiments, the corresponding parameters may include anchor parameters of the model anchors in the first neural network model. In some embodiments, the corresponding parameters may include camera parameters of the camera capturing the 2D image.

[0049] In some embodiments, the recognition unit 806 may be configured to set the corresponding predicted depth included in the partial image detection result for each partial image of the plurality of partial images as the predicted depth of the 3D object corresponding to the corresponding detected 2D object in the partial image detection result. In some embodiments, the recognition unit 806 may be configured to determine the orientation of the corresponding 3D object in the plan view based on the predicted depth and the center point of the model anchor provided for obtaining the partial image detection result. In some embodiments, the recognition unit 806 may be configured to determine the predicted length, width, and height of the corresponding 3D object based on a preset average length, width, and height corresponding to the type of the detected 2D object.In some embodiments, the recognition unit 806 may be configured to determine the position of the corresponding 3D object in 3D space based on the predicted depth, length, width, height, and camera parameters.

[0050] In the device 800 according to the embodiments of the present disclosure, the 3D object detection can be performed using the predetermined neural network model according to the embodiments of the present disclosure based solely on the 2D image and the depth data, thereby increasing the efficiency and accuracy of the 3D object detection.

[0051] Fig.9 shows a schematic block diagram of an example device 900 suitable for implementing embodiments of the present disclosure. The above-mentioned control device can be realized using the device 900. As shown in the figure, the device 900 includes a processor 901 that can perform various suitable actions and processes according to computer program instructions stored in a read-only memory (ROM) 902 and loaded into a random access memory (RAM) 903. Various programs and data required for the operation of the device 900 can also be stored in the RAM 903. The processor 901, the ROM 902, and the RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0052] Each process and processing described above, e.g., method 200 and processes 400, 406A, 500, 600, 700, may be executed by processor 901. In some embodiments, method 200 and processes 400, 406A, 500, 600, 700 may be implemented, for example, as a computer software program physically embodied in a machine-readable medium. In some embodiments, the computer program may be loaded and / or installed, in whole or in part, onto device 900 via ROM 902. When the computer program is loaded into RAM 903 and executed by processor 901, one or more actions of method 200 and processes 400, 406A, 500, 600, 700 described above may be performed.

[0053] The present disclosure may be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions for carrying out the various aspects of the present disclosure.

[0054] A computer-readable storage medium may be a tangible device capable of receiving and storing instructions for use by an instruction-executing device. The computer-readable storage medium may be, for example, but not limited to, an electronic storage medium, a magnetic storage medium, an optical storage medium, an electromagnetic storage medium, a semiconductor storage medium, or any suitable combination of the foregoing.More specific examples (non-exhaustive list) of computer-readable storage media include: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punched cards or raised structures in grooves on which instructions are stored, and any suitable combination of the above. The computer-readable storage media used here are not to be interpreted as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g.,light pulses traveling through a fiber optic cable) or electrical signals transmitted through wires.

[0055] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to various computing / processing devices or to an external computer or storage device over a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage on a computer-readable storage medium in the respective computing / processing device.

[0056] The computer program instructions for performing operations within the scope of the present disclosure may be assembly language instructions, Instruction Set Architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, where the programming languages include object-oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code may execute entirely on one computer, partially on one computer, as a standalone software package, partially on one computer and partially on a remote computer, or entirely on a remote computer or server.In the latter case, the remote computer may be connected to a computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or may be connected to an external computer (e.g., via an Internet service provider over the Internet). In some embodiments, the electronic circuit, such as a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA), may be adapted by using state information from computer-readable program instructions, where the electronic circuit may execute the computer-readable program instructions to implement various aspects of the present disclosure.

Claims

[1] A method for three-dimensional 3D object detection, comprising: Capturing a two-dimensional 2D image of a 3D target scene; Obtaining depth data corresponding to the 2D image; Detecting a 3D object in the 3D target scene using a given neural network model based on the 2D image and depth data. [2] The method of claim 1, wherein the predetermined neural network model comprises a first neural network model and a second neural network model, wherein the first neural network model is created based on a neural network model for 2D object detection, the second neural network model is a head neural network model for the 3D object detection task, and where the neural network model for 2D object detection is a neural network model with 2D anchors for one-stage 2D object detection. [3] The method of claim 2, wherein creating the first neural network model based on a neural network model for 2D object detection comprises: Obtaining predefined depth data; Clustering the given depth data to group the given depth data into multiple clusters; and Placing the 2D anchors in each of the multiple clusters to obtain a model anchor for each of the clusters and thus create the first neural network model, where the setting for each of the clusters is done such that the 2D anchors are linked to the mean of the given depth data in the cluster to obtain the model anchor for the cluster. [4] The method of claim 3, wherein detecting a 3D object in the 3D target scene using the predetermined neural network model based on the 2D image and the depth data comprises: Obtaining a first model detection result using the first neural network model based on the 2D image and the depth data; and Determining a 3D object corresponding to the detected 2D object using the second neural network model based on the first model detection result and corresponding parameters, wherein the first model detection result comprises a detected 2D object for the 2D image and a predicted depth for the detected 2D object, and where the corresponding parameters include anchor parameters of the model anchors in the first neural network model and camera parameters of the camera that captures the 2D image. [5] The method of claim 4, wherein obtaining a first model detection result using the first neural network model based on the 2D image and the depth data comprises: Extracting features from the 2D image using the first neural network model to obtain a feature map of the 2D image; Dividing the feature map into several sub-images; Obtaining a corresponding anchor detection result for each of the plurality of sub-images using all model anchors; and Selecting a target anchor detection result from all anchor detection results based on the Non-Maximum Suppression (NMS) algorithm as the sub-image detection result for this sub-image, wherein the corresponding anchor detection result comprises a corresponding detected 2D object for the partial image and a corresponding predicted depth for the corresponding detected 2D object, and wherein the first model detection result includes the partial image detection results for each of the partial images. [6] The method of claim 5, wherein determining a 3D object corresponding to the detected 2D object using the second neural network model based on the first model detection result and corresponding parameters comprises: setting the predicted depth included in the partial image detection result for each partial image of the plurality of partial images as the predicted depth of the 3D object corresponding to the corresponding detected 2D object in the partial image detection result; Determine the orientation of the corresponding 3D object in plan view based on the predicted depth and the center of the model anchor, which is intended to obtain the partial image acquisition result; Determine the predicted length, width and height of the corresponding 3D object based on the preset average length, width and Height corresponding to the type of detected 2D object; and Determine the position of the corresponding 3D object in 3D space based on the predicted depth, length, width, height and camera parameters. [7] The method of claim 5, wherein for each model anchor of the all model anchors of the first neural network model, the corresponding predicted depth is determined based on the average and the standard deviation of the corresponding depth data from the depth data corresponding to the partial image. [8] The method of claim 3, wherein the number of 2D anchors is a first number, the number of clusters is a second number, and the number of all model anchors in the first neural network model is the product of the first number and the second number. [9] Device for three-dimensional 3D object detection, comprising: a capture unit configured to capture a two-dimensional 2D image of a 3D target scene; a generating unit configured to acquire depth data corresponding to the 2D image; and a recognition unit configured to recognize a 3D object in the 3D target scene using a predetermined neural network model based on the 2D image and the depth data. [10] Control device comprising: at least one processor; and a memory coupled to the at least one processor and storing instructions which, when executed by the at least one processor, cause the control unit to execute a method according to any one of claims 1 to 8. [11] A vehicle comprising a control device according to claim 10 and the predetermined neural network model. [12] A computer-readable storage medium having computer-executable instructions stored thereon, the computer-executable instructions being executed by a processor to implement a method according to any one of claims 1 to 8.

Images