3D target detection method, device, equipment, storage medium and program product

Abstract

The application discloses a 3D target detection method, device, equipment, storage medium and program product. The method comprises the following steps: inputting a first image into a first model after training to obtain first information, wherein the first information comprises a heat map of a category of a target object, a heat map of an orientation of the target object, a heat map of a landing point category of the target object, a length-width of the target object, a center point offset of the target object, a center point offset of a ridge line of the target object, a landing point offset of the target object, a landing point center point offset of the target object, decoding the first information to obtain second information of the target object, wherein the second information comprises the category of the target object and at least one of the following: a 2D bounding box of the target object, the orientation of the target object, the landing point of the target object, and obtaining 3D information of the target object according to the second information of the target object. The application can improve the reliability of 3D target detection based on a camera.

Description

Technical Field

[0001] This application belongs to the field of autonomous driving, and in particular relates to a 3D target detection method, device, equipment, storage medium and program product. Background Technology

[0002] In the process of autonomous driving, ensuring the safety of occupants and surrounding vehicles and pedestrians places extremely high demands on environmental perception. In the field of autonomous driving technology, perception is a prerequisite for human-machine interaction, directly impacting the vehicle's understanding of its surroundings. Accurate perception of the surrounding environment is essential for the vehicle's decision-making and planning, ensuring safe driving and the safety of passengers. There are numerous technical approaches in the field of perception, which can be categorized by sensor type into LiDAR-based and camera-based approaches.

[0003] 3D (3) rd 3D object detection is a very important basic task in the field of autonomous driving. Compared with LiDAR, cameras have the advantage of high resolution and are widely used in autonomous driving. However, due to perspective projection, cameras lack depth information, which means that cameras have poor depth estimation accuracy, resulting in low reliability of 3D object detection. Summary of the Invention

[0004] This application provides a 3D target detection method, apparatus, device, storage medium, and program product that can solve the problem of low reliability in existing camera-based 3D target detection.

[0005] In a first aspect, embodiments of this application provide a 3D target detection method applied to a vehicle, the vehicle being equipped with a calibrated camera, the method comprising:

[0006] The first image captured by the camera is input into the trained first model to obtain first information about the target object in the first image. This first information includes: a heatmap representing the category of the target object; a heatmap representing the orientation of the target object; a heatmap representing the category of the grounding point of the target object; the length and width of the target object; the offset of the center point of the target object; the offset of the edge of the target object from its center point; the offset of the grounding point of the target object; and the offset of the grounding point of the target object from its center point.

[0007] Decoding the first information yields the second information of the target object. The second information includes the category of the target object and at least one of the following: the 2D bounding box of the target object, the orientation of the target object, and the grounding point of the target object.

[0008] Based on the second information of the target object, the 3D information of the target object is obtained.

[0009] Secondly, embodiments of this application provide a 3D target detection device applied to a vehicle, the vehicle being equipped with a calibrated camera, the device comprising:

[0010] A first acquisition module is used to input the first image captured by the camera into a trained first model to obtain first information about the target object in the first image. The first information includes: a heatmap representing the category of the target object; a heatmap representing the orientation of the target object; a heatmap representing the category of the grounding point of the target object; the length and width of the target object; the offset of the center point of the target object; the offset of the edge of the target object from its center point; the offset of the grounding point of the target object; and the offset of the grounding point of the target object from its center point.

[0011] A decoding module is used to decode the first information to obtain second information about the target object. The second information includes the category of the target object and at least one of the following: the 2D bounding box of the target object, the orientation of the target object, and the grounding point of the target object.

[0012] The second acquisition module is used to obtain the 3D information of the target object based on the second information of the target object.

[0013] Thirdly, embodiments of this application provide a 3D target detection device, the device including: a processor and a memory storing computer program instructions, wherein the processor executes the computer program instructions to implement the 3D target detection method as described in the first aspect.

[0014] Fourthly, embodiments of this application provide a computer storage medium storing computer program instructions, which, when executed by a processor, implement the 3D target detection method as described in the first aspect.

[0015] Fifthly, embodiments of this application provide a computer program product, characterized in that, when the instructions in the computer program product are executed by the processor of an electronic device, the electronic device performs the 3D target detection method as described in the first aspect.

[0016] In this embodiment, by inputting images acquired using a calibrated camera into a trained first model, eight types of information about the target object in the image can be obtained: a heatmap representing the target object's category, a heatmap representing the target object's orientation, a heatmap representing the target object's ground point category, the target object's length and width, the target object's center point offset, the target object's edge offset from its center point, the target object's ground point offset, and the target object's ground point offset from its center point. Thus, based on these eight types of information, the target object's category and at least one of the following can be decoded: 2D bounding box, orientation, and ground point. Furthermore, the target object's 3D information can be calculated. In this embodiment, since the first model can output eight types of information about 2D targets in the image, the performance of 2D target detection can be greatly improved, thereby enhancing the reliability of camera-based 3D target detection. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of the 3D target detection method provided in the embodiments of this application.

[0019] Figure 2 This is a network structure diagram of the first model provided in the embodiments of this application.

[0020] Figure 3 This is a schematic diagram of the vehicle provided in the embodiments of this application.

[0021] Figure 4 This is a schematic diagram of the orientation provided in the embodiments of this application.

[0022] Figure 5 This is one of the schematic diagrams illustrating the calculation of the height of the detection object provided in the embodiments of this application.

[0023] Figure 6 This is the second schematic diagram illustrating the calculation of the height of the detection object provided in the embodiments of this application.

[0024] Figure 7 This is a schematic diagram illustrating the calculation of 3D information provided in an embodiment of this application.

[0025] Figure 8 This is a schematic diagram of the algorithm provided in the embodiments of this application.

[0026] Figure 9This is a schematic diagram of the modules provided in an embodiment of this application.

[0027] Figure 10 This is a structural diagram of the 3D target detection device provided in the embodiments of this application.

[0028] Figure 11 This is a structural diagram of the 3D target detection device provided in the embodiments of this application. Detailed Implementation

[0029] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0030] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0031] The 3D target detection method provided in this application will be described in detail below with reference to the accompanying drawings and through some embodiments and application scenarios.

[0032] See Figure 1 , Figure 1 This is a flowchart of the 3D target detection method provided in this application embodiment. The 3D detection algorithm of this application embodiment can be applied to vehicles equipped with calibrated cameras. In practical applications, the camera can be a monocular camera, thereby further reducing the cost of 3D target detection. Of course, in other embodiments, the camera can also be other types of cameras, such as binocular cameras, etc. This application embodiment does not limit the form of the camera. Furthermore, the camera can be a forward-looking camera, in which case the image acquired by the camera is a forward-looking image.

[0033] like Figure 1As shown, a 3D object detection method may include the following steps:

[0034] Step 101: Input the first image captured by the camera into the trained first model to obtain the first information of the target object in the first image. The first information includes: a heat map representing the category of the target object, a heat map representing the orientation of the target object, a heat map representing the category of the grounding point of the target object, the length and width of the target object, the offset of the center point of the target object, the offset of the edge of the target object from the center point of the target object, the offset of the grounding point of the target object, and the offset of the grounding point of the target object from the center point of the target object.

[0035] In this embodiment of the application, the first model is used for 2D detection of target objects in an image.

[0036] In practice, during vehicle operation, a camera can be activated to capture images of the vehicle in motion. Before inputting the captured images into the first model, to improve the reliability of the first model's 2D detection of target objects in the images, the images can be preprocessed. The preprocessing process includes distortion correction operations, among other things.

[0037] The operation process of the first model can be found in [reference needed]. Figure 2 .like Figure 2 As shown, the operation of the first model may include the following parts:

[0038] Data Input 21: Receive the preprocessed image.

[0039] Feature Extraction 22: Perform feature extraction and feature fusion operations on the image. The extracted features will include semantic and positional information of the target objects contained in the image. In one optional implementation, this part of the operation can be performed based on the backbone network of a convolutional neural network.

[0040] Output: Output eight types of information about the target object in the image, namely:

[0041] A heatmap used to characterize the category of a target object, in Figure 2 The representation is as follows: Heatmap: Target category 231,

[0042] A heatmap used to characterize the orientation of a target object, in Figure 2 The representation is as follows: Heatmap: Vehicle orientation 232,

[0043] A heatmap used to characterize the type of grounding point of a target object, in Figure 2 The data is represented in Heatmap as follows: Grounding point category 233.

[0044] The length and width of the target object, in Figure 2 The dimensions are as follows: Box size: Target width and height 234.

[0045] The offset of the center point of the target object, in Figure 2 This is represented as: Offset: Target center point offset by 235.

[0046] The offset of the edge of the target object from the center point of the target object, in Figure 2 This is represented as: Offset: The edge is offset from the center point by 236.

[0047] The offset of the grounding point of the target object, in Figure 2 The result is: Offset: Grounding point offset 237.

[0048] The offset of the grounding point of the target object from the center point of the target object, in Figure 2 In this context, it is represented as: Offset: The offset of the grounding point from the center point is 238.

[0049] In this embodiment, the first model includes eight output heads, each used to output the aforementioned eight types of information, specifically including three heatmaps, one dimension information, and four offsets. Thus, the target object's position information, orientation information, dimension information, and offset relative to its true position can be obtained.

[0050] In this application embodiment, the target object can be categorized into eight types: small car, medium car, large car, person, cyclist, bicycle, tricycle, animal, and handcart. For target information categorized as a vehicle, its orientation can be categorized into eight types: front, rear, left, right, left front, right front, left rear, and right rear, which can be used to roughly represent the vehicle's heading angle. Therefore, as... Figure 2 As shown, when the target object is a vehicle, the vehicle target orientation 24 can be found based on the heatmap used to characterize the target object's category and the heatmap used to characterize the target object's orientation.

[0051] When the target object is a vehicle, such as Figure 3 As shown, the types of grounding points may include, but are not limited to, front wheel grounding point 31 and rear wheel grounding point 32. The edge line is the boundary line in the target 2D bounding box that does not overlap with the vehicle body 2D bounding box; the target 2D bounding box is either the front or rear 2D bounding box. Figure 3 In the middle, edge 33 is the boundary line in the target 2D bounding box 34 that does not overlap with the rear 2D bounding box 35.

[0052] In this embodiment of the application, since the first model can output eight types of information about 2D targets in the image, the performance of 2D target detection in the image can be greatly improved by the first model, thereby improving the reliability of 3D target detection based on the camera.

[0053] Step 102: Decode the first information to obtain the second information of the target object. The second information includes the category of the target object and at least one of the following: the 2D bounding box of the target object, the orientation of the target object, and the grounding point of the target object.

[0054] The information output by the first model can be input into the channel decoder, which decodes it to obtain the 2D information of the target object in the image. Specifically, it can include, but is not limited to, the target's 2D bounding box and key points (i.e., grounding points).

[0055] When the target object is a vehicle, the 2D bounding box of the target object is the vehicle body 2D bounding box. The orientation of the target object can be represented by the target 2D bounding box, which is the front 2D bounding box or the rear 2D bounding box. That is, the front 2D bounding box or the rear 2D bounding box contains the orientation of the vehicle.

[0056] It is worth noting that, considering that the target object in the image may be spatially occluded, the target object may have missing 2D bounding boxes and / or key points.

[0057] like Figure 4 The angle of the vehicle in the image can be divided into 8 cases, which can be categorized into four main types:

[0058] Look at the front and rear of the car, including Figure 4 The numbers 1 and 5 in the text.

[0059] Looking at the front and rear of the car from the side, including Figure 4 The numbers 2, 4, 6, and 8 in the text are:

[0060] Looking at the side of the vehicle, including Figure 4 The numbers 3 and 7 in the text.

[0061] Spatial obstruction.

[0062] When viewing the front and rear of the vehicle directly, the key points of the wheels are lost, and the 2D bounding box of the vehicle body overlaps with the 2D bounding box of the front / rear of the vehicle. The second information of the target object only includes a 2D bounding box.

[0063] For situations where the front and rear of the car are viewed at an angle, such as Figure 3 As shown, the second information of the target object includes the vehicle body 2D bounding box 34, the target 2D bounding box 35, the front wheel contact point 31, and the rear wheel contact point 32.

[0064] For a side view of the vehicle body, the second information of the target object only includes the 2D bounding box of the vehicle body, the contact point of the front wheel, and the contact point of the rear wheel.

[0065] For spatial occlusion, the situation can be categorized based on the missing keypoints and 2D bounding boxes. This can be divided into four cases: missing one keypoint; missing one keypoint and the front / rear 2D bounding boxes; missing two keypoints; no front / rear bounding boxes; or no keypoints. However, this is not the only possible scenario.

[0066] Step 103: Obtain the 3D information of the target object based on the second information of the target object.

[0067] In practice, the 2D information of the target object in the first image can be used to calculate and process the 3D information of the target object in the first image.

[0068] An object in three-dimensional space can be represented by a 3D box, whose parameters can include spatial coordinates, dimensional information (length, width, and height), and heading angle. Therefore, the 3D information of the target object can include at least one of the following: the distance between the detected object and the vehicle, the dimensional information of the detected object, and the heading angle of the detected object, where the target object is the image of the detected object in the image, that is, the detected object is the object of the target object in three-dimensional space.

[0069] The 3D target detection method of this embodiment, by inputting images acquired using a calibrated camera into a first model, can obtain eight types of information about the target object in the image: a heatmap representing the target object's category, a heatmap representing the target object's orientation, a heatmap representing the target object's ground point category, the target object's length and width, the target object's center point offset, the target object's edge offset from its center point, the target object's ground point offset, and the target object's ground point offset from its center point. Thus, based on these eight types of information, the target object's category and at least one of the following can be decoded: 2D bounding box, orientation, and ground point, and subsequently, the target object's 3D information can be calculated. In this embodiment, since the first model can output eight types of information about 2D targets in the image, the performance of 2D target detection can be greatly improved, thereby enhancing the reliability of camera-based 3D target detection.

[0070] In some embodiments, before inputting the first image captured by the camera into the trained first model to obtain the first information, the method further includes:

[0071] Based on the CenterNet network, a first model is constructed, comprising eight network output heads. The outputs of these eight output heads are: a heatmap representing the category of the target object, a heatmap representing the orientation of the target object, a heatmap representing the category of the target object's grounding point, the length and width of the target object, the offset of the target object's center point, the offset of the target object's edge from its center point, the offset of the target object's grounding point, and the offset of the target object's grounding point from its center point.

[0072] The target objects in the training sample images are labeled. The labeled information includes: the category of the target object, the 2D bounding box of the target object, the orientation of the target object, and the ground point of the target object.

[0073] The first model is trained using the labeled training sample images until it converges, thus obtaining the trained first model.

[0074] In this embodiment, the first model is built based on CenterNet.

[0075] The CenterNet network model is a single-stage object detection algorithm that does not require anchor boxes. Compared with other single-stage or two-stage object detection algorithms, this algorithm has the following advantages:

[0076] This algorithm eliminates the inefficient and complex operation of pre-defined anchor boxes, thus improving the performance of the detection algorithm.

[0077] This algorithm performs filtering directly on the heatmap, eliminating the time-consuming non-maximum suppression (NMS) processing of the bounding boxes, thus further improving the overall algorithm speed.

[0078] This algorithm can be applied not only to 2D object detection, but also, by modification, to other tasks such as 3D object detection and human keypoint detection, thus demonstrating its excellent versatility.

[0079] Therefore, this embodiment obtains a first model by improving CenterNet. Specifically, the first model can be obtained by adjusting the network output headers of CenterNet. The adjusted CenterNet includes eight network output headers for inputting data such as... Figure 2 The Heatmap, Box size, and Offset are shown below. Thus, by making the above adjustments, the 2D object detection performance of CenterNet can be improved, thereby increasing the reliability of 3D object detection.

[0080] The first model obtained can be trained using labeled training sample images.

[0081] In practice, training the first model may include the following steps:

[0082] The training sample images are labeled with information such as the category, 2D bounding box, orientation, and grounding point of the target object in the image.

[0083] The labeled training sample images are input into the constructed first model, which then outputs the first information of the target object in the training sample images.

[0084] Decode the first information of the target object in the training sample image to obtain the second information of the target object in the training sample image.

[0085] The loss function is obtained by comparing the second information of the target object in the training sample image with the labeled information in the training sample image.

[0086] Update the model parameters of the first model using the loss function.

[0087] This process continues until the first model converges, resulting in the trained first model.

[0088] In this way, using the trained first model to predict target objects in the image can simplify the prediction of target objects, improve the 2D target detection performance, and thus improve the reliability of 3D target detection.

[0089] It should be noted that in other embodiments, the first model can be built on other keypoint-based object detection networks, such as CornerNet or ExtremeNet.

[0090] The following describes how the 3D information of the target object is obtained in the embodiments of this application.

[0091] 1. The 3D information of the target object includes the distance between the detected object and the vehicle, wherein the target object is the image of the detected object in the image.

[0092] In this case, obtaining the 3D information of the target object based on the second information of the target object may include:

[0093] Calculate the pixel coordinates of the target ground point corresponding to the target object and transform them to the first coordinate in the world coordinate system.

[0094] Based on the first coordinates and the camera calibration parameters, the distance between the target detection point and the vehicle is calculated to obtain the distance between the detection object and the vehicle, wherein the target grounding point is the image of the target detection point in the image.

[0095] Based on the principle of perspective transformation, the coordinates of a point in the world coordinate system can be mapped to those of a point in the pixel plane using formula (1):

[0096] (1)

[0097] in Represents the coordinates in the pixel coordinate system. This represents the distance in the camera coordinate system. This represents the real-world distance corresponding to a single pixel. Indicates the position of the camera's center point. Indicates the camera's focal length. Represents the rotation matrix. Represents the translation matrix. Represents coordinates in the world coordinate system.

[0098] Therefore, the coordinates of the target's grounding point in the world coordinate system, i.e., the first coordinate, can be calculated through inverse perspective transformation. Additionally, using the camera's calibration parameters, the distance between the target detection point and the vehicle can be obtained. The specific calculation principle can be found in related technical descriptions and will not be elaborated here.

[0099] The distance between the target detection point and the vehicle can be used to determine the distance between the detected object and the vehicle. Specifically, the distance between the detected object and the vehicle can be determined based on the positional relationship between the target detection point and the detected object. For example, if the distance between the detected object and the vehicle is specifically expressed as the distance between the center point of the detected object and the vehicle, the distance between the detected object and the vehicle can be calculated based on the distance between the target detection point and the center point of the detected object, as well as the distance between the target detection point and the vehicle.

[0100] After determining the distance between the object to be detected and the vehicle, the coordinates of the object to be detected can be calculated based on the vehicle's positioning information and the distance value. In other words, the coordinates of the target object in the world coordinate system can be obtained.

[0101] It is worth noting that the target grounding point may or may not be the grounding point of the target object, as explained below:

[0102] In some embodiments, when the second information of the target object includes a 2D bounding box of the target object, any point on the lower boundary of the 2D bounding box can be determined as the target grounding point.

[0103] In other embodiments, when the target object is classified as a vehicle, the 2D bounding box of the target object is a vehicle body 2D bounding box, and the orientation of the target object is represented by a target 2D bounding box, which is either a front 2D bounding box or a rear 2D bounding box.

[0104] Before calculating the pixel coordinates of the target ground point corresponding to the target object and transforming them to the first coordinate of the world coordinate system, the method further includes:

[0105] The target grounding point is determined based on the second information of the target object.

[0106] As can be seen from the foregoing, in practical applications, the target object in the image may be spatially occluded. Different occlusion situations will lead to different specific contents of the second object of the target object. Therefore, the target grounding point can be determined based on the specific contents of the second information of the target object.

[0107] In some optional implementations, determining the target grounding point based on the second information of the target object may include:

[0108] If the second information of the target object includes only a 2D bounding box, the midpoint of the lower boundary of the 2D bounding box is determined as the target grounding point.

[0109] When the second information of the target object includes the vehicle body 2D bounding box, the target 2D bounding box, the front wheel contact point, and the rear wheel contact point, the intersection of the extension of the first connecting line and the extension of the edge line is determined as the target contact point. The first connecting line is the line connecting the front wheel contact point and the rear wheel contact point, and the edge line is the boundary line in the target 2D bounding box that does not overlap with the vehicle body 2D bounding box.

[0110] If the second information of the target object only includes the 2D bounding box of the vehicle body, the front wheel contact point and the rear wheel contact point, the midpoint between the front wheel contact point and the rear wheel contact point is determined as the target contact point.

[0111] In other words, for the aforementioned case of viewing the front and rear of the vehicle directly, the midpoint of the lower boundary of the 2D bounding box can be determined as the target ground point.

[0112] For the aforementioned cases of oblique views of the front / rear of the vehicle, the intersection of the extended line connecting the vehicle's ground contact point and the extended line of the vehicle's edge can be determined as the target ground contact point.

[0113] For the aforementioned situation where the vehicle body is viewed from the side, the endpoint of the vehicle's contact point can be determined as the target contact point.

[0114] Therefore, in this embodiment, the target grounding points are not considered target objects. However, it is understandable that in other implementations, the vehicle grounding point can be directly identified as the target grounding point, in which case the target grounding point would be considered a target object.

[0115] As can be seen from the above, for different vehicle conditions, the target grounding point used to calculate the distance between the detection object and the vehicle can be determined in different ways, thereby improving the flexibility of determining the distance between the detection object and the vehicle.

[0116] Second, the 3D information of the target object also includes the height of the detected object, wherein the target object is the image of the detected object in the image.

[0117] In this case, obtaining the 3D information of the target object based on the second information of the target object may include:

[0118] The height of the object being detected is calculated using the first and second calculation formulas.

[0119] The first calculation formula is:

[0120]

[0121] The second calculation formula is:

[0122]

[0123] Where H is the distance between the target detection point and the horizon, Z is the distance between the target detection point and the vehicle, y is the height of the target object in the image coordinate system, and f is the focal length of the camera. It is the height of the 2D bounding box of the target object. is the distance between the target ground point corresponding to the target object and the horizon in the pixel coordinate system, the target ground point is the image of the target detection point in the image, and h is the height of the detected object in the world coordinate system.

[0124] In this embodiment, given the known distance Z between the target detection point and the vehicle, the camera focal length f, and the height y of the target object in the image coordinate system, the distance between the target detection point and the horizon can be calculated using similar triangles. For easier understanding, please refer to... Figure 5 ,exist Figure 5 In the diagram, the vehicles on the left are the autonomous vehicles, and the vehicles on the right are the detected vehicles. It should be noted that... Figure 5 It is not an actual image, but an image obtained through reasonable transformation, in order to facilitate the understanding of the calculation of H.

[0125] Additionally, the height of the 2D bounding box can be calculated based on the first image. Height from the target's ground contact point to the horizon (vanishing line of sight) The ratio of .

[0126] Thus, for reference Figure 6 The height h of the detected object in the world coordinate system can be calculated based on H and the ratio mentioned above. For specific calculation, please refer to the second calculation formula mentioned above.

[0127] By using the above method, the height of the target object can be calculated by using the ratio of the height of the target object's 2D bounding box in the pixel coordinate system to the height from the target ground point to the horizon, and the distance between the target detection point and the vehicle. This ensures the accuracy of the target object's height detection.

[0128] As described above, in this embodiment, the target object includes eight categories. When the target object is a person or animal, due to its small size and slow speed, autonomous driving planning can be performed based on the calculated distance between the corresponding detection object and the vehicle. However, for vehicles, to improve driving safety, the 3D information of the target object may further include at least one of the following: the heading angle of the detection object, the length of the detection object, and the width of the detection object.

[0129] Third, the 3D information of the target object also includes the heading angle of the detected object.

[0130] In this case, obtaining the 3D information of the target object based on the second information of the target object may include:

[0131] When the second information of the target object includes only a 2D bounding box, the heading angle of the detected object is determined according to the category of the 2D bounding box. Specifically, if the 2D bounding box is a front 2D bounding box, the heading angle of the detected object is determined to be -90 degrees; if the 2D bounding box is a rear 2D bounding box, the heading angle of the detected object is determined to be 90 degrees.

[0132] If the second information of the target object includes the vehicle body 2D bounding box, the target 2D bounding box, the front wheel contact point and the rear wheel contact point, or if the second information of the target object only includes the vehicle body 2D bounding box, the front wheel contact point and the rear wheel contact point, the heading angle of the detected object is determined based on the front wheel contact point and the rear wheel contact point.

[0133] In other words, for the aforementioned case of viewing the front and rear of the vehicle directly, the heading angle of the detected object can be determined directly through the category of the 2D bounding box in the second information of the target object. Specifically, when the 2D bounding box is the front 2D bounding box, the heading angle of the detected object is -90 degrees, and when the 2D bounding box is the rear 2D bounding box, the heading angle of the detected object is 90 degrees.

[0134] In other implementations, the heading angle can be determined using some prior knowledge, such as based on lane conditions and previous tracking results of the target object.

[0135] For both the aforementioned cases of oblique views of the front / rear of the vehicle and the aforementioned cases of frontal views of the side of the vehicle body, the grounding point and its category in the second information of the target object can be used to determine the situation.

[0136] In some alternative implementations, determining the heading angle of the detected object based on the contact points of the front and rear wheels may include:

[0137] The front wheel contact point is transformed to the world coordinate system to obtain the second detection point.

[0138] The rear wheel contact point is converted to the world coordinate system to obtain the third detection point.

[0139] The angle between the direction from the third detection point to the second detection point and the horizontal direction is determined as the heading angle of the object being detected.

[0140] For easier understanding, please refer to Figure 7 After obtaining the grounding point in the pixel coordinate system, the grounding point can be projected onto the world coordinate system to obtain a line segment in the top view. The angle formed by this line segment and the horizontal line in the top view can be used to calculate the heading angle of the detected object.

[0141] Since the first model can identify the type of wheel contact point, that is, it can know which of the two wheel contact points is the front wheel contact point and the rear wheel contact point, the line segments in the top view in the world coordinate system have a direction, and the angle between this direction and the horizontal line direction can be determined as the heading angle of the detected object.

[0142] This ensures the accuracy of obtaining the heading angle of the object being detected.

[0143] In some alternative implementations, the angle between the direction from the rear wheel contact point to the front wheel contact point and the direction of the horizontal line in the pixel coordinate system can be determined as the heading angle of the detected object. In this way, there is no need to convert between the pixel coordinate system and the world coordinate system, which simplifies the acquisition of the heading angle of the detected object.

[0144] Fourth, the 3D information of the target object also includes the length of the detected object.

[0145] In this case, obtaining the 3D information of the target object based on the second information of the target object may include:

[0146] If the second information of the target object includes only a 2D bounding box, the reference length is determined to be the length of the detected object.

[0147] When the second information of the target object includes the vehicle body 2D bounding box, the target 2D bounding box, the front wheel contact point, and the rear wheel contact point, the first length is transformed to the world coordinate system and determined as the length of the detected object, wherein the first length is the length of the lower boundary of the vehicle body 2D bounding box.

[0148] When the second information of the target object only includes the 2D bounding box of the vehicle body, the ground contact point of the front wheel, and the ground contact point of the rear wheel, the length of the 2D bounding box of the vehicle body transformed to the world coordinate system is determined as the length of the detected object.

[0149] In the case of viewing the front and rear of the vehicle directly, since the length of the vehicle is unknown due to the viewing of the front / rear, a prior value, i.e. a reference length value, can be used as the length of the vehicle.

[0150] For the aforementioned case of oblique view of the front / rear of the vehicle, the length of the lower boundary of the 2D bounding box of the vehicle body is the length of the vehicle in the pixel coordinate system. Therefore, the length of the lower boundary of the 2D bounding box of the vehicle body can be transformed to the world coordinate system to obtain the length of the vehicle.

[0151] For the aforementioned case of a vehicle body viewed from the side, the length of the 2D bounding box of the vehicle body is the length of the vehicle in the pixel coordinate system. Therefore, the length of the 2D bounding box of the vehicle body can be converted to the world coordinate system to obtain the length of the vehicle.

[0152] This ensures the accuracy of the vehicle length measurement.

[0153] Fifth, the 3D information of the target object also includes the width of the detected object.

[0154] In this case, obtaining the 3D information of the target object based on the second information of the target object may include:

[0155] If the second information of the target object includes only a 2D bounding box, the width of the 2D bounding box is transformed to the width in the world coordinate system and determined as the width of the detected object.

[0156] When the second information of the target object includes the vehicle body 2D bounding box, the target 2D bounding box, the front wheel contact point, and the rear wheel contact point, the width of the detected object is calculated based on the heading angle and the second length of the detected object. The second length is the length transformed to the world coordinate system by the third length, which is the length between the target wheel contact point and the target vertex of the target 2D bounding box. The target vertex is the vertex in the target 2D bounding box that is closest to the target wheel contact point. If the target 2D bounding box is the front 2D bounding box, the target wheel contact point is the front wheel contact point; if the target 2D bounding box is the rear 2D bounding box, the target wheel contact point is the rear wheel contact point.

[0157] If the second information of the target object only includes the 2D bounding box of the vehicle body, the front wheel contact point, and the rear wheel contact point, the reference width is determined as the width of the detected object.

[0158] In the case of viewing the front and rear of the vehicle from the front, the width of the 2D bounding box is the width of the vehicle in the pixel coordinate system. Therefore, the width of the 2D bounding box can be converted to the world coordinate system to obtain the width of the vehicle.

[0159] Regarding the aforementioned case of viewing the front / rear of the vehicle at an angle, the second length is... Figure 7 In the figure, D, assume the vehicle's heading angle is... So, in Figure 7 In the equation, the vehicle width w = D / cos(π / 2 - ... ).

[0160] Regarding the aforementioned situation of viewing the vehicle body from the side, since the vehicle width is unknown due to the perspective of viewing the vehicle body from the side, a priori value, namely a reference width value, can be used as the vehicle width.

[0161] This ensures the accuracy of vehicle width measurement.

[0162] In some embodiments, after obtaining the 3D information of the target object based on the second information of the target object, the method may further include:

[0163] Establish a correspondence between the 3D information and the identification information of the target object.

[0164] Obtain the second 3D information corresponding to the identification information, wherein the second 3D information is obtained based on a second image, which is the image preceding the first image captured by the camera.

[0165] Based on the 3D information and the second 3D information, the dynamic information of the detected object is calculated, wherein the target object is the image of the detected object in the image.

[0166] In practice, a target tracking algorithm can be used to determine the identification information corresponding to the 3D information of the target object in the first image and establish the correspondence between the two.

[0167] In this way, the dynamic information of the detected object, such as speed and acceleration, can be estimated based on the time difference between the acquisition of two consecutive frames of images captured by the camera, and the 3D information of the same detected object obtained from these two frames. The dynamic information of the detected object can then be used to control the dynamic information of the vehicle, thereby improving the safety of autonomous driving.

[0168] It should be noted that the various optional implementation methods described in the embodiments of this application can be combined with each other or implemented individually without conflict, and the embodiments of this application do not limit this.

[0169] For ease of understanding, the following example is provided:

[0170] The overall algorithm flow of this application embodiment can be found in [reference]. Figure 8 .

[0171] First, a forward-view image 81 of the vehicle in motion is captured using a forward-view monocular camera.

[0172] Then, the image is input into network model 82 (i.e. the first model mentioned above). This process involves image preprocessing, including distortion correction. The network model can output eight types of information about the target object in the image, which will be parsed in the channel decoder.

[0173] The output of the network model is input into the post-processing module 83, which integrates multiple pieces of information to output the target's category, center point coordinates, three-dimensional dimensions, and heading angle in the world coordinate system.

[0174] Next, the multi-target tracking algorithm 84 is used to obtain information such as the target's velocity and acceleration. Finally, all results are sent to the vehicle via the Ethernet (ETH) data transmission module 85.

[0175] exist Figure 8 In the network model 82, which is a 2D box / key point detection network, the post-processing module 83 includes: channel decoder parsing / quantization fusion 831, 3D pose calculation algorithm 832, and camera / vehicle coordinate system transformation 833.

[0176] The system in this application embodiment is divided into, as follows: Figure 9The module shown includes: target 2D bounding box and key point detection module 91, 3D information calculation and post-processing module 92, target tracking module 93, and obstacle velocity estimation module 94. Each module will be described in detail below.

[0177] (1) Target 2D bounding box and key point detection module 91

[0178] In order to calculate the 3D information of the target based on the prediction of 2D images, the embodiments of this application design the output head of the network and specify the annotation rules for training the neural network.

[0179] The network training data is labeled with eight categories: small cars, medium cars, large cars, people, cyclists, bicycles, tricycles, animals, and handcarts, along with their orientation and ground contact information. During labeling, this embodiment requires labeling a 2D bounding box encompassing the entire vehicle from top left to bottom right. For each vehicle, a 2D bounding box containing either the front or rear is also needed. The labels for these front and rear bounding boxes contain the vehicle's orientation information, specifically the eight orientation categories: front, rear, left, right, left-front, right-front, left-rear, and right-rear, roughly representing the vehicle's heading angle. After labeling the 2D bounding boxes, the ground contact information is labeled. The labeler needs to assign categories to the front and rear wheels' ground contact points, namely front wheel point and rear wheel point. After this labeling, a vehicle in this embodiment has two 2D bounding boxes and two wheel ground contact points. This labeled data will guide the model training process later.

[0180] In the network model design section, embodiments of this application can use a keypoint-based object detection network to construct the network model. The network architecture constructed in embodiments of this application is as follows: Figure 2 As shown.

[0181] (2) 3D information calculation and post-processing module 92

[0182] In order to extract the 3D information of the target in the world coordinate system from a 2D image based on limited information (2D bounding box and key points), this application embodiment will divide the target vehicle condition into eight working conditions for discussion.

[0183] For distance estimation:

[0184] By using inverse perspective transformation, the result (u,v) of the pixel plane prediction can be transformed into the world coordinate system (X,Y,Z). With the help of camera calibration, the distance of a pixel point belonging to the ground (such as the vehicle grounding point) from the vehicle in this embodiment can be obtained. In this way, the distance estimate can be obtained.

[0185] For the estimation of height:

[0186] Given the distance Z, camera focal length f, and image coordinate system height y, the distance from the vehicle's ground contact point to the horizon can be calculated, such as... Figure 5 As shown, it can be calculated using the method for similar triangles, specifically using formula (2):

[0187] (2)

[0188] Additionally, the ratio of the vehicle's 2D bounding box height to the height from the ground point to the horizon (vanishing line) is calculated, such as... Figure 6 As shown, the height of the vehicle is calculated using this ratio, specifically using formula (3):

[0189] (3)

[0190] When looking directly at the front / rear of the car:

[0191] Because the key points of the wheels are lost, and the overall 2D frame of the vehicle body overlaps with the front / rear frame, the information obtainable in this embodiment is only a single 2D frame. Therefore, this embodiment uses the two points on the lower edge of the 2D frame as the target grounding points, and the midpoint of the two points as the nearest collision point between the target vehicle and the vehicle itself. The distance to the vehicle is calculated, and then a height value is calculated based on the grounding point distance. Due to the unknown vehicle length caused by the frontal view, this embodiment provides the vehicle length using a priori value. The vehicle's heading angle is determined based on the category of the 2D bounding box: -90 degrees when the 2D frame is identified as the front of the vehicle, and 90 degrees when it is identified as the rear of the vehicle. The heading angle can also be determined based on some prior knowledge, such as the lane configuration and tracking results.

[0192] For situations where the front / rear of the car is viewed at an angle:

[0193] Under these conditions, the contact points of the front and rear wheels, the 2D bounding box of the vehicle body, the 2D bounding box of the front and rear of the vehicle, and clear edge lines can be identified. Therefore, the distance to the vehicle can be determined by the intersection of the extended line connecting the vehicle's contact points and the extended line of the edge line. Figure 7 The method for calculating vehicle 3D information is analyzed in detail, and the vehicle height will be determined by similar triangles.

[0194] The vehicle's heading angle can be determined using the line connecting the two contact points, the camera model, and ground priors (such as the angle with the lane lines). After obtaining the contact points in the pixel coordinate system, projecting them into the world coordinate system yields a line segment in the top-view perspective. The angle between this line segment and the centerline of the top-view can then be calculated. Since the types of wheel contact points can be identified—that is, knowing which of the two points represents the front and rear wheels—the line segment in the top-view in the world coordinate system has a direction. The angle between this direction and the horizontal line is the vehicle's heading angle, thus determining the vehicle's orientation. The vehicle's width can be calculated... Figure 7 The width of the vehicle is inferred from the value of D.

[0195] For situations where the vehicle is viewed from the side:

[0196] Only the 2D bounding box of the entire vehicle body and the wheel contact points can be identified. The orientation information of the target vehicle can be estimated using the front and rear wheel contact points with their respective categories. Distance is estimated using the vehicle contact points. Due to the loss of vehicle width, a priori value (e.g., approximately 2.8 meters) can be provided based on the vehicle model. The length and height of the vehicle can be estimated using the side lengths of the 2D bounding box.

[0197] In cases where there is spatial obstruction:

[0198] First, it can be determined whether the target affects the vehicle. The first target radiating outward from the vehicle is defined as an influential target, and the second target is an uninfluenced target. Targets that do not affect the vehicle will not be considered in this embodiment. For other influential targets, this embodiment will use other sensors (such as wide-angle cameras and side-view cameras) to detect them using the same algorithm.

[0199] (3) Target tracking module 93

[0200] Since stable target detection results are required over a period of time, predictions based on a single frame can be used, and a unique ID information can be assigned through the target tracking module. The target can be represented by an eight-dimensional vector. ,in Represents the coordinates of the center point of the target. It represents a ratio in the horizontal and vertical directions, that is, the aspect ratio. Used to indicate height, These represent the velocity information of each component in the image coordinate system. The overall tracking algorithm uses a Kalman filter to predict and update the trajectory, and then uses a Hungarian algorithm for data association. The cost matrix used for data association is composed of the intersection-over-union (IoU) ratio between the predicted position in the current frame and the detection result in the current frame. Through this tracking module, a stable target ID and its dynamic information can be obtained.

[0201] (4) Obstacle velocity estimation module 94

[0202] The image recognition-based velocity estimation module provides velocity information for the identified targets. It can transform targets identified in the image coordinate system to the global coordinate system, and use the difference in target position coordinates between two frames, as well as the time difference between the two frames, to calculate the target's velocity estimate in the global coordinate system.

[0203] As can be seen, the embodiments of this application can rely solely on a single forward-looking monocular camera to detect obstacles and obtain 3D information of the target, without the need for LiDAR to assist in obtaining the true value of 3D information, thus reducing sensor dependence. For labeled data, this improves labeling efficiency. Compared to labeled point clouds, labeled images are easier to inspect, more convenient to label, and require a simpler environment. It also reduces data acquisition costs. When conducting large-scale data acquisition, each acquisition vehicle does not need to be equipped with LiDAR, further reducing costs.

[0204] Based on the 3D target detection method provided in the above embodiments, this application also provides specific implementations of a 3D target detection device. Please refer to the following embodiments.

[0205] See Figure 10 The 3D target detection device provided in this application embodiment may include:

[0206] The first acquisition module 1001 is used to input the first image captured by the camera into the trained first model to obtain first information about the target object in the first image. The first information includes: a heatmap representing the category of the target object; a heatmap representing the orientation of the target object; a heatmap representing the category of the grounding point of the target object; the length and width of the target object; the offset of the center point of the target object; the offset of the edge of the target object from the center point of the target object; the offset of the grounding point of the target object; and the offset of the grounding point of the target object from the center point of the target object.

[0207] Decoding module 1002 is used to decode the first information to obtain second information about the target object. The second information includes the category of the target object and at least one of the following: the 2D bounding box of the target object, the orientation of the target object, and the grounding point of the target object.

[0208] The second acquisition module 1003 is used to obtain the 3D information of the target object based on the second information of the target object.

[0209] In some embodiments, the apparatus further includes:

[0210] A construction module is used to build a first model based on the CenterNet network. The first model includes eight network output heads, the outputs of which are: a heatmap representing the category of the target object, a heatmap representing the orientation of the target object, a heatmap representing the category of the target object's grounding point, the length and width of the target object, the offset of the target object's center point, the offset of the target object's edge from its center point, the offset of the target object's grounding point, and the offset of the target object's grounding point from its center point.

[0211] The annotation module is used to annotate target objects in training sample images. The annotation information includes: the target object's category, the target object's 2D bounding box, the target object's orientation, and the target object's ground point.

[0212] The training module is used to train the first model using the labeled training sample images until the first model converges, thus obtaining the trained first model.

[0213] In some embodiments, the 3D information of the target object includes the distance between the detected object and the vehicle, wherein the target object is an image of the detected object in the image.

[0214] Second acquisition module:

[0215] The first calculation unit is used to calculate the first coordinate in the world coordinate system of the pixel coordinates of the target ground point corresponding to the target object.

[0216] The second calculation unit is used to calculate the distance between the target detection point and the vehicle based on the first coordinates and the calibration parameters of the camera, thereby obtaining the distance between the detected object and the vehicle, wherein the target grounding point is the image of the target detection point in the image.

[0217] In some embodiments, when the target object is classified as a vehicle, the 2D bounding box of the target object is a vehicle body 2D bounding box, and the orientation of the target object is represented by a target 2D bounding box, which may be a front 2D bounding box or a rear 2D bounding box.

[0218] The device further includes:

[0219] The second determining module is used to determine the target grounding point based on the second information of the target object.

[0220] In some embodiments, the second acquisition module is specifically used for:

[0221] If the second information of the target object includes only a 2D bounding box, the midpoint of the lower boundary of the 2D bounding box is determined as the target grounding point.

[0222] When the second information of the target object includes the vehicle body 2D bounding box, the target 2D bounding box, the front wheel contact point, and the rear wheel contact point, the intersection of the extension of the first connecting line and the extension of the edge line is determined as the target contact point. The first connecting line is the line connecting the front wheel contact point and the rear wheel contact point, and the edge line is the boundary line in the target 2D bounding box that does not overlap with the vehicle body 2D bounding box.

[0223] If the second information of the target object only includes the 2D bounding box of the vehicle body, the front wheel contact point and the rear wheel contact point, the midpoint between the front wheel contact point and the rear wheel contact point is determined as the target contact point.

[0224] In some embodiments, the 3D information of the target object further includes the height of the detected object, wherein the target object is the image of the detected object in the image.

[0225] The second acquisition module is specifically used for:

[0226] The height of the object being detected is calculated using the first and second calculation formulas.

[0227] The first calculation formula is:

[0228]

[0229] The second calculation formula is:

[0230]

[0231] Where H is the distance between the target detection point and the horizon, Z is the distance between the target detection point and the vehicle, y is the height of the target object in the image coordinate system, f is the focal length of the camera, h_px is the height of the 2D bounding box of the target object, H_px is the distance between the target ground point corresponding to the target object in the pixel coordinate system and the horizon, the target ground point is the image of the target detection point in the image, and h is the height of the detected object in the world coordinate system.

[0232] In some embodiments, when the target object is classified as a vehicle, the 2D bounding box of the target object is a vehicle body 2D bounding box, and the orientation of the target object is represented by a target 2D bounding box, which may be a front 2D bounding box or a rear 2D bounding box.

[0233] The 3D information of the target object also includes at least one of the following: the heading angle of the detected object, the length of the detected object, and the width of the detected object, wherein the target object is the image of the detected object in the image.

[0234] In some embodiments, the 3D information of the target object also includes the heading angle of the detected object.

[0235] The second acquisition module is specifically used for:

[0236] When the second information of the target object includes only a 2D bounding box, the heading angle of the detected object is determined according to the category of the 2D bounding box. Specifically, if the 2D bounding box is a front 2D bounding box, the heading angle of the detected object is determined to be -90 degrees; if the 2D bounding box is a rear 2D bounding box, the heading angle of the detected object is determined to be 90 degrees.

[0237] If the second information of the target object includes the vehicle body 2D bounding box, the target 2D bounding box, the front wheel contact point and the rear wheel contact point, or if the second information of the target object only includes the vehicle body 2D bounding box, the front wheel contact point and the rear wheel contact point, the heading angle of the detected object is determined based on the front wheel contact point and the rear wheel contact point.

[0238] In some embodiments, the second acquisition module is specifically used for:

[0239] The front wheel contact point is transformed to the world coordinate system to obtain the second detection point.

[0240] The rear wheel contact point is converted to the world coordinate system to obtain the third detection point.

[0241] The angle between the direction from the third detection point to the second detection point and the horizontal direction is determined as the heading angle of the object being detected.

[0242] In some embodiments, the 3D information of the target object also includes the length of the detected object.

[0243] The second acquisition module is specifically used for:

[0244] If the second information of the target object includes only a 2D bounding box, the reference length is determined to be the length of the detected object.

[0245] When the second information of the target object includes the vehicle body 2D bounding box, the target 2D bounding box, the front wheel contact point, and the rear wheel contact point, the first length is transformed to the world coordinate system and determined as the length of the detected object, wherein the first length is the length of the lower boundary of the vehicle body 2D bounding box.

[0246] When the second information of the target object only includes the 2D bounding box of the vehicle body, the ground contact point of the front wheel, and the ground contact point of the rear wheel, the length of the 2D bounding box of the vehicle body transformed to the world coordinate system is determined as the length of the detected object.

[0247] In some embodiments, the 3D information of the target object further includes the width of the detected object.

[0248] The second acquisition module is specifically used for:

[0249] If the second information of the target object includes only a 2D bounding box, the width of the 2D bounding box is transformed to the width in the world coordinate system and determined as the width of the detected object.

[0250] When the second information of the target object includes the vehicle body 2D bounding box, the target 2D bounding box, the front wheel contact point, and the rear wheel contact point, the width of the detected object is calculated based on the heading angle and the second length of the detected object. The second length is the length transformed to the world coordinate system by the third length, which is the length between the target wheel contact point and the target vertex of the target 2D bounding box. The target vertex is the vertex in the target 2D bounding box that is closest to the target wheel contact point. If the target 2D bounding box is the front 2D bounding box, the target wheel contact point is the front wheel contact point; if the target 2D bounding box is the rear 2D bounding box, the target wheel contact point is the rear wheel contact point.

[0251] If the second information of the target object only includes the 2D bounding box of the vehicle body, the front wheel contact point, and the rear wheel contact point, the reference width is determined as the width of the detected object.

[0252] In some embodiments, the apparatus further includes:

[0253] The module is used to establish the correspondence between the first 3D information and the identification information of the target object.

[0254] The third acquisition module is used to acquire second 3D information corresponding to the identification information, wherein the second 3D information is acquired based on a second image, and the second image is the image preceding the first image captured by the camera.

[0255] The calculation module is used to calculate the dynamic information of the detected object based on the first 3D information and the second 3D information, wherein the target object is the image of the detected object in the image.

[0256] The 3D target detection device provided in this application embodiment can achieve... Figure 1 To avoid repetition, the various processes in the method embodiments will not be described again here.

[0257] Figure 11 The diagram shows the hardware structure of 3D target detection provided in an embodiment of this application.

[0258] The 3D target detection device may include a processor 1101 and a memory 1102 storing computer program instructions.

[0259] Specifically, the processor 1101 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0260] Memory 1102 may include mass storage for data or instructions. For example, and not limitingly, memory 1102 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 1102 may include removable or non-removable (or fixed) media. Where appropriate, memory 1102 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 1102 is non-volatile solid-state memory.

[0261] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the methods according to one aspect of this disclosure.

[0262] The processor 1101 implements any of the 3D target detection methods described in the above embodiments by reading and executing computer program instructions stored in the memory 1102.

[0263] In one example, the 3D target detection device may further include a communication interface 11011 and a bus 1110. For example, Figure 11 As shown, the processor 1101, memory 1102, and communication interface 11011 are connected through bus 1110 and complete communication with each other.

[0264] The communication interface 11011 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0265] Bus 1110 includes hardware, software, or both, that couples components of a 3D object detection device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 1110 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0266] Furthermore, in conjunction with the 3D target detection methods in the above embodiments, this application embodiment can provide a computer storage medium for implementation. This computer storage medium stores computer program instructions, which, when executed by a processor, implement any of the 3D target detection methods in the above embodiments.

[0267] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0268] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0269] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0270] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0271] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A 3D target detection method, applied to vehicles, characterized in that, The vehicle is equipped with a calibrated camera used to photograph other vehicles, and the method includes: The first image captured by the camera is input into the trained first model to obtain the first information of the target object in the first image. This first information includes: a heatmap representing the category of the target object; a heatmap representing the orientation of the target object; a heatmap representing the category of the grounding point of the target object; the length and width of the target object; the offset of the center point of the target object; the offset of the edge line of the target object from the center point of the target object; the offset of the grounding point of the target object; and the offset of the grounding point of the target object from the center point of the target object. The edge line is a boundary line in the target 2D bounding box that does not overlap with the vehicle body 2D bounding box. The target 2D bounding box is the bounding box of the entire vehicle. The vehicle body 2D bounding box is the bounding box of the vehicle's side. The target 2D bounding box is further divided into a front 2D bounding box or a rear 2D bounding box depending on whether the front or rear of the vehicle is facing forward. Decoding the first information yields the second information of the target object. The second information includes the category of the target object and at least one of the following: the 2D bounding box of the target object, the orientation of the target object, and the grounding point of the target object. The category of the grounding point of the target object includes at least one of front wheel grounding point and rear wheel grounding point. The 2D bounding box of the target object is either the vehicle body 2D bounding box or the target 2D bounding box. Based on the second information of the target object, the 3D information of the target object is obtained, wherein the 3D information of the target object includes the width of the detected object, the target object is the image of the detected object in the image, and the detected object is a vehicle captured by the camera. If the second information of the target object includes only a 2D bounding box of the target object, the width of the 2D bounding box is transformed to the width in the world coordinate system and determined as the width of the detected object. When the second information of the target object includes the vehicle body 2D bounding box, the target 2D bounding box, the front wheel contact point, and the rear wheel contact point, the width of the detected object is calculated based on the heading angle and the second length of the detected object. The second length is the length transformed to the world coordinate system by the third length, which is the length between the target wheel contact point and the target vertex of the target 2D bounding box. The target vertex is the vertex in the target 2D bounding box that is closest to the target wheel contact point. If the target 2D bounding box is the front 2D bounding box, the target wheel contact point is the front wheel contact point; if the target 2D bounding box is the rear 2D bounding box, the target wheel contact point is the rear wheel contact point. If the second information of the target object only includes the 2D bounding box of the vehicle body, the front wheel contact point, and the rear wheel contact point, the reference width is determined as the width of the detected object.

2. The method according to claim 1, characterized in that, Before inputting the first image captured by the camera into the trained first model to obtain the first information, the method further includes: Based on the CenterNet network, a first model is constructed, comprising eight network output heads. The outputs of these eight output heads are: a heatmap representing the category of the target object, a heatmap representing the orientation of the target object, a heatmap representing the category of the target object's grounding point, the length and width of the target object, the offset of the target object's center point, the offset of the target object's edge from its center point, the offset of the target object's grounding point, and the offset of the target object's grounding point from its center point. The target objects in the training sample images are labeled. The labeled information includes: the category of the target object, the 2D bounding box of the target object, the orientation of the target object, and the ground point of the target object. The first model is trained using the labeled training sample images until it converges, thus obtaining the trained first model.

3. The method according to claim 1, characterized in that, The 3D information of the target object includes the distance between the detected object and the vehicle on which the camera is located, wherein the target object is the image of the detected object in the image. The step of obtaining the 3D information of the target object based on the second information of the target object includes: Calculate the pixel coordinates of the target ground point corresponding to the target object and transform them to the first coordinate in the world coordinate system. Based on the first coordinates and the camera calibration parameters, the distance between the target detection point and the vehicle equipped with the camera is calculated to obtain the distance between the detected object and the vehicle equipped with the camera. The target grounding point is the image of the target detection point in the image. When the second information of the target object includes the 2D bounding box of the target object, the target grounding point is any point on the lower boundary of the 2D bounding box of the target object.

4. The method according to claim 3, characterized in that, When the target object is classified as a vehicle, the 2D bounding box of the target object is the vehicle body 2D bounding box. The orientation of the target object is represented by the target 2D bounding box, which is either the front 2D bounding box or the rear 2D bounding box. Before calculating the pixel coordinates of the target ground point corresponding to the target object and transforming them to the first coordinate of the world coordinate system, the method further includes: The target grounding point is determined based on the second information of the target object.

5. The method according to claim 4, characterized in that, Determining the target grounding point based on the second information of the target object includes: If the second information of the target object includes only a 2D bounding box, the midpoint of the lower boundary of the 2D bounding box is determined as the target grounding point. When the second information of the target object includes the vehicle body 2D bounding box, the target 2D bounding box, the front wheel contact point, and the rear wheel contact point, the intersection of the extension of the first connecting line and the extension of the edge line is determined as the target contact point. The first connecting line is the line connecting the front wheel contact point and the rear wheel contact point, and the edge line is the boundary line in the target 2D bounding box that does not overlap with the vehicle body 2D bounding box. If the second information of the target object only includes the 2D bounding box of the vehicle body, the front wheel contact point and the rear wheel contact point, the midpoint between the front wheel contact point and the rear wheel contact point is determined as the target contact point.

6. The method according to claim 1, characterized in that, The 3D information of the target object also includes the height of the detected object, wherein the target object is the image of the detected object in the image. The step of obtaining the 3D information of the target object based on the second information of the target object includes: The height of the object being detected is calculated using the first and second calculation formulas. The first calculation formula is: The second calculation formula is: Where H is the distance between the target detection point and the horizon, Z is the distance between the target detection point and the vehicle where the camera is located, y is the height of the target object in the image coordinate system, and f is the focal length of the camera. It is the height of the 2D bounding box of the target object. is the distance between the target ground point corresponding to the target object and the horizon in the pixel coordinate system, the target ground point is the image of the target detection point in the image, and h is the height of the detected object in the world coordinate system.

7. The method according to any one of claims 1 to 6, characterized in that, When the target object is classified as a vehicle, the 2D bounding box of the target object is the vehicle body 2D bounding box. The orientation of the target object is represented by the target 2D bounding box, which is either the front 2D bounding box or the rear 2D bounding box. The 3D information of the target object also includes at least one of the following: the heading angle of the detected object, the length of the detected object, and the width of the detected object, wherein the target object is the image of the detected object in the image.

8. The method according to claim 7, characterized in that, The 3D information of the target object also includes the heading angle of the detected object. The step of obtaining the 3D information of the target object based on the second information of the target object includes: When the second information of the target object includes only a 2D bounding box, the heading angle of the detected object is determined according to the category of the 2D bounding box. Specifically, if the 2D bounding box is a front 2D bounding box, the heading angle of the detected object is determined to be -90 degrees; if the 2D bounding box is a rear 2D bounding box, the heading angle of the detected object is determined to be 90 degrees. If the second information of the target object includes the vehicle body 2D bounding box, the target 2D bounding box, the front wheel contact point and the rear wheel contact point, or if the second information of the target object only includes the vehicle body 2D bounding box, the front wheel contact point and the rear wheel contact point, the heading angle of the detected object is determined based on the front wheel contact point and the rear wheel contact point.

9. The method according to claim 8, characterized in that, Determining the heading angle of the object being detected based on the contact points of the front and rear wheels includes: The front wheel contact point is transformed to the world coordinate system to obtain the second detection point. The rear wheel contact point is converted to the world coordinate system to obtain the third detection point. The angle between the direction from the third detection point to the second detection point and the horizontal direction is determined as the heading angle of the object being detected.

10. The method according to claim 7, characterized in that, The 3D information of the target object also includes the length of the detected object. The step of obtaining the 3D information of the target object based on the second information of the target object includes: If the second information of the target object includes only a 2D bounding box of the target object, the reference length is determined to be the length of the detected object. When the second information of the target object includes the vehicle body 2D bounding box, the target 2D bounding box, the front wheel contact point, and the rear wheel contact point, the first length is transformed to the world coordinate system and determined as the length of the detected object, wherein the first length is the length of the lower boundary of the vehicle body 2D bounding box. When the second information of the target object only includes the 2D bounding box of the vehicle body, the ground contact point of the front wheel, and the ground contact point of the rear wheel, the length of the 2D bounding box of the vehicle body transformed to the world coordinate system is determined as the length of the detected object.

11. The method according to claim 1, characterized in that, After obtaining the 3D information of the target object based on the second information of the target object, the method further includes: Establish a correspondence between the 3D information and the identification information of the target object. Obtain the second 3D information corresponding to the identification information, wherein the second 3D information is obtained based on a second image, which is the image preceding the first image captured by the camera. Based on the 3D information and the second 3D information, the dynamic information of the detected object is calculated, wherein the target object is the image of the detected object in the image.

12. A 3D target detection device, applied to vehicles, characterized in that, The vehicle is equipped with a calibrated camera for photographing other vehicles, and the device includes: The first acquisition module is used to input the first image captured by the camera into the trained first model to obtain first information about the target object in the first image. The first information includes: a heatmap representing the category of the target object; a heatmap representing the orientation of the target object; a heatmap representing the category of the grounding point of the target object; the length and width of the target object; the offset of the center point of the target object; the offset of the edge line of the target object from the center point of the target object; the offset of the grounding point of the target object; and the offset of the grounding point of the target object from the center point of the target object. The edge line is a boundary line in the target 2D bounding box that does not overlap with the vehicle body 2D bounding box. The target 2D bounding box is the bounding box of the entire vehicle. The vehicle body 2D bounding box is the bounding box of the side of the vehicle, categorized as a front 2D bounding box or a rear 2D bounding box based on whether the front or rear of the vehicle is ahead. A decoding module is used to decode the first information to obtain second information about the target object. The second information includes the category of the target object and at least one of the following: the 2D bounding box of the target object, the orientation of the target object, and the grounding point of the target object. The category of the grounding point of the target object includes at least one of front wheel grounding point and rear wheel grounding point. The 2D bounding box of the target object is either the vehicle body 2D bounding box or the target 2D bounding box. The second acquisition module is used to obtain the 3D information of the target object based on the second information of the target object, wherein the 3D information of the target object includes the width of the detected object, the target object is the image of the detected object in the image, and the detected object is a vehicle captured by the camera. If the second information of the target object includes only a 2D bounding box of the target object, the width of the 2D bounding box is transformed to the width in the world coordinate system and determined as the width of the detected object. When the second information of the target object includes the vehicle body 2D bounding box, the target 2D bounding box, the front wheel contact point, and the rear wheel contact point, the width of the detected object is calculated based on the heading angle and the second length of the detected object. The second length is the length transformed to the world coordinate system by the third length, which is the length between the target wheel contact point and the target vertex of the target 2D bounding box. The target vertex is the vertex in the target 2D bounding box that is closest to the target wheel contact point. If the target 2D bounding box is the front 2D bounding box, the target wheel contact point is the front wheel contact point; if the target 2D bounding box is the rear 2D bounding box, the target wheel contact point is the rear wheel contact point. If the second information of the target object only includes the 2D bounding box of the vehicle body, the front wheel contact point, and the rear wheel contact point, the reference width is determined as the width of the detected object.

13. A 3D target detection device, characterized in that, The device includes a processor and a memory storing computer program instructions, wherein the processor, when executing the computer program instructions, implements the 3D target detection method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processor, implement the 3D target detection method as described in any one of claims 1 to 11.

15. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device causes the electronic device to perform the 3D target detection method as described in any one of claims 1 to 11.

Images