Target detection methods and apparatus, electronic devices and storage media

By correcting the initial pose information through a two-stage processing method, the problem of low accuracy in 3D target detection in existing technologies is solved, higher accuracy object pose estimation is achieved, and the safety of autonomous driving is improved.

CN116343153BActive Publication Date: 2026-11-17BEIJING JINGDONG QIANSHITECHNOLOGY CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202310318803.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-03-29
Publication Date
2026-11-17
Estimated Expiration
2043-03-29

AI Technical Summary

Technical Problem

Existing 3D object detection technologies have low accuracy in the field of autonomous driving and cannot accurately estimate the pose of target objects.

Method used

A two-stage processing method is adopted. First, the initial pose information is extracted from the two-dimensional image through visual 3D target detection technology. Then, the initial pose information is corrected through a correction network, including adding position label information and iterative correction, until the preset deviation threshold is reached.

Benefits of technology

It improves the accuracy of object information detection, enhances the safety of autonomous driving in unmanned vehicles, and obtains more accurate target object pose information through multiple iterative corrections.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116343153B_ABST
    Figure CN116343153B_ABST
Patent Text Reader

Abstract

This disclosure provides a target detection method, apparatus, electronic device, and storage medium, which can be applied to fields such as intelligent driving, autonomous driving, and environmental perception. The target detection method includes: extracting initial pose information of a target object from a scene image to be processed; adding positional annotation information to the target object in the scene image to be processed based on the initial pose information to obtain an annotated scene image; inputting the annotated scene image into a correction network, and outputting the pose deviation represented by the positional annotation information associated with the initial pose information in the annotated scene image; and correcting the initial pose information based on the pose deviation to obtain the target pose information of the target object.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure relates to the technical fields of intelligent driving, autonomous driving, and environmental perception, and specifically to a target detection method, apparatus, device, medium, and program product. Background Technology

[0002] In the field of autonomous driving, 3D object detection of images is an essential task. By mining features from the input image signal, the pose information of the target object can be obtained, enabling the perception of 3D information of the scene. In realizing the concept disclosed herein, the inventors discovered at least the following problems in related technologies: current 3D object detection technologies often have low accuracy and cannot accurately estimate the pose of the target object. Summary of the Invention

[0003] In view of the above problems, this disclosure provides a target detection method, apparatus, device, medium and program product.

[0004] One aspect of this disclosure provides a target detection method, comprising:

[0005] The initial pose information of the target object is extracted from the scene image to be processed, wherein the scene image to be processed is used to characterize the surrounding driving environment of the target vehicle during driving, and the surrounding driving environment of the target vehicle includes at least one object.

[0006] Based on the initial pose information, the labeled scene image is obtained by adding position annotation information to the target objects in the scene image to be processed.

[0007] The labeled scene image is input into the correction network, and the output is the pose deviation represented by the position annotation information in the labeled scene image associated with the initial pose information;

[0008] The initial pose information is corrected based on the pose deviation to obtain the target pose information of the target object.

[0009] According to embodiments of this disclosure, the process of adding position annotation information to target objects in a scene image to be processed based on initial pose information to obtain an annotated scene image includes:

[0010] Based on the initial pose information, the bounding box of the target object is drawn in the scene image to be processed, and the annotated scene image is obtained.

[0011] According to embodiments of this disclosure, drawing the bounding box of the target object in the scene image to be processed based on initial pose information to obtain an annotated scene image includes:

[0012] Based on the initial pose information, the first position coordinates of multiple corner points of the bounding box of the target object in the camera coordinate system are calculated;

[0013] The first position coordinates of multiple corner points are transformed to obtain the second position coordinates of multiple corner points of the bounding box of the target object in the image coordinate system.

[0014] Based on the second position coordinates of multiple corner points, the bounding box of the target object is drawn in the scene image to be processed, thus obtaining the labeled scene image.

[0015] According to embodiments of this disclosure, wherein:

[0016] The initial pose information includes: the initial position coordinates of the center point of the target object, the initial three-dimensional dimensions of the target object, and the initial orientation angle of the target object. The initial orientation angle is used to characterize the angle between the first driving direction of the target object and the second driving direction of the target vehicle.

[0017] According to embodiments of this disclosure, wherein:

[0018] Positional deviations include: deviation of the center point of the target object, deviation of the three-dimensional dimensions of the target object, and deviation of the orientation angle of the target object;

[0019] Target pose information includes: the target position coordinates of the center point of the target object, the target three-dimensional dimensions of the target object, and the target orientation angle of the target object;

[0020] Based on the pose deviation, the initial pose information is corrected to obtain the target pose information of the target object, including:

[0021] The target position coordinates are calculated based on the initial position coordinates and the position deviation.

[0022] The target three-dimensional dimensions are calculated based on the initial three-dimensional dimensions and the three-dimensional dimension deviations.

[0023] The target direction angle is calculated based on the initial direction angle and the direction angle deviation.

[0024] According to embodiments of this disclosure, extracting the initial pose information of the target object from the scene image to be processed includes:

[0025] The scene image to be processed is input into the target detection network, which outputs the initial pose information of the target object.

[0026] According to embodiments of this disclosure, after obtaining the target pose information of the target object:

[0027] The target pose information is iteratively corrected multiple times until a preset termination condition is met, resulting in the final pose information.

[0028] According to embodiments of this disclosure, the process of iteratively correcting the target pose information multiple times until a preset termination condition is met, to obtain the final determined pose information, includes:

[0029] The target pose information is corrected in the i-th iteration to obtain the pose information in the i-th iteration.

[0030] Determine the pose deviation of the i-th iteration corresponding to the pose information of the i-th iteration;

[0031] Iterative execution: Based on the pose deviation of the i-th iteration, the pose information of the i-th iteration is corrected in the (i+1)-th iteration to obtain the pose information of the (i+1)-th iteration, and the operation of determining the pose deviation of the (i+1)-th iteration corresponding to the pose information of the (i+1)-th iteration is continued until the pose deviation of the iteration is smaller than the preset deviation threshold, and the final pose information is obtained.

[0032] Another aspect of this disclosure provides a target detection apparatus, comprising:

[0033] An extraction module is used to extract the initial pose information of a target object from a scene image to be processed, wherein the scene image to be processed is used to characterize the surrounding driving environment of the target vehicle during driving, and the surrounding driving environment of the target vehicle includes at least one object.

[0034] The annotation module is used to add position annotation information to target objects in the scene image to be processed based on the initial pose information, so as to obtain an annotated scene image;

[0035] The correction module is used to input the labeled scene image into the correction network and output the pose deviation represented by the position annotation information associated with the initial pose information in the labeled scene image;

[0036] The correction module is used to correct the initial pose information based on the pose deviation to obtain the target pose information of the target object.

[0037] Another aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the target detection method described above.

[0038] Another aspect of this disclosure provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the target detection method described above.

[0039] Another aspect of this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the above-described target detection method.

[0040] The target detection method according to embodiments of this disclosure employs a two-stage processing concept to detect the pose information of objects. In the first stage, vision-based 3D target detection technology is used to mine features from the image, outputting the initial three-dimensional pose information of the target object. In the second stage, the initial three-dimensional pose information is further corrected. The scene image to be processed is a two-dimensional image. Since two-dimensional images lack three-dimensional geometric information (especially depth information), directly using visual detection technology to estimate the three-dimensional information of objects in the first stage often fails to accurately estimate the pose of the target object, resulting in inaccurate detection results. Therefore, embodiments of this disclosure further correct the initially extracted, low-accuracy, three-dimensional pose information of the object to obtain more accurate detection results. It can be seen that through the above two-stage image processing strategy, the low-quality detection results from the first stage are reprojected onto a two-dimensional image and visualized. The labeled image is then automatically evaluated and corrected by the correction model in the second stage. The fine-tuned pose information after correction is more accurate and has higher precision, improving the accuracy of object information detection. Furthermore, in the process of autonomous driving vehicles achieving autonomous driving based on more accurate object information, it has higher autonomous driving safety. Attached Figure Description

[0041] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0042] Figure 1 The illustration shows an application scenario of the target detection method, apparatus, device, medium, and program product according to embodiments of the present disclosure;

[0043] Figure 2 A flowchart illustrating a target detection method according to an embodiment of the present disclosure is shown schematically.

[0044] Figure 3 An exemplary schematic diagram of the processing principle of a target detection method according to an embodiment of the present disclosure is shown;

[0045] Figure 4 This schematically illustrates a flowchart of processing a scene image to be processed to obtain an annotated scene image according to an embodiment of the present disclosure;

[0046] Figure 5 A schematic block diagram of a target detection apparatus according to an embodiment of the present disclosure is shown.

[0047] Figure 6 A block diagram schematically illustrates an electronic device suitable for implementing a target detection method according to an embodiment of the present disclosure. Detailed Implementation

[0048] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0049] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0050] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0051] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0052] In the embodiments disclosed herein, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of data (e.g., including but not limited to user personal information) comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures have been taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0053] In the embodiments disclosed herein, user authorization or consent is obtained before acquiring or collecting user personal information.

[0054] Embodiments of this disclosure provide a target detection method, including:

[0055] The process involves extracting initial pose information of the target object from a scene image to be processed, wherein the scene image represents the surrounding driving environment of the target vehicle during its driving process, and the surrounding driving environment of the target vehicle includes at least one object; based on the initial pose information, position annotation information is added to the target object in the scene image to obtain an annotated scene image; the annotated scene image is input into a correction network, and the output network shows the pose deviation represented by the position annotation information associated with the initial pose information in the annotated scene image; the initial pose information is corrected based on the pose deviation to obtain the target pose information of the target object.

[0056] Figure 1 The illustration shows an application scenario of the target detection method, apparatus, device, medium, and program product according to embodiments of the present disclosure.

[0057] like Figure 1 As shown, application scenario 100 according to this embodiment may include vehicle 101 and server 102. Vehicle 101 and server 102 can communicate via a network, which may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0058] Vehicle 101 is equipped with an autonomous driving module to guide the vehicle to achieve autonomous driving. Vehicle 101 is also equipped with image acquisition devices, such as cameras, to acquire images of the surrounding environment during the vehicle's operation. Source radar may also be installed to acquire point cloud data of the surrounding environment.

[0059] Server 102 can be a server that provides various services, such as a backend management server, which can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0060] In the application scenario of this disclosure embodiment, during the driving process, the vehicle 101 acquires scene images of the surrounding environment through the image acquisition device installed therein, and sends the scene images to the server 102. After receiving the scene images, the server 102 can process the scene images based on the image data processing request. For example, by executing the target detection method of this disclosure embodiment, the initial pose information of the target object is extracted from the scene image to be processed; based on the initial pose information, position annotation information is added to the target object in the scene image to be processed to obtain an annotated scene image; the annotated scene image is input into a correction network, and the pose deviation shown by the position annotation information associated with the initial pose information in the annotated scene image is output; and the initial pose information is corrected based on the pose deviation to obtain the target pose information of the target object and return it to the autonomous driving module in the vehicle 101, so that the autonomous driving module in the vehicle 101 can assist the vehicle 101 in achieving autonomous driving based on the identified object pose information in the surrounding environment.

[0061] The following will be based on Figure 1 The described scene, through Figures 2-6 The target detection method of the disclosed embodiments will be described in detail.

[0062] Figure 2 A flowchart illustrating a target detection method according to an embodiment of the present disclosure is shown schematically.

[0063] like Figure 2 As shown, the target detection method in this embodiment includes operations S201 to S204.

[0064] In operation S201, the initial pose information of the target object is extracted from the scene image to be processed, wherein the scene image to be processed is used to characterize the surrounding driving environment of the target vehicle during driving, and the surrounding driving environment of the target vehicle includes at least one object.

[0065] In operation S202, based on the initial pose information, position annotation information is added to the target objects in the scene image to be processed to obtain an annotated scene image;

[0066] In operation S203, the labeled scene image is input into the correction network, and the pose deviation shown by the position annotation information associated with the initial pose information in the labeled scene image is output.

[0067] In operation S204, the initial pose information is corrected based on the pose deviation to obtain the target pose information of the target object.

[0068] According to embodiments of this disclosure, the target vehicle can be an autonomous vehicle of current interest. During operation, the autonomous vehicle needs to obtain the pose information of objects (such as obstacles) in the surrounding environment. By combining the pose information of the objects with other sensory information (such as traffic light status, road indication information, etc.), the autonomous vehicle can achieve autonomous driving.

[0069] An image acquisition device, such as a camera, can be installed in the target vehicle to acquire images of the surrounding driving environment during vehicle operation. The method described in this embodiment is used to detect objects in the driving environment image to obtain the object's pose information. It should be noted that the method described in this embodiment is a method based on processing a single-frame scene image. That is, the scene image to be processed is a single-frame scene image corresponding to a certain moment, acquired by the image acquisition device.

[0070] According to embodiments of this disclosure, the target detection method described above employs a two-stage processing concept for detecting the pose information of obstacles.

[0071] Specifically, in the first stage, by performing the above-mentioned operation S201, preliminary three-dimensional pose information is extracted from the two-dimensional image, resulting in detection results with low accuracy. In the above-mentioned operation S201, the initial pose information of the target object is extracted from the scene image to be processed. For example, the initial pose information of the target object can be output using an object detection network. Alternatively, an image detection algorithm can be used to process the scene image to be processed to obtain the initial pose information of the target object (such as three-dimensional information such as object position, size, orientation, tilt angle, etc.).

[0072] Furthermore, in the second stage, by performing the above operations S202 to S204, the initially extracted three-dimensional pose information of the object, which has low accuracy, is further processed to obtain more accurate detection results.

[0073] Specifically, firstly, through the above operation S202, position annotation information is added to the target object in the scene image to be processed to obtain an annotated scene image. For example, it can be based on the three-dimensional pose information result with low accuracy obtained in the first stage, and position annotation boxes are added to the target object in the scene image to be processed to obtain an annotated scene image.

[0074] Secondly, by performing the above operation S203, the pose deviations of the position annotation information associated with the initial pose information in the network output annotation scene image are corrected, such as obtaining the pose deviations of the object in multiple dimensions, including but not limited to position deviation, size deviation, and orientation tilt angle deviation.

[0075] Then, by executing the above operation S204, the initial pose information can be corrected based on the pose deviation to obtain more accurate object pose information. After obtaining the target pose information of the target object, it can be returned to the autonomous driving module in the target vehicle, so that the autonomous driving module in the vehicle can achieve autonomous driving of the unmanned vehicle by combining the object's pose information with other perception information (such as traffic light status, road indication information, etc.).

[0076] According to embodiments of this disclosure, the scene image to be processed is a two-dimensional image. Since two-dimensional images lack three-dimensional geometric information (especially depth information) of the scene, in the first stage (operation S201 above), features are mined from the image using vision-based 3D object detection technology to output the three-dimensional pose information of the target object, thus achieving perception of the scene's 3D information. Directly using visual detection technology to estimate the three-dimensional information of an object often fails to accurately estimate the pose of the target object, resulting in inaccurate detection results. Therefore, embodiments of this disclosure further refine the initially extracted, low-accuracy three-dimensional pose information of the object through operations S202 to S204 above, yielding more accurate detection results. It is evident that through the above two-stage image processing strategy, the low-quality detection results from the first stage are reprojected onto a two-dimensional image and visualized. The annotated image then undergoes automatic evaluation and correction of the previous results through a second-stage correction model. The fine-tuned pose information is more accurate and has higher precision, improving the accuracy of object information detection. Furthermore, in the process of autonomous driving based on more accurate object information, autonomous driving vehicles exhibit higher autonomous driving safety.

[0077] According to embodiments of this disclosure, specifically, in the first stage, extracting the initial pose information of the target object from the scene image to be processed includes: inputting the scene image to be processed into a target detection network and outputting the initial pose information of the target object. The target detection network can be a YOLO network. The YOLO network directly predicts the object category and position through a convolutional neural network, which is relatively fast in processing speed, can meet real-time requirements, satisfies the information latency requirements in autonomous driving scenarios, and has fewer instances of mistaking image background for objects, as well as strong generalization ability.

[0078] According to embodiments of this disclosure, the pose information of an object may specifically include: the position coordinates of the center point of the target object, the three-dimensional dimensions of the target object (e.g., the length, width, and height of the object's bounding box), and the orientation angle of the target object, wherein the orientation angle is used to characterize the angle between the first driving direction of the target object and the second driving direction of the target vehicle.

[0079] Based on this, in the first stage, preliminary 3D pose information feature extraction is performed on the 2D image to obtain the initial pose information of the target object, including: the initial position coordinates of the target object's center point, the initial 3D dimensions of the target object, and the initial orientation angle of the target object. In the second stage, further correction processing is performed on the initially extracted, less accurate, 3D pose information of the object to obtain the target pose information, including: the target position coordinates of the target object's center point, the target 3D dimensions of the target object, and the target orientation angle of the target object.

[0080] The pose deviations exhibited by the position annotation information associated with the initial pose information in the annotated scene image output by the correction network can include: the position deviation of the center point of the target object, the three-dimensional size deviation of the target object, and the orientation angle deviation of the target object.

[0081] Furthermore, based on the aforementioned multidimensional information types, and by correcting the initial pose information based on pose deviation, the target pose information of the target object is obtained, including:

[0082] The target position coordinates are calculated based on the initial position coordinates and position deviation; the target three-dimensional dimensions are calculated based on the initial three-dimensional dimensions and three-dimensional dimension deviation; and the target direction angle is calculated based on the initial direction angle and direction angle deviation.

[0083] Figure 3 An exemplary schematic diagram of the processing principle of a target detection method according to an embodiment of the present disclosure is shown. The following is in conjunction with... Figure 3 The above-described methods of the embodiments of this disclosure will be described by way of example.

[0084] like Figure 3 As shown, in the first stage, the scene image to be processed is input into the target detection network, which outputs the initial pose information of the target object (initial coordinates of the target object's center point, initial 3D dimensions of the target object, and initial orientation angle of the target object). The target object can be multiple obstacles of various predetermined types, such as pedestrians, motor vehicles, tricycles, bicycles, etc. The initial pose information includes, for example: the initial coordinates of the center point of object 1 (L1, Location1), the initial 3D dimensions of the target object (D1, Dimension1, such as the length, width, and height of the bounding box of object 1), and the initial orientation angle of the target object (Y1, Yaw1); the initial pose information of object 2 (L2, D2, Y2); the initial pose information of object 3 (L3, D3, Y3)... These three sets of pose parameters can completely describe the pose information of a rigid target object in the 3D scene.

[0085] like Figure 3As shown, because the 2D input image lacks scene depth information, the target object pose output by the object detection network is an inaccurate and low-quality detection result. Therefore, in the second stage, firstly, based on the initial pose information, the low-quality detection result is reprojected onto the 2D image and visualized. That is, positional annotation information associated with the initial pose information is added to the target object in the scene image to be processed, resulting in an annotated scene image, such as... Figure 3 As shown, for example, a labeled scene image can be obtained by adding location bounding boxes to the target objects in the scene image to be processed.

[0086] Then, by correcting the pose deviation exhibited by the position annotation information associated with the initial pose information in the network output annotated scene image, such as... Figure 3 As shown, the output results include, for example, the position deviation of the center point of object 1 (ΔL1), the three-dimensional size deviation of object 1 (ΔD1), the orientation angle deviation of object 1 (ΔY1); the position deviation of object 2 (ΔL2, ΔD2, ΔY2); the position deviation of object 3 (ΔL3, ΔD3, ΔY3)...

[0087] In annotated scene images, if the output of the object detection network is inaccurate, the bounding box of the object will not be able to perfectly enclose the object. The correction network can perceive the quality of the first stage output, and based on the degree of deviation of the bounding box position, the correction network can estimate the positional deviation of the 3D pose.

[0088] Subsequently, the initial pose information is corrected based on the pose deviation to obtain the target pose information of the target object. For example, the initial pose information and the pose deviation are added together to obtain the target pose information. In this way, automatic perception and fine-tuning correction of the detection result quality are achieved, resulting in a more accurate estimation result.

[0089] like Figure 3 As shown, the target position coordinates can be calculated based on the initial position coordinates and position deviation: Ln'=Ln+ΔLn; the target three-dimensional dimensions can be calculated based on the initial three-dimensional dimensions and three-dimensional dimension deviation: Dn'=Dn+ΔDn; the target direction angle can be calculated based on the initial direction angle and direction angle deviation: Yn'=Yn+ΔYn (n=1, 2, 3……).

[0090] According to embodiments of this disclosure, based on initial pose information, low-quality detection results are reprojected onto a 2D image and visualized. Specifically, a labeled scene image is obtained by adding position annotation information associated with the initial pose information to the target object in the scene image to be processed. Alternatively, the labeled scene image can be obtained by drawing the bounding box of the target object in the scene image to be processed based on the initial pose information.

[0091] The following combination Figure 4The method for drawing the bounding box of the target object in the scene image to be processed is further explained.

[0092] Figure 4 The flowchart illustrating the process of processing a scene image to be processed to obtain an annotated scene image according to an embodiment of the present disclosure is shown.

[0093] like Figure 4 As shown, the method for processing the scene image to obtain the labeled scene image includes operations S401 to S403.

[0094] In operation S401, based on the initial pose information, the first position coordinates of multiple corner points of the target object's bounding box in the camera coordinate system are calculated. Given the initial position coordinates L of the object's center point, the initial three-dimensional dimensions D of the object (length, width, and height of the object's bounding box), and the initial orientation angle Y of the target object, the position coordinates of the eight corner points of the object's bounding box can be easily calculated.

[0095] In operation S402, the first position coordinates of multiple corner points are transformed to obtain the second position coordinates of multiple corner points of the outer frame of the target object in the image coordinate system.

[0096] In operation S403, based on the second position coordinates of multiple corner points, the bounding box of the target object is drawn in the scene image to be processed, and the annotated scene image is obtained.

[0097] For example, the first position coordinates of these eight corner points are transformed using the camera parameter matrix to obtain the second position coordinates of these eight corner points in the image coordinate system, and the second position coordinates of the eight corner points are projected onto the 2D image to obtain a visualized labeled scene image of the low-quality detection results.

[0098] The projection equation is shown in equation (1) below:

[0099] (ut, Vt, t, 1) τ =P·(x, y, z, 1) τ ----(1)

[0100] Where x, y, z are the first position coordinates of the eight corner points of the bounding box in the camera coordinate system, and u, v are the second position coordinates of the eight corner points of the bounding box in the image coordinate system. P is the camera parameter matrix, and t is the projection vector parameter.

[0101] According to the embodiments of this disclosure, in the above processing method, the first position coordinates of multiple corner points of the bounding box of the target object in the camera coordinate system are calculated based on the initial pose information. Then, through coordinate transformation, the low-quality detection results are reprojected onto the 2D image using the pre-calibrated camera parameter matrix, thereby realizing the visualization annotation of the preliminary detection results. The resulting annotated image can be used as the basic data of the correction network, which facilitates the subsequent automatic evaluation and correction of the detection results by the correction network to obtain a more accurate pose estimate.

[0102] According to embodiments of this disclosure, after obtaining the target pose information of the target object, the target detection method of this disclosure further includes: performing multiple iterative corrections on the target pose information until a preset termination condition is met, to obtain the final determined pose information.

[0103] Furthermore, the multiple iterative corrections to the target pose information specifically include the following operations:

[0104] The first operation is to perform the i-th iteration correction on the target pose information (initial pose information of the i-th iteration) of the i-1-th iteration based on the pose deviation of the i-th iteration, so as to obtain the target pose information of the i-th iteration, wherein the target pose information of the i-th iteration is used as the initial pose information of the i+1-th iteration.

[0105] The second operation is to determine the pose deviation of the (i+1)th iteration associated with the target pose information of the i-th iteration (the initial pose information of the (i+1)th iteration).

[0106] The third operation is iterative execution: Based on the pose deviation of the (i+1)th iteration, the target pose information of the (i+1)th iteration (initial pose information of the (i+1)th iteration) is corrected for the (i+1)th iteration to obtain the target pose information of the (i+2)th iteration (initial pose information of the (i+2)th iteration), and the pose deviation of the (i+2)th iteration associated with the target pose information of the (i+1)th iteration (initial pose information of the (i+2)th iteration) is determined, until the iterative pose deviation is smaller than the preset deviation threshold, and the final determined pose information is obtained.

[0107] For the specific method of each iteration operation in the above-mentioned iterative correction, please refer to operations S202 to S204 in the foregoing embodiments.

[0108] For example, operations S202 to S204 can be regarded as performing the first iteration. In operation S204, the initial pose information of the first iteration is corrected based on the pose deviation to obtain the target pose information of the target object. The target pose information is used as the pose information of the first iteration.

[0109] Determine the pose deviation for the second iteration associated with the target pose information of the first iteration (initial pose information of the second iteration). Specifically, referring to operations S202 to S203, it can be as follows: Based on the target pose information of the first iteration, add position annotation information to the target object in the scene image to be processed to obtain the annotated scene image 1; input the annotated scene image 1 into the correction network, and output the pose deviation for the second iteration associated with the target pose information of the first iteration (initial pose information of the second iteration).

[0110] Based on the pose deviation of the second iteration, the target pose information of the first iteration (the initial pose information of the second iteration) is corrected in the second iteration to obtain the target pose information of the second iteration.

[0111] Determine the pose deviation of the third iteration associated with the target pose information of the second iteration (initial pose information of the third iteration).

[0112] Based on the pose deviation of the third iteration, the target pose information of the second iteration (the initial pose information of the third iteration) is corrected in the third iteration to obtain the target pose information of the third iteration.

[0113] The pose deviation of the fourth iteration, which is associated with the target pose information of the third iteration (the initial pose information of the fourth iteration), is determined... This process is repeated iteratively until the pose deviation of a certain iteration is smaller than a preset deviation threshold, at which point the final pose information is obtained.

[0114] According to embodiments of this disclosure, by performing multiple iterative corrections on the target pose information as described above, more accurate detection results can be obtained.

[0115] Based on the above-described target detection method, this disclosure also provides a target detection device. The following will be combined with... Figure 5 The device is described in detail.

[0116] Figure 5 A schematic block diagram of a target detection apparatus according to an embodiment of the present disclosure is shown.

[0117] like Figure 5 As shown, the target detection device 500 of this embodiment includes an extraction module 501, a labeling module 502, a correction module 503, and a correction module 504.

[0118] The extraction module 501 is used to extract the initial pose information of the target object from the scene image to be processed. The scene image to be processed is used to characterize the surrounding driving environment of the target vehicle during driving. The surrounding driving environment of the target vehicle includes at least one object.

[0119] The annotation module 502 is used to add position annotation information to the target objects in the scene image to be processed based on the initial pose information to obtain an annotated scene image.

[0120] The correction module 503 is used to input the labeled scene image into the correction network and output the pose deviation represented by the position annotation information associated with the initial pose information in the labeled scene image.

[0121] The correction module 504 is used to correct the initial pose information based on the pose deviation to obtain the target pose information of the target object.

[0122] According to embodiments of this disclosure, in the above-described apparatus, the extraction module 501 extracts features from the image using vision-based 3D target detection technology in the first stage, outputting the initial three-dimensional pose information of the target object. The annotation module 502, correction module 503, and refinement module 504 further refine the initial three-dimensional pose information in the second stage. Since the scene image to be processed is a two-dimensional image, and two-dimensional images lack three-dimensional geometric information (especially depth information), directly using vision detection technology to estimate the three-dimensional information of the object in the first stage by the extraction module 501 often fails to accurately estimate the pose of the target object, resulting in inaccurate detection results. Therefore, embodiments of this disclosure further refine the initially extracted, inaccurate three-dimensional pose information of the object using the annotation module 502, correction module 503, and refinement module 504, achieving more accurate detection results with higher precision. This improves the accuracy of object information detection and, consequently, enhances the safety of autonomous driving in the process of achieving autonomous driving based on more accurate object information.

[0123] According to an embodiment of this disclosure, the annotation module 502 includes a drawing unit for drawing the bounding box of the target object in the scene image to be processed based on initial pose information, thereby obtaining an annotated scene image.

[0124] According to embodiments of this disclosure, the drawing unit includes a calculation subunit, a transformation subunit, and a drawing subunit.

[0125] The calculation subunit is used to calculate the first position coordinates of multiple corner points of the bounding box of the target object in the camera coordinate system based on the initial pose information; the transformation subunit is used to transform the first position coordinates of the multiple corner points to obtain the second position coordinates of the multiple corner points of the bounding box of the target object in the image coordinate system; and the drawing subunit is used to draw the bounding box of the target object in the scene image to be processed based on the second position coordinates of the multiple corner points to obtain the labeled scene image.

[0126] According to an embodiment of this disclosure, the initial pose information includes: the initial position coordinates of the center point of the target object, the initial three-dimensional dimensions of the target object, and the initial orientation angle of the target object, wherein the initial orientation angle is used to characterize the angle between the first driving direction of the target object and the second driving direction of the target vehicle.

[0127] According to embodiments of this disclosure, the pose deviation includes: the position deviation of the center point of the target object, the three-dimensional size deviation of the target object, and the orientation angle deviation of the target object; the target pose information includes: the target position coordinates of the center point of the target object, the target three-dimensional size of the target object, and the target orientation angle of the target object.

[0128] The correction module includes a first calculation unit, a second calculation unit, and a third calculation unit.

[0129] The first calculation unit is used to calculate the target position coordinates based on the initial position coordinates and the position deviation; the second calculation unit is used to calculate the target three-dimensional dimensions based on the initial three-dimensional dimensions and the three-dimensional dimension deviation; and the third calculation unit is used to calculate the target direction angle based on the initial direction angle and the direction angle deviation.

[0130] According to an embodiment of this disclosure, the extraction module 501 includes a detection unit, which is used to input the scene image to be processed into the target detection network and output the initial pose information of the target object.

[0131] According to embodiments of this disclosure, the above-mentioned device further includes an iteration module, which is used to perform multiple iterations to correct the target pose information after obtaining the target pose information of the target object, until a preset termination condition is met, so as to obtain the final pose information.

[0132] According to embodiments of this disclosure, the iteration module includes a correction unit, a determination unit, and an iteration unit.

[0133] The correction unit is used to perform an i-th iteration correction on the target pose information of the (i-1)th iteration based on the pose deviation of the i-th iteration, to obtain the target pose information of the i-th iteration, wherein the target pose information of the i-th iteration is used as the initial pose information of the i+1th iteration; the determination unit is used to determine the pose deviation of the i+1th iteration associated with the initial pose information of the i+1th iteration; the iteration unit is used to iteratively execute the following operations: performing an i+1th iteration correction on the target pose information of the i-th iteration based on the pose deviation of the i+1th iteration, to obtain the target pose information of the i+1th iteration, and determining the pose deviation of the i+2th iteration associated with the target pose information of the i+1th iteration, until the iterative pose deviation is smaller than a preset deviation threshold, to obtain the final determined pose information.

[0134] According to embodiments of this disclosure, any multiple modules among the extraction module 501, annotation module 502, correction module 503, and correction module 504 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the extraction module 501, annotation module 502, correction module 503, and correction module 504 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in any one of software, hardware, and firmware methods, or in a suitable combination of any of these. Alternatively, at least one of the extraction module 501, annotation module 502, correction module 503, and correction module 504 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0135] Figure 6 A block diagram schematically illustrates an electronic device suitable for implementing a target detection method according to an embodiment of the present disclosure.

[0136] like Figure 6 As shown, an electronic device 600 according to an embodiment of this disclosure includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage portion 608 into a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this disclosure.

[0137] RAM 603 stores various programs and data required for the operation of electronic device 600. Processor 601, ROM 602, and RAM 603 are interconnected via bus 604. Processor 601 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 602 and / or RAM 603. It should be noted that the programs may also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0138] According to embodiments of this disclosure, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to a bus 604. The electronic device 600 may also include one or more of the following components connected to the input / output (I / O) interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 610 as needed so that computer programs read from it can be installed into the storage section 608 as needed.

[0139] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0140] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 602 and / or RAM 603 and / or one or more memories other than ROM 602 and RAM 603 described above.

[0141] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the target detection method provided in the embodiments of this disclosure.

[0142] When the computer program is executed by the processor 601, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0143] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 609, and / or installed from the removable medium 611. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0144] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0145] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0146] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0147] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0148] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A target detection method, comprising: Extracting initial pose information of the target object from the scene image to be processed includes performing preliminary three-dimensional pose information extraction on the two-dimensional image input to the target detection network to obtain initial pose information with low accuracy. The initial pose information includes the initial position coordinates of the center point of the target object and the initial three-dimensional dimensions of the target object. The initial three-dimensional dimensions include the length, width, and height of the object's bounding box. The scene image to be processed is used to characterize the surrounding driving environment of the target vehicle during driving. The surrounding driving environment of the target vehicle includes at least one object. Based on the initial pose information, the bounding box of the target object is drawn in the scene image to be processed to obtain the labeled scene image. The initial pose information, which has low accuracy, is reprojected into a two-dimensional image and visualized. This includes: calculating the first position coordinates of multiple corner points of the bounding box of the target object in the camera coordinate system based on the initial pose information; performing coordinate transformation on the first position coordinates of the multiple corner points to obtain the second position coordinates of the multiple corner points of the bounding box of the target object in the image coordinate system; drawing the bounding box of the target object in the scene image to be processed based on the second position coordinates of the multiple corner points to obtain the labeled scene image. When the output result of the target detection network is inaccurate, the bounding box cannot completely cover the target object. The labeled scene image is input into the correction network, and the pose deviation represented by the position annotation information associated with the initial pose information in the labeled scene image is output. Correcting the initial pose information based on the pose deviation to obtain the target pose information of the target object includes adding the initial pose information and the pose deviation to obtain the target pose information.

2. The method according to claim 1, wherein: The initial pose information further includes: the initial orientation angle of the target object, wherein the initial orientation angle is used to characterize: the angle between the first driving direction of the target object and the second driving direction of the target vehicle.

3. The method according to claim 2, wherein: The pose deviation includes: the position deviation of the center point of the target object, the three-dimensional size deviation of the target object, and the orientation angle deviation of the target object; The target pose information includes: the target position coordinates of the center point of the target object, the target three-dimensional dimensions of the target object, and the target orientation angle of the target object; Based on the pose deviation, the initial pose information is corrected to obtain the target pose information of the target object, including: The target position coordinates are calculated based on the initial position coordinates and the position deviation. The target three-dimensional dimensions are calculated based on the initial three-dimensional dimensions and the three-dimensional dimension deviations. The target direction angle is calculated based on the initial direction angle and the direction angle deviation.

4. The method according to claim 1, wherein, The extraction of the initial pose information of the target object from the scene image to be processed includes: The scene image to be processed is input into the target detection network, which outputs the initial pose information of the target object.

5. The method according to claim 1, further comprising, after obtaining the target pose information of the target object: The target pose information is iteratively corrected multiple times until a preset termination condition is met, resulting in the final determined pose information.

6. The method according to claim 5, wherein, The target pose information is iteratively corrected multiple times until a preset termination condition is met, resulting in the final determined pose information, including: Based on the pose deviation of the i-th iteration, the target pose information of the i-1th iteration is corrected in the i-th iteration to obtain the target pose information of the i-th iteration, where the target pose information of the i-th iteration is used as the initial pose information of the i+1th iteration. Determine the pose deviation of the (i+1)th iteration associated with the initial pose information of the (i+1)th iteration; Iterative execution: Based on the pose deviation of the (i+1)th iteration, the target pose information of the i-th iteration is corrected in the (i+1)th iteration to obtain the target pose information of the (i+1)th iteration, and the operation of the pose deviation of the (i+2)th iteration associated with the target pose information of the (i+1)th iteration is determined until the iterative pose deviation is less than the preset deviation threshold, and the final determined pose information is obtained.

7. A target detection device, comprising: The extraction module is used to extract the initial pose information of the target object from the scene image to be processed. This includes performing preliminary three-dimensional pose information extraction on the two-dimensional image input to the target detection network to obtain initial pose information with low accuracy. The initial pose information includes the initial position coordinates of the center point of the target object and the initial three-dimensional dimensions of the target object. The initial three-dimensional dimensions include the length, width, and height of the object's bounding box. The scene image to be processed is used to characterize the surrounding driving environment of the target vehicle during driving. The surrounding driving environment of the target vehicle includes at least one object. The annotation module is used to draw the bounding box of the target object in the scene image to be processed based on the initial pose information, thereby obtaining an annotated scene image. It reprojects the initially pose information, which may have low accuracy, into a two-dimensional image and visualizes it. The module includes: calculating the first position coordinates of multiple corner points of the bounding box of the target object in the camera coordinate system based on the initial pose information; transforming the first position coordinates of the multiple corner points to obtain the second position coordinates of the multiple corner points in the image coordinate system; and drawing the bounding box of the target object in the scene image to be processed based on the second position coordinates of the multiple corner points, thereby obtaining the annotated scene image. In cases where the output of the target detection network is inaccurate, the bounding box may not completely encompass the target object. The correction module is used to input the labeled scene image into the correction network and output the pose deviation represented by the position annotation information associated with the initial pose information in the labeled scene image; The correction module is used to correct the initial pose information based on the pose deviation to obtain the target pose information of the target object, including adding the initial pose information and the pose deviation to obtain the target pose information.

8. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 6.

Citation Information

Patent Citations

  • Monocular image-based three-dimensional target detection system and method

    CN110070025A

  • Pose determination method and device, electronic equipment and readable storage medium

    CN113793251A