A pose prediction method and apparatus, an electronic device, and a computer readable medium

Abstract

The application discloses a pose prediction method and device, electronic equipment and a computer readable medium, and relates to the technical field of computers. The method comprises the following steps: receiving a pose prediction request, obtaining corresponding image data and annotation data corresponding to the image data; inputting the image data and the annotation data into a detector model, and outputting at least one three-dimensional frame, a depth estimation standard deviation and a classification confidence score; determining a three-dimensional frame confidence score corresponding to the at least one three-dimensional frame according to the depth estimation standard deviation and the classification confidence score, and determining a target three-dimensional frame in the at least one three-dimensional frame according to the three-dimensional frame confidence score; obtaining the depth of each feature point in the target three-dimensional frame, and predicting the pose of a three-dimensional target corresponding to the target three-dimensional frame based on the depth. Thus, the accuracy of three-dimensional frame prediction is improved, and the accuracy of pose prediction of a three-dimensional target is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a pose prediction method, apparatus, electronic device, and computer-readable medium. Background Technology

[0002] Currently, monocular 3D object detection is an autonomous driving perception task that identifies foreground objects in 2D visual images and provides the object's category, location, and pose. Compared to LiDAR, ultrasonic recognition technology has lower sensor costs for environmental perception, but it also suffers from unreliable depth perception. When the extracted depth information is inaccurate, 3D perception tasks become extremely difficult.

[0003] In the process of developing this application, the inventors discovered at least the following problems in the prior art:

[0004] Because the depth perception of monocular 3D target detection using ultrasonic recognition technology is unreliable, the detected 3D box is inaccurate, making the 3D perception task extremely difficult. Summary of the Invention

[0005] In view of this, embodiments of this application provide a pose prediction method, apparatus, electronic device, and computer-readable medium, which can solve the problem that the depth perception of existing monocular 3D target detection is unreliable, resulting in inaccurate detected 3D boxes and making 3D perception tasks extremely difficult.

[0006] To achieve the above objectives, according to one aspect of the embodiments of this application, a pose prediction method is provided, comprising: receiving a pose prediction request and acquiring corresponding image data and labeled data corresponding to the image data; inputting the image data and labeled data into a detector model and outputting at least one 3D bounding box, a depth estimation standard deviation, and a classification confidence score; determining a 3D bounding box confidence score corresponding to at least one 3D bounding box based on the depth estimation standard deviation and the classification confidence score, and determining a target 3D bounding box within the at least one 3D bounding box based on the 3D bounding box confidence score; acquiring the depth of each feature point within the target 3D bounding box, and predicting the pose of the 3D target corresponding to the target 3D bounding box based on the depth.

[0007] Optionally, before receiving the pose prediction request, the method further includes: obtaining an initial neural network model; obtaining a training set, the training set including sample image data and the annotation data of the 3D bounding boxes corresponding to the sample image data and the depth annotation data of the feature points in the 3D bounding boxes; determining the standard deviation of the depth estimation, and determining a depth loss function based on the standard deviation; inputting the sample image data into the initial neural network model, using the annotation data of the 3D bounding boxes corresponding to the sample image data and the depth annotation data of the feature points in the 3D bounding boxes as the expected output of the initial neural network model, and training the initial neural network model based on the depth loss function to obtain a detector model.

[0008] Optionally, before obtaining the detector model, the pose prediction method also includes:

[0009] The feature extraction network is invoked to extract image features from the sample image data, and a feature map is generated based on the image features.

[0010] Invoke the depth detection network to generate a depth map based on the feature map;

[0011] Project the 3D target surface feature points corresponding to the annotation data of the 3D bounding box onto the corresponding positions of the feature map to obtain the depth value of each 3D target surface feature point, and obtain the mask of the effective target surface feature points;

[0012] Based on the depth map, the depth value of each 3D target surface feature point, and the mask, determine the depth loss function of the 3D target surface feature points corresponding to the annotation data of the 3D bounding box;

[0013] The detector model is obtained by backpropagating the initial neural network model using the depth loss function based on the feature points of the three-dimensional target surface.

[0014] Optionally, image data and labeled data are input into the detector model, which outputs at least one 3D bounding box, depth estimation standard deviation, and classification confidence score, including:

[0015] The feature extraction network is invoked to extract image features from the image data, and a feature map is generated based on the image features.

[0016] The 3D bounding box detection network is invoked to obtain the corresponding depth features from the feature map, the depth value is obtained by decoding based on the depth features, and at least one 3D bounding box is determined based on the depth value.

[0017] Call the 3D bounding box detection network to obtain the logarithm of the depth estimation standard deviation from the feature map, and calculate the depth estimation standard deviation from the logarithm.

[0018] The classification detection network is invoked to determine the classification confidence score of the classification corresponding to at least one 3D bounding box obtained from the feature map.

[0019] Optionally, based on the depth estimation standard deviation and the classification confidence score, a 3D bounding box confidence score is determined for at least one 3D bounding box, including:

[0020] The depth estimation confidence score is determined based on the standard deviation of the depth estimation.

[0021] Determine the product of the depth estimation confidence score and the classification confidence score, and then determine the corresponding 3D bounding box confidence score based on the product.

[0022] Optionally, determining the target 3D bounding box within at least one 3D bounding box based on the 3D bounding box confidence score includes:

[0023] The confidence scores of the 3D bounding boxes are sorted, and a preset number of confidence scores of the 3D bounding boxes are sequentially obtained from the sorted confidence scores. The 3D bounding boxes corresponding to the preset number of confidence scores of the 3D bounding boxes are determined as the target 3D bounding boxes.

[0024] In addition, this application also provides a pose prediction device, including:

[0025] The receiving unit is configured to receive pose prediction requests and obtain the corresponding image data and the annotation data corresponding to the image data.

[0026] The model analysis unit is configured to input image data and labeled data into the detector model and output at least one 3D bounding box, depth estimation standard deviation and classification confidence score.

[0027] The target 3D bounding box determination unit is configured to determine the 3D bounding box confidence score corresponding to at least one 3D bounding box based on the depth estimation standard deviation and the classification confidence score, and to determine the target 3D bounding box in at least one 3D bounding box based on the 3D bounding box confidence score.

[0028] The pose prediction unit is configured to acquire the depth of each feature point within the target 3D bounding box, and predict the pose of the 3D target corresponding to the target 3D bounding box based on the depth.

[0029] Optionally, the device also includes a model training unit configured to:

[0030] Obtain the initial neural network model;

[0031] Obtain the training set, which includes sample image data, the annotation data of the corresponding 3D bounding boxes, and the depth annotation data of the feature points in the 3D bounding boxes;

[0032] Determine the standard deviation of the depth estimate, and then determine the depth loss function based on the standard deviation.

[0033] The sample image data is input into the initial neural network model. The annotation data of the 3D bounding box corresponding to the sample image data and the depth annotation data of the feature points in the 3D bounding box are used as the expected output of the initial neural network model. The initial neural network model is trained based on the depth loss function to obtain the detector model.

[0034] Optionally, the model training unit is further configured to:

[0035] The feature extraction network is invoked to extract image features from the sample image data, and a feature map is generated based on the image features.

[0036] Invoke the depth detection network to generate a depth map based on the feature map;

[0037] Project the 3D target surface feature points corresponding to the annotation data of the 3D bounding box onto the corresponding positions of the feature map to obtain the depth value of each 3D target surface feature point, and obtain the mask of the effective target surface feature points;

[0038] Based on the depth map, the depth value of each 3D target surface feature point, and the mask, determine the depth loss function of the 3D target surface feature points corresponding to the annotation data of the 3D bounding box;

[0039] The detector model is obtained by backpropagating the initial neural network model using the depth loss function based on the feature points of the three-dimensional target surface.

[0040] Optionally, the model analysis unit is further configured to:

[0041] The feature extraction network is invoked to extract image features from the image data, and a feature map is generated based on the image features.

[0042] The 3D bounding box detection network is invoked to obtain the corresponding depth features from the feature map, the depth value is obtained by decoding based on the depth features, and at least one 3D bounding box is determined based on the depth value.

[0043] Call the 3D bounding box detection network to obtain the logarithm of the depth estimation standard deviation from the feature map, and calculate the depth estimation standard deviation from the logarithm.

[0044] The classification detection network is invoked to determine the classification confidence score of the classification corresponding to at least one 3D bounding box obtained from the feature map.

[0045] Optionally, the target 3D bounding box determination unit is further configured to:

[0046] The depth estimation confidence score is determined based on the standard deviation of the depth estimation.

[0047] Determine the product of the depth estimation confidence score and the classification confidence score, and then determine the corresponding 3D bounding box confidence score based on the product.

[0048] Optionally, the target 3D bounding box determination unit is further configured to:

[0049] The confidence scores of the 3D bounding boxes are sorted, and a preset number of confidence scores of the 3D bounding boxes are sequentially obtained from the sorted confidence scores. The 3D bounding boxes corresponding to the preset number of confidence scores of the 3D bounding boxes are determined as the target 3D bounding boxes.

[0050] In addition, this application also provides a pose prediction electronic device, including: one or more processors; and a storage device for storing one or more programs, which, when executed by one or more processors, enable the one or more processors to implement the pose prediction method as described above.

[0051] In addition, this application also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the pose prediction method as described above.

[0052] One embodiment of the above invention has the following advantages or beneficial effects: This application receives a pose prediction request, obtains corresponding image data and corresponding annotation data; inputs the image data and annotation data into a detector model, and outputs at least one 3D bounding box, a depth estimation standard deviation, and a classification confidence score; determines the 3D bounding box confidence score corresponding to at least one 3D bounding box based on the depth estimation standard deviation and the classification confidence score, and determines the target 3D bounding box within the at least one 3D bounding box based on the 3D bounding box confidence score; obtains the depth of each feature point within the target 3D bounding box, and predicts the pose of the 3D target corresponding to the target 3D bounding box based on the depth. This improves the accuracy of 3D bounding box prediction and allows for accurate prediction of the 3D target pose based on the predicted 3D bounding box, thus enhancing the accuracy of 3D target pose prediction.

[0053] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0054] The accompanying drawings are provided to better understand this application and do not constitute an undue limitation thereof. Wherein:

[0055] Figure 1 This is a schematic diagram of the main flow of the pose prediction method according to the first embodiment of this application;

[0056] Figure 2 This is a schematic diagram of the main flow of the pose prediction method according to the second embodiment of this application;

[0057] Figure 3 This is a schematic diagram of the execution logic of the detector model of the pose prediction method according to the third embodiment of this application;

[0058] Figure 4 This is a schematic diagram of the main units of the pose prediction device according to an embodiment of this application;

[0059] Figure 5 This is an exemplary system architecture diagram to which embodiments of this application can be applied;

[0060] Figure 6 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers in the embodiments of this application. Detailed Implementation

[0061] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description. The acquisition, storage, use, and processing of data in the technical solutions of this application all comply with relevant national laws and regulations.

[0062] Figure 1 This is a schematic diagram of the main flow of the pose prediction method according to the first embodiment of this application, as shown below. Figure 1 As shown, pose prediction methods include:

[0063] Step S101: Receive pose prediction request and obtain the corresponding image data and the annotation data corresponding to the image data.

[0064] In this embodiment, the entity executing the pose prediction method (e.g., a server) can receive pose prediction requests via wired or wireless connections. For example, it could be a request to identify a foreground target from a 2D visual image and provide the pose of the foreground target.

[0065] After receiving a pose prediction request, the executing entity can obtain the image acquisition address carried in the request, and then retrieve the image data corresponding to the request, as well as the corresponding annotation data. The image data can be, for example, 2D visual image data, i.e., two-dimensional planar image data. The annotation data can be, for example, data annotating 3D objects in the planar data. For instance, the annotation data could be dynamic obstacle targets on a road as shown in the image data, such as cars, pedestrians, bicycles, trucks, and buses; that is, data obtained by annotating dynamic obstacle targets such as cars, pedestrians, bicycles, trucks, and buses in the image data.

[0066] Step S102: Input the image data and labeled data into the detector model and output at least one 3D bounding box, depth estimation standard deviation and classification confidence score.

[0067] In autonomous driving scenarios, 3D bounding boxes are used to represent which regions in image data contain 3D objects, such as cars, pedestrians, and bicycles. This facilitates the subsequent classification of 3D objects in those regions, identifying which are obstacles, and enabling autonomous vehicles to navigate around these obstacles, thereby improving the safety of autonomous vehicles.

[0068] The execution logic of the detector model is as follows: Figure 3 As shown: Initially, 2D visual image data is input into the feature extraction network for feature extraction. After obtaining the extracted features, the extracted features are input into a detection head for classification, a detection head for determining 3D boxes, and a detection head for obtaining depth. Based on the output results of the three detection heads, the category, location, and shape of the 3D target in the input 2D visual image data are determined.

[0069] Specifically, image data and labeled data are input into the detector model, which outputs at least one 3D bounding box, depth estimation standard deviation, and classification confidence score, including:

[0070] The process involves: calling a feature extraction network (backbone network) to extract image features from the image data; generating a feature map based on these features; then calling a 3D box detection network (a detection head for identifying 3D boxes) to obtain the corresponding depth features from the feature map; decoding the depth features to obtain depth values; and determining at least one 3D box based on the depth values. The process also involves calling the 3D box detection network (a detection head for identifying 3D boxes) to obtain the logarithm of the depth estimation standard deviation from the feature map; and calculating the depth estimation standard deviation based on the logarithm. Finally, calling a classification detection network (a detection head for classification) to determine the classification confidence score corresponding to the at least one 3D box obtained from the feature map.

[0071] Step S103: Based on the depth estimation standard deviation and classification confidence score, determine the confidence score of the 3D bounding box corresponding to at least one 3D bounding box, and determine the target 3D bounding box in the at least one 3D bounding box based on the confidence score of the 3D bounding box.

[0072] Specifically, based on the depth estimation standard deviation and classification confidence score, the confidence score of at least one 3D bounding box is determined, including:

[0073] Determine the confidence score of the depth estimate based on the standard deviation of the depth estimate. Where d is short for depth, and d means: the estimated target depth.

[0074] Determine the product of the depth estimation confidence score and the classification confidence score, and then determine the corresponding 3D bounding box confidence score based on the product.

[0075] During the inference phase, the execution entity calls the detector model to read in image data and labeled data. The detector model extracts a feature map of H / 4*W / 4*256 through a feature extraction network. This feature map is fed into the 3D box detection head of the detector model. The regression branch regresses depth features and decodes depth values, while simultaneously regressing the logarithm of the standard deviation a = log(σ). d Therefore, the standard deviation σ of the depth estimate can be calculated using the index. d =e a , this σ d >0; Define the confidence level of the depth estimate for the target center point based on the depth standard deviation. The classification confidence score P is predicted using the classification branch of the monocular 3D detector model (the classification branch refers to the network branch that classifies each feature vector on the feature map of the target; the input is the feature map, and the output is the type score). class The credibility score P of the 3D target can then be calculated. 3d =P depth *P class Based on this, we can obtain H / 4*W / 4*C target scores (i.e., the confidence scores of 3D targets), where H is the height of the original image, W is the width of the original image, and C refers to the number of feature vector channels in the feature map, i.e., the feature dimension. The top K effectively detected 3D targets (effective detection means that the detected 3D targets cannot be abstract or vague; the targets must be measurable and quantifiable) are selected as the final detection results.

[0076] Specifically, determining the target 3D bounding box within at least one 3D bounding box based on the 3D bounding box confidence score includes:

[0077] The confidence scores of the 3D bounding boxes are sorted, and a preset number of confidence scores of the 3D bounding boxes are sequentially obtained from the sorted confidence scores. The 3D bounding boxes corresponding to the preset number of confidence scores of the 3D bounding boxes are determined as the target 3D bounding boxes.

[0078] For example, the executing agent can select the top K effectively detected 3D targets as the final detection result based on H / 4*W / 4*C target scores. Here, H / 4*W / 4 is the feature map spatial resolution. Specifically, the H / 4*W / 4*C target scores are sorted in descending order, and then the top K target scores are selected sequentially. The 3D bounding boxes corresponding to the selected top K target scores are defined as the target 3D bounding boxes. The executing agent can use the 3D targets (such as cars, pedestrians, tree stumps, etc.) corresponding to the target 3D bounding boxes (the target 3D bounding box can be a box that encloses the 3D target, can match the outline of the 3D target, or it can be a bounding box that does not match the outline of the 3D target but still encloses the 3D target; the object used to represent the object in the 3D bounding box is the 3D target) as the final detection result.

[0079] Step S104: Obtain the depth of each feature point within the target 3D bounding box, and predict the pose of the 3D target corresponding to the target 3D bounding box based on the depth.

[0080] After determining the target 3D bounding box, the executing agent can input the target 3D bounding box into a detection head to obtain depth, thereby outputting the depth of each feature point within the target 3D bounding box. The executing agent then inputs the depth of each feature point within the target 3D bounding box into a trained pose estimation network to output the corresponding pose. The pose estimation network is used to represent the correspondence between depth and pose.

[0081] This embodiment receives a pose prediction request and obtains the corresponding image data and its corresponding annotation data. The image data and annotation data are input into a detector model, which outputs at least one 3D bounding box, a depth estimation standard deviation, and a classification confidence score. Based on the depth estimation standard deviation and classification confidence score, a 3D bounding box confidence score is determined for each bounding box. Based on the 3D bounding box confidence score, a target 3D bounding box within the at least one bounding box is determined. The depth of each feature point within the target 3D bounding box is obtained, and the pose of the 3D target corresponding to the target 3D bounding box is predicted based on the depth. This improves the accuracy of 3D bounding box prediction and allows for accurate prediction of the 3D target's pose based on the predicted 3D bounding box, thus enhancing the accuracy of 3D target pose prediction.

[0082] Figure 2 This is a schematic diagram of the main flow of the pose prediction method according to the second embodiment of this application, as shown below. Figure 2 As shown, pose prediction methods include:

[0083] Detector model training steps:

[0084] Step S201: Obtain an initial neural network model, such as a convolutional neural network (CNN).

[0085] Step S202: Obtain a training set. The training set includes sample image data, labeled data of the corresponding 3D bounding boxes, and depth labeled data of feature points within the 3D bounding boxes. The sample image data can be images containing obstacles obtained from various autonomous driving scenarios; this embodiment does not specifically limit the source of the sample image data in the training set. The labeled data of the corresponding 3D bounding boxes can be data of 3D objects in the labeled sample image data. The depth labeled data of feature points within the 3D bounding boxes is the depth data of each feature point that makes up the point cloud when a point cloud exists within the 3D bounding box.

[0086] Step S203: Determine the standard deviation of the depth estimate, and determine the depth loss function based on the standard deviation.

[0087] During the training phase, multi-task learning is employed to model the uncertainty of depth. Due to the presence of annotation errors in the training samples, the depth of the modeled target follows a Laplace distribution. First, sample image data and corresponding annotation data are read in. This annotation data includes the annotation data of the 3D bounding boxes corresponding to the sample image data and the depth annotation data of the feature points within the 3D bounding boxes. The initial neural network model extracts feature maps through the backbone network (i.e., the feature extraction network). These feature maps are fed into the 3D box detector head, where the regression branch regresses the depth features and decodes the depth values, simultaneously regressing the logarithm of the standard deviation a = log(σ). d Therefore, the standard deviation σ of the depth estimate can be calculated using the index. d =e a , this σ d >0.

[0088] According to formula (1) which defines depth loss (i.e., the loss function for training the detector model):

[0089]

[0090] Where, μ d σ represents the depth value of the target center point predicted by the model. d d represents the standard deviation of the target center point depth predicted by the model. gt This is the true depth value corresponding to the center point of the target.

[0091] Formula (1) can be used to calculate the sum of the depth losses of the effective 3D target center points on the feature map obtained by the feature extraction network and take the average value, so that the model can be backpropagated to update the model parameters. The input of the finally trained detector model is a single frame image and camera intrinsic parameters, and the output is the category of the obstacle target and the 3D box, as well as the dense depth of each feature point in the 3D box.

[0092] Step S204: Input the sample image data into the initial neural network model, use the annotation data of the 3D bounding box corresponding to the sample image data and the depth annotation data of the feature points in the 3D bounding box as the expected output of the initial neural network model, and train the initial neural network model based on the depth loss function to obtain the detector model.

[0093] Specifically, before obtaining the detector model, pose prediction methods also include:

[0094] A feature extraction network is invoked to extract image features from the sample image data, and a feature map of H / 4*W / 4*256 is generated based on the image features. A depth detection network is invoked to generate a depth map of H / 4*W / 4*1 based on the feature map. The 3D target surface feature points corresponding to the annotation data of the 3D bounding box are projected onto the corresponding positions in the feature map to obtain the depth value D of each 3D target surface feature point. gt And obtain the mask M of the effective target surface feature points; based on the depth map of H / 4*W / 4*1 and the depth value D of each three-dimensional target surface feature point. gt And a mask M (for example, a mask of 1 for areas with point clouds and a mask of 0 for areas without point clouds), to determine the depth loss function L of the 3D target surface feature points corresponding to the annotation data of the 3D bounding box. surface_depth =|DD gt |*M, where D is an abbreviation for depth, meaning: the predicted surface depth of the target; D gt Meaning: The true value of the surface depth of the target; DD gt It is the difference between the predicted and true surface depth values ​​of the target. Based on the depth loss function determined by the standard deviation, a further depth loss function L is derived from the feature points of the 3D target surface. surface_depth =|DD gt |*M performs backpropagation on the initial neural network model to update the model parameters, thereby obtaining the detector model, which makes the output of the trained detector model more accurate and more practical.

[0095] This application embodiment uses the depth of points on the object's surface measured by point cloud as supervisory information for model pre-training to improve the accuracy of the model's estimation of the object's center point. For example, when extracting a 3D target from a monocular image, relying solely on the depth supervisory information of the center point of the labeled 3D box target has limitations in estimating the target's depth. This method predicts the depth of the target's surface points and simultaneously uses the point cloud target surface point information within the 3D box as supervisory information to improve the model's 3D box depth estimation capability, as detailed below:

[0096] During the training phase, input sample image data and labeled data, the model extracts a feature map of H / 4*W / 4*256 through a backbone network. This feature map is then fed into a depth detection head, which predicts and outputs a depth map of H / 4*W / 4*1. The depth detection head predicts the depth at every location on the feature map, forming a dense depth map.

[0097] Setting D gt The depth of each point on the feature map spatial resolution H / 4*W / 4 is determined by the following steps: the depth value of each feature point is obtained by projecting the surface points of the point cloud object in the labeled 3D box onto the corresponding position in the feature map, and the mask M of the surface points of the effective target (that is, the target cannot be abstract or fuzzy, the target must be measurable and quantifiable, such a target is an effective target) is obtained (the mask M is a string of binary code obtained by performing a bitwise AND operation on the target field corresponding to the surface points of the effective target).

[0098] The depth loss of points on the surface of a 3D object is calculated as shown in formula (2):

[0099] L surface_depth =|DD gt |*M (2)

[0100] The depth loss obtained by formula (2) is used to backpropagate the initial neural model to update the model parameters of the initial neural network model. This multi-task learning helps to improve the model's ability to estimate the depth of the 3D box.

[0101] Step S205: Receive pose prediction request and obtain the corresponding image data and the annotation data corresponding to the image data.

[0102] Step S206: Input the image data and labeled data into the detector model and output at least one 3D bounding box, depth estimation standard deviation and classification confidence score.

[0103] In this embodiment of the application, the standard deviation of depth estimation σ d =e a , σ d >0 can be the logarithm of the standard deviation regressed from the image data and labeled data input to the detector model, a = log(σ d ) was calculated.

[0104] Step S207, estimate the standard deviation σ based on depth. d =e a And classification confidence score P class Determine the confidence score P of at least one 3D bounding box. 3d =P depth *P class Based on the 3D bounding box confidence score P3d =P depth *P class Determine the target 3D bounding box within at least one 3D bounding box, where P depth P represents the depth credibility score. class This is the classification confidence score. The target 3D bounding box can be a 3D bounding box with a confidence score greater than a preset score threshold.

[0105] Step S208: Obtain the depth of each feature point within the target 3D bounding box, and predict the pose of the 3D target corresponding to the target 3D bounding box based on the depth.

[0106] The principles of steps S205 to S208 are similar to those of steps S101 to S104, and will not be repeated here.

[0107] Figure 3 This is a schematic diagram illustrating an application scenario of the pose prediction method according to the third embodiment of this application. The pose prediction method of this application is applied to an obstacle perception scenario in autonomous driving. Figure 3 As shown: Initially, 2D visual image data is input into a feature extraction network for feature extraction. After obtaining the extracted features, these features are input to three separate detection heads: one for classification, one for determining 3D boxes, and one for obtaining depth. Based on the outputs of these three detection heads, the category, location, and shape of the 3D target in the input 2D visual image data are determined. This embodiment of the application is based on a one-stage monocular 3D detector. It predicts the standard deviation of the target center point depth by modeling the uncertainty of depth estimation, and uses the depth of object surface points measured by point cloud as supervisory information to improve the accuracy of the model's estimation of the object center point, thereby improving the accuracy of determining the category, location, and shape of the 3D target in the input 2D visual image data.

[0108] In this embodiment, "dense depth" refers to depth; "3D box" refers to a three-dimensional bounding box; and "lidar" refers to a laser radar. This embodiment uses the standard deviation of the detection head's depth. During the training phase, the logarithm of the standard deviation is used as a constraint on the uncertainty of depth, improving the accuracy of the model's depth prediction. Simultaneously, during the inference phase, the confidence level of the depth estimate is calculated based on the standard deviation. Then, based on the classification score and depth score, the score of the 3D box is calculated, and effectively detected 3D boxes are selected. Specifically: P 3d =P depth *P class By using the surface depth values ​​of the radar lidar point cloud of the 3D target as supervision information during the training phase, the model's ability to estimate the depth of the target's center point is improved.

[0109] Figure 4This is a schematic diagram of the main units of a pose prediction device according to an embodiment of this application. Figure 4 As shown, the pose prediction device 400 includes a receiving unit 401, a model analysis unit 402, a target 3D bounding box determination unit 403, and a pose prediction unit 404.

[0110] The receiving unit 401 is configured to receive pose prediction requests and obtain the corresponding image data and the annotation data corresponding to the image data.

[0111] The model analysis unit 402 is configured to input image data and labeled data into the detector model and output at least one 3D bounding box, depth estimation standard deviation, and classification confidence score.

[0112] The target 3D bounding box determination unit 403 is configured to determine the 3D bounding box confidence score corresponding to at least one 3D bounding box based on the depth estimation standard deviation and the classification confidence score, and to determine the target 3D bounding box in at least one 3D bounding box based on the 3D bounding box confidence score.

[0113] The pose prediction unit 404 is configured to acquire the depth of each feature point within the target 3D bounding box and predict the pose of the 3D target corresponding to the target 3D bounding box based on the depth.

[0114] In some embodiments, the apparatus further includes Figure 4 The model training unit (not shown) is configured to: acquire an initial neural network model; acquire a training set, which includes sample image data and the annotation data of the 3D bounding boxes corresponding to the sample image data and the depth annotation data of the feature points in the 3D bounding boxes; determine the standard deviation of the depth estimation and determine the depth loss function based on the standard deviation; input the sample image data into the initial neural network model, use the annotation data of the 3D bounding boxes corresponding to the sample image data and the depth annotation data of the feature points in the 3D bounding boxes as the expected output of the initial neural network model, and train the initial neural network model based on the depth loss function to obtain the detector model.

[0115] In some embodiments, the model training unit is further configured to: invoke a feature extraction network to extract image features from the sample image data and generate a feature map based on the image features; invoke a depth detection network to generate a depth map based on the feature map; project the 3D target surface feature points corresponding to the annotation data of the 3D bounding box onto the corresponding positions of the feature map to obtain the depth value of each 3D target surface feature point and obtain a mask for the effective target surface feature points; determine the depth loss function of the 3D target surface feature points corresponding to the annotation data of the 3D bounding box based on the depth map, the depth value of each 3D target surface feature point and the mask; and perform backpropagation on the initial neural network model to update the model parameters based on the depth loss function of the 3D target surface feature points to obtain the detector model.

[0116] In some embodiments, the model analysis unit 402 is further configured to: invoke a feature extraction network to extract image features from image data, and generate a feature map based on the image features; invoke a 3D bounding box detection network to obtain corresponding depth features based on the feature map, decode depth values ​​based on the depth features, and determine at least one 3D bounding box based on the depth values; invoke the 3D bounding box detection network to obtain the logarithm of the depth estimation standard deviation based on the feature map, and calculate the depth estimation standard deviation based on the logarithm; and invoke a classification detection network to determine the classification confidence score of the classification corresponding to the at least one 3D bounding box obtained based on the feature map.

[0117] In some embodiments, the target 3D bounding box determination unit 403 is further configured to: determine a depth estimation confidence score based on the depth estimation standard deviation; determine the product of the depth estimation confidence score and the classification confidence score; and determine the 3D bounding box confidence score of the corresponding 3D bounding box based on the product.

[0118] In some embodiments, the target 3D bounding box determination unit 403 is further configured to: sort the 3D bounding box confidence scores, sequentially obtain a preset number of 3D bounding box confidence scores from the sorted 3D bounding box confidence scores, and determine the 3D bounding boxes corresponding to the sequentially obtained preset number of 3D bounding box confidence scores as target 3D bounding boxes.

[0119] It should be noted that the pose prediction method and pose prediction device in this application are related in terms of specific implementation, so repeated content will not be described again.

[0120] Figure 5 An exemplary system architecture 500 is shown that can be applied to the pose prediction method or pose prediction apparatus of the present application embodiments.

[0121] like Figure 5 As shown, system architecture 500 may include terminal devices 501, 502, and 503, a network 504, and a server 505. Network 504 serves as the medium for providing communication links between terminal devices 501, 502, and 503 and server 505. Network 504 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0122] Users can use terminal devices 501, 502, and 503 to interact with server 505 via network 504 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 501, 502, and 503, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0123] Terminal devices 501, 502, and 503 can be various electronic devices with a pose prediction processing screen and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0124] Server 505 can be a server providing various services, such as a backend management server supporting pose prediction requests submitted by users using terminal devices 501, 502, and 503 (for example only). The backend management server can receive pose prediction requests, obtain corresponding image data and corresponding annotation data; input the image data and annotation data into a detector model, and output at least one 3D bounding box, a depth estimation standard deviation, and a classification confidence score; determine the 3D bounding box confidence score corresponding to at least one 3D bounding box based on the depth estimation standard deviation and classification confidence score, and determine the target 3D bounding box within the at least one 3D bounding box based on the 3D bounding box confidence score; obtain the depth of each feature point within the target 3D bounding box, and predict the pose of the 3D target corresponding to the target 3D bounding box based on the depth. This improves the accuracy of 3D bounding box prediction and allows for accurate prediction of the 3D target's pose based on the predicted 3D bounding box, thus enhancing the accuracy of 3D target pose prediction.

[0125] It should be noted that the pose prediction method provided in this application embodiment is generally executed by server 505, and correspondingly, the pose prediction device is generally set in server 505.

[0126] It should be understood that Figure 5 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0127] The following is for reference. Figure 6 It shows a schematic diagram of the structure of a computer system 600 suitable for implementing a terminal device according to the embodiments of this application. Figure 6 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0128] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 602 or programs loaded from storage section 608 into random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of the computer system 600. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0129] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0130] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit (CPU) 601, it performs the functions defined above in the system of this application.

[0131] It should be noted that the computer-readable medium shown in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. Computer-readable storage media can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0132] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0133] The units described in the embodiments of this application can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor can be described as including a receiving unit, a model analysis unit, a target 3D bounding box determination unit, and a pose prediction unit. The names of these units do not necessarily limit the specific unit itself.

[0134] In another aspect, this application also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs that, when executed by the device, cause the device to receive a pose prediction request, acquire corresponding image data and corresponding annotation data; input the image data and annotation data into a detector model, and output at least one 3D bounding box, a depth estimation standard deviation, and a classification confidence score; determine the 3D bounding box confidence score corresponding to at least one 3D bounding box based on the depth estimation standard deviation and the classification confidence score, and determine the target 3D bounding box within the at least one 3D bounding box based on the 3D bounding box confidence score; acquire the depth of each feature point within the target 3D bounding box, and predict the pose of the 3D target corresponding to the target 3D bounding box based on the depth.

[0135] According to the technical solution of the embodiments of this application, the accuracy of 3D bounding box prediction can be improved, and the pose of 3D target can be accurately predicted based on the predicted 3D bounding box, thereby improving the accuracy of 3D target pose prediction.

[0136] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A pose prediction method, characterized in that, include: Receive pose prediction request and obtain corresponding image data; The process involves inputting the image data into a detector model and outputting at least one 3D bounding box, a depth estimation standard deviation, and a classification confidence score. This includes: invoking a feature extraction network to extract image features from the image data and generating a feature map based on the image features; invoking a 3D bounding box detection network to obtain corresponding depth features from the feature map, decoding depth values ​​based on the depth features, and determining at least one 3D bounding box based on the depth values; invoking the 3D bounding box detection network to obtain the logarithm of the depth estimation standard deviation from the feature map and calculating the depth estimation standard deviation based on the logarithm; and invoking a classification detection network to determine the classification confidence score corresponding to the at least one 3D bounding box based on the feature map. Based on the depth estimation standard deviation and the classification confidence score, determine the 3D box confidence score corresponding to the at least one 3D box, and determine the target 3D box in the at least one 3D box based on the 3D box confidence score; The depth of each feature point within the target 3D bounding box is obtained, and the pose of the 3D target corresponding to the target 3D bounding box is predicted based on the depth.

2. The method according to claim 1, characterized in that, Before receiving the pose prediction request, the method further includes: Obtain the initial neural network model; Obtain a training set, which includes sample image data, annotation data of the 3D bounding boxes corresponding to the sample image data, and depth annotation data of feature points in the 3D bounding boxes; Determine the standard deviation of the depth estimate, and determine the depth loss function based on the standard deviation; The sample image data is input into the initial neural network model. The annotation data of the 3D bounding box corresponding to the sample image data and the depth annotation data of the feature points in the 3D bounding box are used as the expected output of the initial neural network model. The initial neural network model is trained based on the depth loss function to obtain the detector model.

3. The method according to claim 2, characterized in that, Before obtaining the detector model, the method further includes: A feature extraction network is invoked to extract image features from the sample image data, and a feature map is generated based on the image features. Invoke a depth detection network to generate a depth map based on the feature map; Project the 3D target surface feature points corresponding to the annotation data of the 3D bounding box onto the corresponding positions of the feature map to obtain the depth value of each 3D target surface feature point, and obtain the mask of the effective target surface feature points; Based on the depth map, the depth value of each 3D target surface feature point, and the mask, determine the depth loss function of the 3D target surface feature points corresponding to the annotation data of the 3D bounding box; The initial neural network model is backpropagated and its parameters are updated based on the depth loss function of the feature points on the three-dimensional target surface to obtain the detector model.

4. The method according to claim 1, characterized in that, The step of determining the 3D bounding box confidence score corresponding to the at least one 3D bounding box based on the depth estimation standard deviation and the classification confidence score includes: The depth estimation confidence score is determined based on the standard deviation of the depth estimation. Determine the product of the depth estimation confidence score and the classification confidence score, and determine the 3D bounding box confidence score of the corresponding 3D bounding box based on the product.

5. The method according to claim 1, characterized in that, Determining the target 3D bounding box within the at least one 3D bounding box based on the 3D bounding box confidence score includes: The confidence scores of the three-dimensional boxes are sorted, and a preset number of confidence scores of three-dimensional boxes are sequentially obtained from the sorted confidence scores. The three-dimensional boxes corresponding to the preset number of confidence scores of three-dimensional boxes are determined as the target three-dimensional boxes.

6. A pose prediction device, characterized in that, include: The receiving unit is configured to receive pose prediction requests and acquire corresponding image data. The model analysis unit is configured to input the image data into a detector model and output at least one 3D bounding box, a depth estimation standard deviation, and a classification confidence score. The unit includes: invoking a feature extraction network to extract image features from the image data and generating a feature map based on the image features; invoking a 3D bounding box detection network to obtain corresponding depth features from the feature map, decoding depth values ​​based on the depth features, and determining at least one 3D bounding box based on the depth values; invoking the 3D bounding box detection network to obtain the logarithm of the depth estimation standard deviation from the feature map and calculating the depth estimation standard deviation based on the logarithm; and invoking a classification detection network to determine the classification confidence score corresponding to the at least one 3D bounding box based on the feature map. The target 3D bounding box determination unit is configured to determine the 3D bounding box confidence score corresponding to the at least one 3D bounding box based on the depth estimation standard deviation and the classification confidence score, and to determine the target 3D bounding box in the at least one 3D bounding box based on the 3D bounding box confidence score; The pose prediction unit is configured to acquire the depth of each feature point within the target 3D bounding box, and predict the pose of the 3D target corresponding to the target 3D bounding box based on the depth.

7. The apparatus according to claim 6, characterized in that, The device further includes a model training unit, configured to: Obtain the initial neural network model; Obtain a training set, which includes sample image data, annotation data of the 3D bounding boxes corresponding to the sample image data, and depth annotation data of feature points in the 3D bounding boxes; Determine the standard deviation of the depth estimate, and determine the depth loss function based on the standard deviation; The sample image data is input into the initial neural network model. The annotation data of the 3D bounding box corresponding to the sample image data and the depth annotation data of the feature points in the 3D bounding box are used as the expected output of the initial neural network model. The initial neural network model is trained based on the depth loss function to obtain the detector model.

8. A pose prediction electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-5.

9. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-5.

Images