A method for verifying output pose accuracy of a pose estimation model and related apparatuses

Abstract

The application discloses a pose estimation model output pose accuracy verification method and related devices, and the method comprises the following steps: obtaining a to-be-estimated image sequence, and converting target objects in the to-be-estimated image sequence into a 2D mask image sequence based on a mask conversion model; loading a CAD model corresponding to the target objects in the to-be-estimated image sequence, and obtaining the edge frame vertex of the CAD model; performing projection operation based on the edge frame vertex and the camera internal parameter corresponding to the camera for collecting the to-be-estimated image sequence, and obtaining the pose mask corresponding to the target objects in each to-be-estimated image in the to-be-estimated image sequence; and performing pose accuracy verification processing according to the relationship between the 2D mask in the 2D mask sequence and the corresponding pose mask. In the embodiment of the application, the good or bad degree of the pose estimation model output pose result can be evaluated without a real pose reference, and the result can be quickly checked by integrating into any pose estimation model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a pose estimation model output pose accuracy verification method and related device. BACKGROUND

[0002] 6D object pose estimation (6-DoF Object Pose Estimation) is an important research direction in the field of computer vision and robotics, and its core goal is to determine the position (x, y, z) and direction (roll, pitch, yaw) of an object in three-dimensional space; In the process of continuous pose estimation of moving objects using the Foundationpose pose estimation model, it is found that the model cannot perceive the error pose in the current output pose during continuous estimation; The error pose is reflected in that the estimated pose position xyz is offset compared with the true position, and the subsequent offset will become larger and larger with the estimation, and the pose will also be inconsistent with reality; resulting in that in the actual use process of the model, if the output result starts to deviate, all subsequent results are error poses and cannot be used. SUMMARY

[0003] The purpose of the present application is to overcome the shortcomings of the prior art, and the present application provides a pose estimation model output pose accuracy verification method and related device, which can evaluate the good and bad degree of the pose estimation model output pose result without real pose reference, and can be quickly integrated into any pose estimation model to verify the result.

[0004] To solve the above technical problems, the embodiment of the present application provides a pose estimation model output pose accuracy verification method, which comprises:

[0005] Obtain a to-be-estimated image sequence, and convert a target object in the to-be-estimated image sequence into a 2D mask image sequence based on a mask conversion model, the 2D mask image sequence corresponding one-to-one to the to-be-estimated image sequence;

[0006] Load a CAD model corresponding to the target object in the to-be-estimated image sequence, and obtain a bounding box vertex of the CAD model;

[0007] Perform projection operation based on the bounding box vertex and an intrinsic parameter of a camera corresponding to the camera collecting the to-be-estimated image sequence, to obtain a pose mask corresponding to the target object in each to-be-estimated image in the to-be-estimated image sequence;

[0008] According to the relationship between each 2D mask in the 2D mask sequence and the corresponding pose mask, perform pose accuracy verification processing.

[0009] Optionally, the mask conversion model converts the target object in the to-be-estimated image sequence into a 2D mask image sequence, comprising:

[0010] The to-be-estimated image sequence is input into the mask conversion model, and a corresponding foreground point and a background point are selected in the first frame of the to-be-estimated image in a selected point manner, and the mask conversion model is a SAM2 model;

[0011] The mask conversion model generates a 2D mask of the target object according to the foreground point in the first frame of the to-be-estimated image;

[0012] The mask conversion model uses a time sequence module to automatically generate a mask for the to-be-estimated image in the subsequent frame of the to-be-estimated image sequence according to the inter-frame change between the to-be-estimated image sequence and the 2D mask formed by the previous frame, and forms a 2D mask image sequence.

[0013] Optionally, the CAD model corresponding to the target object in the to-be-estimated image sequence is loaded, and the edge box vertex of the CAD model is obtained, comprising:

[0014] The CAD model corresponding to the target object in the to-be-estimated image sequence is loaded, and a 3D edge box corresponding to the CAD model is obtained by using an edge box extraction function library;

[0015] The edge box vertex of the CAD model is obtained based on the 3D edge box corresponding to the CAD.

[0016] Optionally, the camera intrinsic parameter corresponding to the camera for collecting the to-be-estimated image sequence is projected based on the edge box vertex, comprising:

[0017] An attitude matrix of the CAD model is obtained, and an estimated attitude of the target object in each frame of the to-be-estimated image sequence is obtained by inputting the to-be-estimated image sequence into an attitude estimation model;

[0018] The estimated attitude of the target object in each frame of the to-be-estimated image sequence and the attitude matrix are calculated and processed to obtain a pose representation of the target object in the camera coordinate system in each frame of the to-be-estimated image sequence;

[0019] The position of each edge box vertex in the camera coordinate system is obtained by calculating the pose representation of the target object in the camera coordinate system in each frame of the to-be-estimated image sequence;

[0020] The position of each edge box vertex in the camera coordinate system is projected to the image coordinate system to obtain each edge box vertex projected to the image coordinate system;

[0021] The pose mask corresponding to the target object in each of the to-be-estimated images in the to-be-estimated image sequence is obtained by performing a convex hull operation on each of the frame top points projected into the image coordinate system, and the pose mask is a region surrounded after the convex hull operation.

[0022] Optionally, the calculation formula for projecting the position of each frame top point in the camera coordinate system into the image coordinate system is as follows:

[0023]

[0024] wherein Z x , Z y , and Z c are the coordinates of the frame top points of the target object in the camera coordinate system; K is the intrinsic parameter of the camera; and u and v are the position points in the image coordinate system.

[0025] Optionally, the formula of the convex hull operation is as follows:

[0026]

[0027] wherein P i is the i-th frame top point; n = 1, 2, 3, …, 8; λ i is the weight coefficient corresponding to the i-th frame top point; and CH(P i ) is convex and contains the i-th frame top point in the pixel coordinate system.

[0028] Optionally, the pose accuracy verification processing according to the relationship between each 2D mask in the 2D mask sequence and the corresponding pose mask includes:

[0029] calculating the intersection between each 2D mask in the 2D mask sequence and the corresponding pose mask to obtain the intersection calculation result between each 2D mask and the corresponding pose mask;

[0030] calculating the area ratio between each 2D mask and the corresponding 2D mask based on the intersection calculation result between each 2D mask and the corresponding pose mask, and performing pose accuracy verification processing according to the calculated area ratio.

[0031] In addition, the embodiment of the present application also provides an output pose accuracy verification device of a pose estimation model, the device comprising:

[0032] a conversion module configured to obtain a to-be-estimated image sequence, and convert a target object in the to-be-estimated image sequence into a 2D mask image sequence based on a mask conversion model, wherein the 2D mask image sequence corresponds to the to-be-estimated image sequence one by one;

[0033] A loading module is configured to load a CAD model corresponding to the target object in the image sequence to be estimated and obtain a bounding box vertex of the CAD model.

[0034] A projection operation module is configured to perform a projection operation based on the bounding box vertex and camera intrinsic parameters of a camera used to capture the image sequence to be estimated, and obtain a pose mask corresponding to the target object in each image to be estimated in the image sequence.

[0035] A verification module is configured to perform pose accuracy verification processing according to a relationship between each 2D mask in the 2D mask sequence and the corresponding pose mask.

[0036] In addition, an electronic device is also provided in an embodiment of the present application, which includes a processor and a memory. The processor runs a computer program or code stored in the memory to implement the output pose accuracy verification method according to any one of the above embodiments.

[0037] In addition, a computer readable storage medium is also provided in an embodiment of the present application, which is configured to store a computer program or code. When the computer program or code is executed by a processor, the output pose accuracy verification method according to any one of the above embodiments is implemented.

[0038] In an embodiment of the present application, the output pose result of the pose estimation model is evaluated without a real pose reference, and the better pose and the worse pose in the output result are distinguished. The method can be quickly integrated into any pose estimation model to verify the result without relying on a specific model. The verification can be performed simultaneously with the output result of the model. For the pose estimation model, the ability to find the pose estimation error in the output pose in real time can be provided, and the pose with estimation error can be additionally processed. Therefore, the accuracy of the output result of the pose estimation model can be improved. The output pose is verified by using the mask information. In the actual use stage without a labeled pose reference, the output pose quality can be basically evaluated, and the quality of the output pose can be distinguished. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application. Those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0040] Figure 1 is a flowchart of the output pose accuracy verification method of the pose estimation model in an embodiment of the present application.

[0041] Figure 2 is a flowchart of an output pose accuracy verification method of a pose estimation model in another embodiment of the present application;

[0042] Figure 3 is a structural composition diagram of an output pose accuracy verification device of a pose estimation model in an embodiment of the present application;

[0043] Figure 4 is a structural composition diagram of an electronic device in an embodiment of the present application;

[0044] Figure 5 is a schematic diagram of the relationship between the pose of an object and its pose mask in an embodiment of the present application;

[0045] Figure 6 is a schematic diagram of the positional relationship between a 2D mask and a pose mask in an embodiment of the present application;

[0046] Figure 7 is a pose verification score graph of a motion sequence in an embodiment of the present application. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0048] Embodiment one, please refer to Figure 1 , Figure 1 is a flowchart of an output pose accuracy verification method of a pose estimation model in an embodiment of the present application.

[0049] As Figure 1 shown, an output pose accuracy verification method of a pose estimation model, the method comprises:

[0050] S101: obtaining a to-be-estimated image sequence, and converting a target object in the to-be-estimated image sequence into a 2D mask image sequence based on a mask conversion model, the 2D mask image sequence corresponding to the to-be-estimated image sequence one by one;

[0051] In the specific implementation of the present application, the mask conversion model converts the target object in the image sequence to be estimated into a 2D mask image sequence, which includes: inputting the image sequence to be estimated into the mask conversion model, and selecting corresponding foreground points and background points in the first frame of the image sequence to be estimated in a point selection manner, the mask conversion model is a SAM2 model; the mask conversion model generates a 2D mask of the target object according to the foreground points in the first frame of the image to be estimated; the mask conversion model uses a time sequence module to automatically generate a mask for the image to be estimated in the subsequent frames of the image sequence in sequence according to the inter-frame changes between the image sequence to be estimated and the 2D mask formed by the previous frame, forming a 2D mask image sequence.

[0052] Specifically, first, the image sequence to be estimated is obtained, and the image sequence to be estimated is input into the mask conversion model, which is a SAM2 (Segment Anything Model 2) model; it supports general segmentation of images and videos, has core capabilities such as zero-shot generalization, interactive prompting, and real-time processing; through optimization of the architecture and introduction of a spatiotemporal memory mechanism, compared with the previous SAM model, the segmentation accuracy is improved and the processing speed is accelerated by 6 times, becoming an important tool in the fields of autonomous driving, film production, AR / VR, etc.; at this time, the corresponding foreground points and background points are selected in the first frame of the image sequence to be estimated in a point selection manner; the SAM2 model generates a 2D mask of the target object in the image based on the foreground points; then the SAM2 uses its time sequence module to automatically generate a series of 2D masks of the target object in the subsequent frame images in the image sequence to be estimated, combining the inter-frame changes and the previous frame mask, until the 2D mask image generated by processing the last input image is saved in order, the name remains the same as the input image name, so that the mask corresponds to the image one by one.

[0053] S102: load the CAD model corresponding to the target object in the image sequence to be estimated, and obtain the edge box vertex of the CAD model;

[0054] In the specific implementation of the present application, the loading of the CAD model corresponding to the target object in the image sequence to be estimated and the obtaining of the edge box vertex of the CAD model include: loading the CAD model corresponding to the target object in the image sequence to be estimated, and using an edge box extraction function library to obtain the 3D edge box corresponding to the CAD model; based on the 3D edge box corresponding to the CAD, the edge box vertex of the CAD model is obtained.

[0055] Specifically, a CAD model corresponding to a target object in a to-be-estimated image sequence is loaded first, and then a 3D bounding box corresponding to the CAD model is obtained by using a bounding box extraction function library; the bounding box vertices of the CAD model are obtained through the 3D bounding box corresponding to the CAD; and in this embodiment, there are eight bounding box vertices.

[0056] In S103, projection operation is performed based on the bounding box vertices and camera intrinsic parameters corresponding to a camera for collecting the to-be-estimated image sequence, so as to obtain a pose mask corresponding to the target object in each to-be-estimated image in the to-be-estimated image sequence.

[0057] In the specific implementation of the present application, the projection operation based on the bounding box vertices and camera intrinsic parameters corresponding to a camera for collecting the to-be-estimated image sequence comprises: obtaining a pose matrix of the CAD model, and obtaining an estimated pose of the target object in each to-be-estimated image in the to-be-estimated image sequence by inputting the to-be-estimated image sequence into a pose estimation model; performing calculation and processing based on the estimated pose of the target object in each to-be-estimated image in the to-be-estimated image sequence and the pose matrix, so as to obtain a pose representation of the target object in each to-be-estimated image in the to-be-estimated image sequence in a camera coordinate system; performing calculation based on the pose representation of the target object in each to-be-estimated image in the to-be-estimated image sequence in the camera coordinate system, so as to obtain the position of each bounding box vertex in the camera coordinate system; performing projection operation on the position of each bounding box vertex in the camera coordinate system to the image coordinate system, so as to obtain each bounding box vertex projected to the image coordinate system; and performing convex hull operation on each bounding box vertex projected to the image coordinate system, so as to obtain the pose mask corresponding to the target object in each to-be-estimated image in the to-be-estimated image sequence, wherein the pose mask is the region surrounded after the convex hull operation.

[0058] Further, the calculation formula of the projection operation of the position of each bounding box vertex in the camera coordinate system to the image coordinate system is as follows:

[0059]

[0060] wherein, Z x , z y , Z c are the coordinates of the bounding box vertices of the target object in the camera coordinate system; K is the intrinsic parameter of the camera; and u, v are the position points in the image coordinate system.

[0061] Further, the formula of the convex hull operation is as follows:

[0062]

[0063] wherein, P iLet λ be the i-th border vertex; n = 1, 2, 3, ..., 8; λ i CH(P) represents the weight coefficient corresponding to the i-th bounding box vertex. i ) is convex and contains i bounding vertices in pixel coordinate system.

[0064] Specifically, the first step is to obtain the pose matrix (Pose) of the CAD model. origin By performing matrix multiplication between the pose matrix and the output of the pose estimation model, the pose representation of the target object in the camera coordinate system in each image can be obtained. The pose estimation model output is obtained by inputting the image sequence to be estimated into the model, thus outputting the estimated pose of the target object in each frame of the image sequence. Specifically, it is expressed as follows: The Pose est To estimate the pose of the target object within each frame of the image to be estimated, Pose is used in the model's output. cam This represents the pose of the object in the camera coordinate system.

[0065] Let M be the pose of the target object in the camera coordinate system in each image to be estimated. j Border vertex P i =(x i ,y i ,z i ,1) T Then the position L of the bounding box vertex in the camera coordinate system j =M j P i Next, the vertices of the bounding box in the camera coordinate system are projected onto the image coordinate system to obtain their 2D positions. By performing convex hull operations on the eight bounding box vertices projected onto the image coordinate system, a pose mask containing pose information is obtained. The pose mask is the region enclosed by the convex hull operation. The projection process is as follows:

[0066]

[0067] Among them, Z x Z y Z c , where are the coordinates of the vertices of the target object's bounding box in the camera coordinate system; K is the camera's intrinsic parameter; u and v are the position points in the image coordinate system, respectively.

[0068] The convex hull process is represented as:

[0069]

[0070] Among them, P i Let λ be the i-th border vertex; n = 1, 2, 3, ..., 8; λ iis the weight coefficient corresponding to the i-th edge frame vertex; CH(P i ) is convex and contains i edge frame vertices in the pixel coordinate system.

[0071] After obtaining the pose of the target object, the relationship between the pose of the target object and the pose mask corresponding thereto can refer to Figure 5 .

[0072] S104: Perform pose accuracy verification processing according to the relationship between each 2D mask in the 2D mask sequence and the corresponding pose mask.

[0073] In the specific implementation of the present application, the pose accuracy verification processing according to the relationship between each 2D mask in the 2D mask sequence and the corresponding pose mask includes: calculating the intersection between each 2D mask in the 2D mask sequence and the corresponding pose mask to obtain the intersection calculation result between each 2D mask and the corresponding pose mask; calculating the area ratio between each 2D mask and the corresponding 2D mask based on the intersection calculation result between each 2D mask and the corresponding pose mask, and performing pose accuracy verification processing according to the calculated area ratio.

[0074] Specifically, first, the position relationship between the 2D mask and the pose mask needs to be obtained (for reference Figure 6 ), the intersection between the 2D mask and the pose mask is calculated, and the ratio between the area and the 2D mask area is calculated. The value is the pose verification score, which is represented as

[0075] where S pose is the pose mask, S mask is the 2D mask; the higher the score, the better the quality of the current estimated pose. The verification threshold is set to 0.8, and when the verification result is greater than 0.8, it is considered that the estimated pose is correct and usable; when the verification result is less than 0.8, it is considered that the quality of the current output pose is poor, and the input image at this time is re-performed pose estimation process. The specific motion sequence pose verification score can refer to Figure 7 .

[0076] In the embodiment of the present application, the output posture result of the posture estimation model is evaluated without real posture reference, and the better posture and the poor posture in the output result are distinguished. The specific model is not required, and the result of the posture estimation model can be quickly checked. The checking can be performed simultaneously with the output result of the model. The posture estimation model can be provided with the ability to find the posture estimation error in the output posture in real time, and the posture with the estimation error can be additionally processed. Therefore, the accuracy of the output result of the posture estimation model can be improved. The output posture is checked by using the mask information, and the output posture quality can be basically evaluated without the labeled posture in the actual use stage. Therefore, the quality of the output posture can be distinguished.

[0077] Embodiment two, please refer to Figure 2 , Figure 2 is a flowchart of the output posture accuracy verification method of the posture estimation model in another embodiment of the present application.

[0078] As Figure 2 indicated, an output posture accuracy verification method of a posture estimation model comprises the following steps.

[0079] S201: obtaining a to-be-estimated image sequence, and converting a target object in the to-be-estimated image sequence into a 2D mask image sequence based on a mask conversion model, wherein the 2D mask image sequence corresponds to the to-be-estimated image sequence one by one;

[0080] S202: loading a CAD model corresponding to the target object in the to-be-estimated image sequence, and obtaining a bounding box vertex of the CAD model;

[0081] S203: obtaining a posture matrix of the CAD model, and obtaining an estimated posture of the target object in each frame of the to-be-estimated image sequence by inputting the to-be-estimated image sequence into a posture estimation model;

[0082] S204: performing calculation and processing based on the estimated posture of the target object in each frame of the to-be-estimated image sequence and the posture matrix, to obtain a pose representation of the target object in each frame of the to-be-estimated image sequence in a camera coordinate system;

[0083] S205: performing calculation based on the pose representation of the target object in each frame of the to-be-estimated image sequence in the camera coordinate system, to obtain a position of each bounding box vertex in the camera coordinate system;

[0084] S206: Projecting the position of each edge frame vertex in the camera coordinate system to the image coordinate system to obtain each edge frame vertex projected to the image coordinate system;

[0085] S207: Obtaining the pose mask corresponding to the target object in each of the to-be-estimated image sequence by performing convex hull operation on each edge frame vertex projected to the image coordinate system, wherein the pose mask is the region enclosed after the convex hull operation;

[0086] S208: Performing pose accuracy verification processing according to the relationship between each 2D mask in the 2D mask sequence and the corresponding pose mask.

[0087] The specific implementation of the second embodiment is described above, and will not be repeated here.

[0088] The third embodiment is described above Figure 3 , Figure 3 is a structural composition schematic diagram of the output pose accuracy verification device of the pose estimation model in the embodiment of the application.

[0089] As shown in Figure 3 , an output pose accuracy verification device of a pose estimation model, the device comprises:

[0090] The conversion module 301 is configured to obtain a to-be-estimated image sequence, and convert a target object in the to-be-estimated image sequence into a 2D mask image sequence based on a mask conversion model, wherein the 2D mask image sequence corresponds to the to-be-estimated image sequence one by one.

[0091] In the specific implementation of the application, the conversion of the target object in the to-be-estimated image sequence into the 2D mask image sequence based on the mask conversion model comprises: inputting the to-be-estimated image sequence into the mask conversion model, and selecting corresponding foreground points and background points in the first frame of the to-be-estimated image sequence in a selected point manner, wherein the mask conversion model is a SAM2 model; the mask conversion model generates a 2D mask of the target object according to the foreground points in the first frame of the to-be-estimated image; and the mask conversion model automatically performs mask generation processing for the to-be-estimated images in the subsequent frames of the to-be-estimated image sequence in sequence according to the 2D mask formed by the inter-frame changes between the to-be-estimated image sequence and the previous frame by using a time sequence module, to form a 2D mask image sequence.

[0092] Specifically, first, the to-be-estimated image sequence needs to be obtained, and the to-be-estimated image sequence is input into a mask conversion model, which is a SAM2 (Segment Anything Model 2) model; the SAM2 model supports general segmentation of images and videos, has core capabilities such as zero-shot generalization, interactive prompting, and real-time processing; by optimizing the architecture and introducing a spatiotemporal memory mechanism, compared with the previous SAM model, the segmentation accuracy is improved and the processing speed is accelerated by 6 times, becoming an important tool in the fields of autonomous driving, film production, AR / VR, etc.; at this time, the corresponding foreground points and background points are selected in the first frame of the to-be-estimated image of the to-be-estimated image sequence by using a point selection method; the SAM2 model generates a 2D mask of the target object in the foreground point generation graph; then the SAM2 uses its time sequence module to automatically generate a series of 2D masks of the target object in the subsequent frame images of the to-be-estimated image sequence in combination with the inter-frame changes and the previous frame mask until the 2D mask image generated by processing the last input image is saved in order, and the name remains the same as that of the input image, so that the mask corresponds to the image one by one.

[0093] The loading module 302 is configured to load a CAD model corresponding to the target object in the to-be-estimated image sequence, and obtain a bounding box vertex of the CAD model.

[0094] In the specific implementation process of the present application, the loading of the CAD model corresponding to the target object in the to-be-estimated image sequence and the obtaining of the bounding box vertex of the CAD model include: loading the CAD model corresponding to the target object in the to-be-estimated image sequence, and using a bounding box extraction function library to obtain a 3D bounding box corresponding to the CAD model; and obtaining the bounding box vertex of the CAD model based on the 3D bounding box corresponding to the CAD.

[0095] Specifically, the CAD model corresponding to the target object in the to-be-estimated image sequence is first loaded, and then the 3D bounding box corresponding to the CAD model is obtained by using a bounding box extraction function library; the bounding box vertex of the CAD model is obtained through the 3D bounding box corresponding to the CAD; in this embodiment, there are 8 bounding box vertices.

[0096] The projection operation module 303 is configured to perform projection operation based on the bounding box vertex and camera intrinsic parameters corresponding to a camera for collecting the to-be-estimated image sequence, to obtain a pose mask corresponding to the target object in each to-be-estimated image in the to-be-estimated image sequence.

[0097] In the implementation of the present application, the projection operation based on the frame top vertex and the camera internal parameter corresponding to the camera collecting the image sequence to be estimated comprises: obtaining a pose matrix of the CAD model, and obtaining an estimated pose of a target object in each frame of the image sequence to be estimated by inputting the image sequence to be estimated into a pose estimation model; performing calculation and processing based on the estimated pose of the target object in each frame of the image sequence to be estimated and the pose matrix, to obtain a pose representation of the target object in each frame of the image sequence to be estimated in a camera coordinate system; performing calculation based on the pose representation of the target object in each frame of the image sequence to be estimated in the camera coordinate system, to obtain a position of each frame top vertex in the camera coordinate system; performing projection operation of the position of each frame top vertex in the camera coordinate system to an image coordinate system, to obtain each frame top vertex projected to the image coordinate system; and performing convex hull operation on each frame top vertex projected to the image coordinate system, to obtain a pose mask corresponding to the target object in each frame of the image sequence to be estimated, wherein the pose mask is a region surrounded after the convex hull operation.

[0098] Further, the calculation formula of the projection operation of the position of each frame top vertex in the camera coordinate system to the image coordinate system is as follows:

[0099]

[0100] wherein, Z x , Z y , and Z c are coordinates of the frame top vertex of the target object in the camera coordinate system; K is an internal parameter of the camera; u and v are position points in the image coordinate system.

[0101] Further, the formula of the convex hull operation is as follows:

[0102]

[0103] wherein, P i is the i-th frame top vertex; n = 1, 2, 3, …, 8; λ i is a weight coefficient corresponding to the i-th frame top vertex; CH(P i ) is a convex region containing the i-th frame top vertex in the pixel coordinate system.

[0104] Specifically, first, an estimated pose and a pose matrix Pose originThe pose matrix is multiplied by the pose estimation model output to obtain the pose representation of the target object in the camera coordinate system in each image. The pose estimation model output is obtained by inputting the image sequence to be estimated into the pose estimation model, and then outputting the estimated pose corresponding to the target object in each frame of the image sequence to be estimated. Specifically, the pose representation of the target object in the camera coordinate system in each frame of the image sequence to be estimated is represented as: Here, Pose est is the estimated pose of the target object in each frame of the image to be estimated output by the estimation model, and Pose cam is the pose representation of the object in the camera coordinate system.

[0105] Let the estimated pose of the target object in the camera coordinate system in each image to be estimated be M j , and the top vertex P i of the bounding box be (x i , y i , z i , 1) T . Then the position L j of the top vertex of the bounding box in the camera coordinate system is M j P i . The 2D position of the top vertex of the bounding box in the camera coordinate system is obtained by projecting it onto the image coordinate system, and the pose mask containing the pose information is obtained by performing convex hull operation on the eight top vertices of the bounding box projected onto the image coordinate system. The pose mask is the area surrounded after the convex hull operation. The projection process is as follows:

[0106]

[0107] where Z x , Z y , and Z c are the coordinates of the top vertices of the bounding box of the target object in the camera coordinate system; K is the intrinsic parameter of the camera; and u and v are the positions of the points in the image coordinate system.

[0108] The convex hull process is represented as:

[0109]

[0110] where P i is the i-th top vertex of the bounding box; n = 1, 2, 3, …, 8; λ i is the weight coefficient corresponding to the i-th top vertex of the bounding box; and CH(P i ) is the convex hull containing the i-th top vertex in the pixel coordinate system.

[0111] After obtaining the pose of the target object, the relationship between the pose of the target object and its corresponding pose mask can be referred to Figure 5 .

[0112] The verification module 304 is configured to perform pose accuracy verification processing according to a relationship between each 2D mask in the 2D mask sequence and the corresponding pose mask.

[0113] In the implementation of the present application, the pose accuracy verification processing according to the relationship between each 2D mask in the 2D mask sequence and the corresponding pose mask includes: calculating an intersection between each 2D mask in the 2D mask sequence and the corresponding pose mask to obtain an intersection calculation result between each 2D mask and the corresponding pose mask; calculating an area ratio between the corresponding 2D mask based on the intersection calculation result between each 2D mask and the corresponding pose mask, and performing pose accuracy verification processing according to the calculated area ratio.

[0114] Specifically, first, the position relationship between the 2D mask and the pose mask needs to be obtained (for reference Figure 6 ), the intersection between the 2D mask and the pose mask is calculated, and the ratio between the area of the intersection and the area of the 2D mask is calculated. The value is the pose verification score, which is represented as

[0115] where S pose is the pose mask, and S mask is the 2D mask. The higher the score, the better the quality of the current estimated pose. The verification threshold is set to 0.8. When the verification result is greater than 0.8, it is considered that the estimated pose is correct and usable. When the verification result is less than 0.8, it is considered that the quality of the current output pose is poor. The input image at this time is re-estimated for the pose, and the specific motion sequence pose verification score can be referred to in Figure 7 .

[0116] In the embodiment of the present application, the good and bad degree of the pose estimation model output pose result is evaluated without real pose reference, and the better pose and the worse pose in the output result are distinguished. The specific model can be quickly integrated into any pose estimation model to verify the result. The verification can be performed at the same time as the model output result. For the pose estimation model, the ability to find the pose estimation error in the output pose in real time can be given, and then the pose with estimation error is additionally processed. Therefore, the accuracy of the pose estimation model output result can be improved. The mask information is used to verify the output pose, which can be used in the actual use stage without labeled pose reference. The output pose quality can be basically evaluated, so that the quality of the output pose can be distinguished.

[0117] This invention provides a computer-readable storage medium storing a computer program. When executed by a processor, this program implements the output attitude accuracy verification method of any of the above embodiments. The computer-readable storage medium includes, but is not limited to, any type of disk (including floppy disk, hard disk, optical disk, CD-ROM, and magneto-optical disk), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic cards, or optical cards. In other words, the storage device includes any medium that stores or transmits information in a readable form by a device (e.g., a computer, a mobile phone), and can be a read-only memory, a disk, or an optical disk, etc.

[0118] This invention also provides a computer application running on a computer, which is used to execute the output posture accuracy verification method of any of the above embodiments.

[0119] also, Figure 4 This is a schematic diagram of the structural composition of the electronic device in an embodiment of the present invention.

[0120] This invention also provides an electronic device, such as... Figure 4 As shown. The electronic device includes components such as a processor 402, a memory 403, an input unit 404, and a display unit 405. Those skilled in the art will understand that... Figure 4 The structural components of the illustrated electronic device do not constitute a limitation on all devices and may include more or fewer components than illustrated, or combine certain components. Memory 403 can be used to store application program 401 and various functional modules. Processor 402 runs application program 401 stored in memory 403, thereby performing various functional applications and data processing of the device. Memory can be internal memory or external memory, or both. Internal memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, or random access memory. External memory may include hard disks, floppy disks, ZIP disks, USB flash drives, magnetic tapes, etc. The memory disclosed in this invention includes, but is not limited to, these types of memory. The memory disclosed in this invention is only an example and not a limitation.

[0121] The input unit 404 is configured to receive input of signals and receive a keyword input by a user. The input unit 404 can include a touch panel and other input devices. The touch panel can collect a touch operation (e.g., an operation of a user using a finger, a stylus, or any suitable object or accessory near the touch panel) of the user on or near the touch panel and drive a corresponding connection device according to a preset program; the other input devices can include, but are not limited to, one or more of a physical keyboard, function keys (e.g., play control keys, switch keys, etc.), a trackball, a mouse, a joystick, etc. The display unit 405 can be configured to display information input by the user or information provided to the user and various menus of the terminal device. The display unit 405 can take the form of a liquid crystal display, an organic light-emitting diode, etc. The processor 402 is a control center of the terminal device, connects all parts of the entire device through various interfaces and lines, and performs various functions and processes data by running or executing software programs and / or modules stored in the memory 403 and calling data stored in the memory.

[0122] As an embodiment, the electronic device includes: one or more processors 402, a memory 403, and one or more application programs 401, wherein the one or more application programs 401 are stored in the memory 403 and configured to be executed by the one or more processors 402, and the one or more application programs 401 are configured to perform the corresponding output pose accuracy verification method in any one of the above embodiments.

[0123] In the embodiments of the present application, the output pose result of the pose estimation model is evaluated without a real pose reference, and the better pose and the worse pose in the output result are distinguished; the specific model is not dependent, and the result of any pose estimation model can be quickly integrated for verification; and the verification can be performed at the same time as the model output result, and the pose estimation model can be endowed with the ability to find the pose estimation error in the output pose in real time, and then the pose with estimation error is additionally processed; therefore, the accuracy of the output result of the pose estimation model can be improved; the output pose is verified by using the mask information, which can be used in the actual use stage without labeled pose comparison, and a basic evaluation method for the quality of the output pose is provided, so that the quality of the output pose can be distinguished.

[0124] In addition, the output posture accuracy verification method of the posture estimation model provided by the embodiment of the present application and the related device are described in detail above, the principle and implementation mode of the present application are described by using specific examples in this paper, and the above embodiment is only used to help understand the method of the present application and its core idea; at the same time, for the general technical personnel in the art, according to the idea of the present application, the specific implementation mode and the application range will be changed, and the above description should not be understood as the limitation of the present application.

Claims

1. A method for verifying the accuracy of the output attitude of an attitude estimation model, characterized in that, The method includes: A sequence of images to be estimated is obtained, and the target objects in the sequence of images to be estimated are converted into a 2D mask image sequence based on a mask transformation model. The 2D mask image sequence corresponds one-to-one with the sequence of images to be estimated. Load the CAD model corresponding to the target object in the image sequence to be estimated, and obtain the bounding vertices of the CAD model; Projection operations are performed based on the bounding box vertices and the camera intrinsic parameters corresponding to the camera that acquired the image sequence to be estimated, to obtain the pose mask corresponding to the target object in each image to be estimated in the image sequence to be estimated; The attitude accuracy is verified based on the relationship between each 2D mask in the 2D mask sequence and its corresponding attitude mask. The attitude accuracy verification process based on the relationship between each 2D mask in the 2D mask sequence and its corresponding attitude mask includes: Calculate the intersection between each 2D mask in the 2D mask sequence and its corresponding attitude mask to obtain the intersection calculation result between each 2D mask and its corresponding attitude mask; The area ratio between each 2D mask and its corresponding 2D mask is calculated based on the intersection of the two 2D mask and the corresponding 2D mask. The attitude accuracy is then verified based on the calculated area ratio.

2. The output attitude accuracy verification method according to claim 1, characterized in that, The process of converting the target object in the image sequence to be estimated into a 2D masked image sequence based on the mask transformation model includes: The image sequence to be estimated is input into the mask transformation model, and the corresponding foreground and background points are selected in the first frame of the image sequence to be estimated using a point selection method. The mask transformation model is the SAM2 model. The mask conversion model generates a 2D mask for the target object based on the foreground points in the first frame of the image to be estimated. The mask conversion model uses a timing module to automatically generate masks for the images to be estimated in subsequent frames of the image sequence based on the inter-frame changes between the images to be estimated and the 2D mask formed by the previous frame, thus forming a 2D mask image sequence.

3. The output attitude accuracy verification method according to claim 1, characterized in that, The step of loading the CAD model corresponding to the target object in the image sequence to be estimated and obtaining the bounding vertices of the CAD model includes: Load the CAD model corresponding to the target object in the image sequence to be estimated, and use the bounding box extraction function library to obtain the 3D bounding box corresponding to the CAD model; The vertices of the CAD model's border are obtained based on the 3D border corresponding to the CAD model.

4. The output attitude accuracy verification method according to claim 1, characterized in that, The projection operation based on the bounding box vertices and the camera intrinsic parameters corresponding to the camera that acquired the image sequence to be estimated includes: The pose matrix of the CAD model is obtained, and the estimated pose of the target object in each frame of the image to be estimated is obtained by inputting the image sequence to be estimated into the pose estimation model. Based on the estimated pose of the target object in each frame of the image to be estimated in the image to be estimated sequence and the pose matrix, calculation processing is performed to obtain the pose representation of the target object in each frame of the image to be estimated sequence in the camera coordinate system; The position of each bounding vertex in the camera coordinate system is calculated based on the pose representation of the target object in each frame of the image to be estimated in the image sequence to be estimated. Project the position of each border vertex in the camera coordinate system to the image coordinate system to obtain each border vertex projected into the image coordinate system. By performing convex hull operation on each border vertex projected onto the image coordinate system, the pose mask corresponding to the target object in each image to be estimated in the image sequence to be estimated is obtained, and the pose mask is the region enclosed by the convex hull operation.

5. The output attitude accuracy verification method according to claim 4, characterized in that, The calculation formula for projecting the position of each border vertex in the camera coordinate system to the image coordinate system is as follows: ; in, , , These are the coordinates of the vertices of the target object's bounding box in the camera coordinate system; For the camera's internal parameters; , These are the position points in the image coordinate system.

6. The output attitude accuracy verification method according to claim 4, characterized in that, The formula for the convex hull operation is as follows: ; in, For the first Each border vertex; ; For the first The weight coefficients corresponding to each border vertex; Convex and containing the pixel coordinate system Each border vertex.

7. A device for verifying the output attitude accuracy of an attitude estimation model, characterized in that, The device includes: The conversion module is used to obtain the image sequence to be estimated and convert the target objects in the image sequence to be estimated into a 2D mask image sequence based on the mask conversion model. The 2D mask image sequence corresponds one-to-one with the image sequence to be estimated. Loading module: used to load the CAD model corresponding to the target object in the image sequence to be estimated, and obtain the bounding vertices of the CAD model; Projection operation module: used to perform projection operation based on the bounding box vertices and the camera intrinsic parameters corresponding to the camera that acquired the image sequence to be estimated, to obtain the pose mask corresponding to the target object in each image to be estimated in the image sequence to be estimated; Verification module: used to perform attitude accuracy verification processing based on the relationship between each 2D mask in the 2D mask sequence and the corresponding attitude mask; The attitude accuracy verification process based on the relationship between each 2D mask in the 2D mask sequence and its corresponding attitude mask includes: Calculate the intersection between each 2D mask in the 2D mask sequence and its corresponding attitude mask to obtain the intersection calculation result between each 2D mask and its corresponding attitude mask; The area ratio between each 2D mask and its corresponding 2D mask is calculated based on the intersection of the two 2D mask and the corresponding 2D mask. The attitude accuracy is then verified based on the calculated area ratio.

8. An electronic device comprising a processor and a memory, characterized in that, The processor runs a computer program or code stored in the memory to implement the output attitude accuracy verification method as described in any one of claims 1 to 6.

9. A computer-readable storage medium for storing computer programs or code, characterized in that, When the computer program or code is executed by a processor, the output posture accuracy verification method as described in any one of claims 1 to 6 is implemented.

Images