Method for verifying accuracy of output attitude of attitude estimation model and related device

Abstract

The invention discloses an output attitude accuracy verification method of an attitude estimation model and a related device, and the method comprises the steps: obtaining a to-be-estimated image sequence, and converting a target object in the to-be-estimated image sequence into a 2D mask image sequence based on a mask conversion model; loading a CAD model corresponding to a target object in the to-be-estimated image sequence, and obtaining frame vertexes of the CAD model; projection operation is carried out based on the frame vertex and a camera internal reference corresponding to a camera which collects the to-be-estimated image sequence, and an attitude mask corresponding to a target object in each to-be-estimated image in the to-be-estimated image sequence is obtained; and performing attitude accuracy verification processing according to the relationship between the 2D masks in the 2D mask sequence and the corresponding attitude masks. In the embodiment of the invention, under the condition of no real attitude reference, the evaluation on the quality of the attitude result output by the attitude estimation model is realized, and the method can be quickly integrated into any attitude estimation model to verify the result.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for verifying the accuracy of an output posture of a posture estimation model and a related device. Background Art

[0002] 6-DoF Object Pose Estimation is an important research direction in the fields of computer vision and robotics. Its core goal is to determine the position (x, y, z) and orientation (roll, pitch, yaw) of an object in three-dimensional space. When using the Foundationpose pose estimation model for continuous pose estimation of moving objects, it was found that the model was unable to perceive the incorrect pose in the current output pose during the continuous estimation process. The incorrect pose is reflected in the offset of the estimated pose position xyz compared to the actual position, and the subsequent offset will become larger and larger as the estimation proceeds, and the pose will also be inconsistent with reality. As a result, in the actual use of the model, if the output results begin to deviate, all subsequent results will be incorrect poses and cannot be used. Summary of the Invention

[0003] The purpose of the present invention is to overcome the shortcomings of the prior art. The present invention provides a method and related devices for verifying the accuracy of the output posture of a posture estimation model, which can evaluate the quality of the posture results output by the posture estimation model without a real posture reference, and can be quickly integrated into any posture estimation model to verify its results.

[0004] In order to solve the above technical problems, an embodiment of the present invention provides a method for verifying the accuracy of an output posture of a posture estimation model, the method comprising:

[0005] Obtaining an image sequence to be estimated, and converting a target object in the image sequence to be estimated into a 2D mask image sequence based on a mask conversion model, wherein the 2D mask image sequence corresponds one-to-one to the image sequence to be estimated;

[0006] Loading a CAD model corresponding to the target object in the image sequence to be estimated, and obtaining the bounding box vertices of the CAD model;

[0007] Performing a projection operation based on the bounding box vertices and the camera intrinsic parameters corresponding to the camera that collected the image sequence to be estimated, to obtain a posture mask corresponding to the target object in each image to be estimated in the image sequence to be estimated;

[0008] Performing posture accuracy verification processing according to the relationship between each 2D mask in the 2D mask sequence and the corresponding posture mask.

[0009] Optionally, converting the target object in the image sequence to be estimated into a 2D mask image sequence based on a mask conversion model includes:

[0010] Inputting the image sequence to be estimated into the mask transfer model, and selecting corresponding foreground points and background points in the first frame of the image to be estimated in the image sequence to be estimated by a point selection method, wherein the mask transfer model is a SAM2 model;

[0011] The mask conversion model generates a 2D mask of the target object according to the foreground point in the first frame of the image to be estimated;

[0012] The mask conversion model utilizes a timing module to automatically perform mask generation processing for the images to be estimated in subsequent frames in the image sequence to be estimated according to the inter-frame changes between the image sequence to be estimated and the 2D mask formed in the previous frame, thereby forming a 2D mask image sequence.

[0013] Optionally, the loading of a CAD model corresponding to the target object in the image sequence to be estimated and obtaining bounding box vertices of the CAD model includes:

[0014] Loading the CAD model corresponding to the target object in the image sequence to be estimated, and using a border extraction function library to obtain the 3D border corresponding to the CAD model;

[0015] The border vertices of the CAD model are obtained based on the 3D border corresponding to the CAD model.

[0016] Optionally, performing a projection operation based on the bounding box vertices and camera intrinsic parameters corresponding to a camera that acquires the image sequence to be estimated includes:

[0017] Obtaining a pose matrix of the CAD model, and obtaining an estimated pose corresponding to a target object in each frame of the image to be estimated in the image sequence to be estimated by inputting the image sequence to be estimated into a pose estimation model;

[0018] Performing calculations based on the estimated pose corresponding to the target object in each frame of the image to be estimated in the image sequence to be estimated and the pose matrix to obtain a pose representation of the target object in each frame of the image to be estimated in the image sequence to be estimated in the camera coordinate system;

[0019] Calculate the position of each bounding box vertex in the camera coordinate system based on the pose representation of the target object in each frame of the image to be estimated in the image sequence to be estimated;

[0020] Projecting the position of each border vertex in the camera coordinate system to the image coordinate system to obtain each border vertex projected to the image coordinate system;

[0021] By performing a convex hull operation on each vertex of the bounding box projected into the image coordinate system, a posture mask corresponding to the target object in each image to be estimated in the image sequence to be estimated is obtained, and the posture mask is the area enclosed by the convex hull operation.

[0022] Optionally, the calculation formula for projecting the position of each border vertex in the camera coordinate system to the image coordinate system is as follows:

[0023]

[0024] Among them, Z x , Z y , Z c are the coordinates of the bounding box vertices of the target object in the camera coordinate system; K is the intrinsic parameter of the camera; u and v are the position points in the image coordinate system.

[0025] Optionally, the formula for the convex hull operation is as follows:

[0026]

[0027] Among them, P i is the i-th border vertex; n=1,2,3,…,8;λ i is the weight coefficient corresponding to the i-th border vertex; CH(P i ) is convex and contains i bounding box vertices in pixel coordinates.

[0028] Optionally, performing posture accuracy verification processing according to the relationship between each 2D mask in the 2D mask sequence and the corresponding posture mask includes:

[0029] Calculate the intersection between each 2D mask in the 2D mask sequence and the corresponding pose mask, and obtain the intersection calculation result between each 2D mask and the corresponding pose mask;

[0030] Based on the intersection calculation result between each 2D mask and the corresponding posture mask, the area ratio between the mask and the corresponding 2D mask is calculated, and the posture accuracy verification processing is performed according to the calculated area ratio.

[0031] In addition, an embodiment of the present invention further provides a device for verifying the accuracy of an output posture of a posture estimation model, the device comprising:

[0032] A conversion module is used to obtain an image sequence to be estimated, and convert the target object in the image sequence to be estimated into a 2D mask image sequence based on a mask conversion model, wherein the 2D mask image sequence corresponds one-to-one with the image sequence to be estimated;

[0033] A loading module is used to load the CAD model corresponding to the target object in the image sequence to be estimated and obtain the border vertices of the CAD model;

[0034] A projection operation module is configured to perform a projection operation based on the bounding box vertices and the camera intrinsic parameters corresponding to the camera that acquires the image sequence to be estimated, so as to obtain a posture mask corresponding to the target object in each image to be estimated in the image sequence to be estimated;

[0035] Verification module: used for performing posture accuracy verification processing according to the relationship between each 2D mask in the 2D mask sequence and the corresponding posture mask.

[0036] In addition, an embodiment of the present invention further provides an electronic device, including a processor and a memory, wherein the processor runs a computer program or code stored in the memory to implement the output posture accuracy verification method as described in any one of the above.

[0037] In addition, an embodiment of the present invention further provides a computer-readable storage medium for storing a computer program or code. When the computer program or code is executed by a processor, the output posture accuracy verification method as described in any one of the above is implemented.

[0038] In an embodiment of the present invention, it is possible to evaluate the quality of the posture results output by the posture estimation model without a real posture reference, and distinguish between good and bad postures in the output results; it can be independent of a specific model and can be quickly integrated into any posture estimation model to verify its results; and it can be verified while the model outputs the results. For the posture estimation model, it can be given the ability to detect posture estimation errors in the output posture in real time, and then perform additional processing on the erroneous posture; therefore, the accuracy of the output results of the posture estimation model can be improved; the output posture is verified using mask information, which can be used in the actual use stage without labeled posture comparison, to provide a basic evaluation method for the output posture quality, thereby being able to distinguish the quality of the output posture. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 1 is a flow chart of a method for verifying the accuracy of an output posture of a posture estimation model in an embodiment of the present invention;

[0041] Figure 2 is a flow chart of a method for verifying the accuracy of an output posture of a posture estimation model in another embodiment of the present invention;

[0042] Figure 3 Schematic diagram of the structure of the output posture accuracy verification device of the posture estimation model in an embodiment of the present invention;

[0043] Figure 4 is a schematic diagram of the structure of an electronic device in an embodiment of the present invention;

[0044] Figure 5 Schematic diagram of the relationship between the object posture and its posture mask in an embodiment of the present invention;

[0045] Figure 6 Schematic diagram of the positional relationship between the 2D mask and the posture mask in an embodiment of the present invention;

[0046] Figure 7 It is a motion sequence posture verification score graph in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0048] For example 1, please refer to Figure 1 , Figure 1 4 is a flow chart of a method for verifying the accuracy of an output posture of a posture estimation model in an embodiment of the present invention.

[0049] like Figure 1 As shown, a method for verifying the accuracy of an output posture of a posture estimation model, the method comprising:

[0050] S101: obtaining an image sequence to be estimated, and converting a target object in the image sequence to be estimated into a 2D mask image sequence based on a mask conversion model, wherein the 2D mask image sequence corresponds one-to-one to the image sequence to be estimated;

[0051] In a specific implementation process of the present invention, the target object in the image sequence to be estimated is converted into a 2D mask image sequence based on a mask conversion model, including: inputting the image sequence to be estimated into the mask conversion model, and selecting corresponding foreground points and background points in the first frame of the image sequence to be estimated by a point selection method, wherein the mask conversion model is a SAM2 model; the mask conversion model generates a 2D mask of the target object according to the foreground points in the first frame of the image to be estimated; the mask conversion model uses a timing module to automatically perform mask generation processing for the images to be estimated in subsequent frames in the image sequence to be estimated according to the inter-frame changes between the image sequence to be estimated and the 2D mask formed by the previous frame to form a 2D mask image sequence.

[0052] Specifically, the first step is to obtain the image sequence to be estimated and input the image sequence to be estimated into the mask conversion model, which is the SAM2 (Segment Anything Model 2) model; it supports universal segmentation of images and videos, and has core capabilities such as zero-sample generalization, interactive prompts, and real-time processing; by optimizing the architecture and introducing a spatiotemporal memory mechanism, compared with the previous generation SAM model, the segmentation accuracy is improved and the processing speed is accelerated by 6 times, making it an important tool in the fields of autonomous driving, film and television production, AR / VR, etc.; at this time, it is necessary to use a point selection method to select the corresponding foreground and background points in the first frame of the image sequence to be estimated; the SAM2 model will generate a 2D mask of the target object in the image based on the foreground point; then, through SAM2's timing module, combined with the inter-frame changes and the previous frame mask, it automatically generates a series of 2D masks for the target objects in the subsequent frames of the image sequence to be estimated, until the 2D mask images generated after processing the last input image are saved in sequence, and the name is consistent with the name of its input image, so that the mask and image correspond one to one.

[0053] S102: loading a CAD model corresponding to the target object in the image sequence to be estimated, and obtaining vertices of a bounding box of the CAD model;

[0054] In a specific implementation of the present invention, the step of loading the CAD model corresponding to the target object in the image sequence to be estimated and obtaining the border vertices of the CAD model includes: loading the CAD model corresponding to the target object in the image sequence to be estimated, and using a border extraction function library to obtain the 3D border corresponding to the CAD model; and obtaining the border vertices of the CAD model based on the 3D border corresponding to the CAD model.

[0055] Specifically, first load the CAD model corresponding to the target object in the image sequence to be estimated, then use the border extraction function library to obtain the 3D border corresponding to the CAD model; then obtain the border vertices of the CAD model through the 3D border corresponding to the CAD; in this embodiment, there are 8 border vertices.

[0056] S103: performing a projection operation based on the bounding box vertices and the camera intrinsic parameters corresponding to the camera that collected the image sequence to be estimated, to obtain a posture mask corresponding to the target object in each image to be estimated in the image sequence to be estimated;

[0057] In a specific implementation of the present invention, the projection operation based on the bounding box vertices and the camera intrinsic parameters corresponding to the camera that collects the image sequence to be estimated includes: obtaining a pose matrix of the CAD model, and obtaining an estimated pose corresponding to the target object in each frame of the image to be estimated in the image sequence to be estimated by inputting the image sequence to be estimated into a pose estimation model; performing calculations based on the estimated pose corresponding to the target object in each frame of the image to be estimated in the image sequence to be estimated and the pose matrix to obtain a pose representation of the target object in each frame of the image to be estimated in the image sequence to be estimated in a camera coordinate system; performing calculations based on the pose representation of the target object in each frame of the image to be estimated in the image sequence to be estimated in a camera coordinate system to obtain a position of each bounding box vertex in the camera coordinate system; performing a projection operation on the position of each bounding box vertex in the camera coordinate system to the image coordinate system to obtain each bounding box vertex projected into the image coordinate system; and performing a convex hull operation on each bounding box vertex projected into the image coordinate system to obtain a pose mask corresponding to the target object in each frame of the image to be estimated in the image sequence to be estimated, wherein the pose mask is an area enclosed by the convex hull operation.

[0058] Furthermore, the calculation formula for projecting the position of each border vertex in the camera coordinate system to the image coordinate system is as follows:

[0059]

[0060] Among them, Z x 、z y , Z c are the coordinates of the bounding box vertices of the target object in the camera coordinate system; K is the intrinsic parameter of the camera; u and v are the position points in the image coordinate system.

[0061] Furthermore, the formula for the convex hull operation is as follows:

[0062]

[0063] Among them, P iis the i-th border vertex; n=1,2,3,…,8;λ i is the weight coefficient corresponding to the i-th border vertex; CH(P i ) is convex and contains i bounding box vertices in pixel coordinates.

[0064] Specifically, we first need to obtain the CAD model pose matrix Pose origin , the pose matrix is ​​multiplied by the pose estimation model output result to obtain the pose representation of the target object in each picture in the camera coordinate system. The output result of the pose estimation model here is to input the image sequence to be estimated into the pose estimation model, thereby outputting the estimated pose corresponding to the target object in each frame of the image sequence to be estimated; specifically expressed as: The pose here est Pose is the estimated pose of the target object in each frame of the image to be estimated output by the estimation model. cam It represents the object's pose in the camera coordinate system.

[0065] The pose of the target object in each image to be estimated in the camera coordinate system is represented as M j , border vertex P i =(x i ,y i ,z i ,1) T , then the position L of the border vertex in the camera coordinate system j =M j P i Then, project the bounding box vertices in the camera coordinate system to the image coordinate system to obtain their 2D positions. By performing a convex hull operation on the eight bounding box vertices projected to the image coordinate system, we obtain a pose mask containing pose information. The pose mask is the area enclosed by the convex hull operation. The projection process is as follows:

[0066]

[0067] Among them, Z x , Z y , Z c are the coordinates of the bounding box vertices of the target object in the camera coordinate system; K is the intrinsic parameter of the camera; u and v are the position points in the image coordinate system.

[0068] The convex hull process is expressed as:

[0069]

[0070] Among them, P i is the i-th border vertex; n=1,2,3,…,8;λ iis the weight coefficient corresponding to the i-th border vertex; CH(P i ) is convex and contains i bounding box vertices in pixel coordinates.

[0071] After obtaining the posture of the target object, the relationship between the posture of the target object and its corresponding posture mask can be referred to Figure 5 .

[0072] S104: Performing posture accuracy verification processing according to the relationship between each 2D mask in the 2D mask sequence and the corresponding posture mask.

[0073] In the specific implementation process of the present invention, the posture accuracy verification processing is performed based on the relationship between each 2D mask in the 2D mask sequence and the corresponding posture mask, including: calculating the intersection between each 2D mask in the 2D mask sequence and the corresponding posture mask, and obtaining the intersection calculation result between each 2D mask and the corresponding posture mask; calculating the area ratio between each 2D mask and the corresponding posture mask based on the intersection calculation result between each 2D mask and the corresponding 2D mask, and performing the posture accuracy verification processing according to the calculated area ratio.

[0074] Specifically, we first need to obtain the positional relationship between the 2D mask and the pose mask (refer to Figure 6 ), calculate the intersection of the 2D mask and the pose mask, and calculate the ratio between its area and the 2D mask area. This value is the pose verification score, which is expressed as

[0075] Among them S pose is the pose mask, S mask is a 2D mask; the higher the score, the better the quality of the current estimated posture. The verification threshold is set to 0.8. When the verification result is greater than 0.8, the estimated posture is considered to be correct and usable; when the verification result is less than 0.8, the current output posture quality is considered to be poor, and the posture estimation process is repeated for the input image at this time. The specific motion sequence posture verification score can be referred to Figure 7 .

[0076] In an embodiment of the present invention, it is possible to evaluate the quality of the posture results output by the posture estimation model without a real posture reference, and distinguish between good and bad postures in the output results; it can be independent of a specific model and can be quickly integrated into any posture estimation model to verify its results; and it can be verified while the model outputs the results. For the posture estimation model, it can be given the ability to detect posture estimation errors in the output posture in real time, and then perform additional processing on the erroneous posture; therefore, the accuracy of the output results of the posture estimation model can be improved; the output posture is verified using mask information, which can be used in the actual use stage without labeled posture comparison, to provide a basic evaluation method for the output posture quality, thereby being able to distinguish the quality of the output posture.

[0077] For example 2, please refer to Figure 2 , Figure 2 4 is a flow chart of a method for verifying the accuracy of an output posture of a posture estimation model in another embodiment of the present invention.

[0078] like Figure 2 As shown, a method for verifying the accuracy of an output posture of a posture estimation model, the method comprising:

[0079] S201: Obtain an image sequence to be estimated, and convert a target object in the image sequence to be estimated into a 2D mask image sequence based on a mask conversion model, wherein the 2D mask image sequence corresponds one-to-one to the image sequence to be estimated;

[0080] S202: Loading a CAD model corresponding to the target object in the image sequence to be estimated, and obtaining border vertices of the CAD model;

[0081] S203: Obtaining a pose matrix of the CAD model, and obtaining an estimated pose corresponding to the target object in each frame of the image to be estimated in the image sequence to be estimated by inputting the image sequence to be estimated into a pose estimation model;

[0082] S204 performs calculations based on the estimated pose corresponding to the target object in each frame of the image to be estimated in the image sequence and the pose matrix to obtain a pose representation of the target object in each frame of the image to be estimated in the image sequence in the camera coordinate system;

[0083] S205: Calculating based on the pose representation of the target object in each frame of the image to be estimated in the image sequence to be estimated in the camera coordinate system to obtain the position of each bounding box vertex in the camera coordinate system;

[0084] S206: Projecting the position of each bounding box vertex in the camera coordinate system to the image coordinate system to obtain each bounding box vertex projected to the image coordinate system;

[0085] S207: performing a convex hull operation on each vertex of the bounding box projected into the image coordinate system to obtain a posture mask corresponding to the target object in each image to be estimated in the sequence of images to be estimated, wherein the posture mask is the area enclosed by the convex hull operation;

[0086] S208: Performing posture accuracy verification processing according to the relationship between each 2D mask in the 2D mask sequence and the corresponding posture mask.

[0087] The specific implementation of the second embodiment can be found in the above embodiment, which will not be described in detail here.

[0088] For example three, please refer to Figure 3 , Figure 3 Schematic diagram of the structure of the output posture accuracy verification device of the posture estimation model in an embodiment of the present invention.

[0089] like Figure 3 As shown, a device for verifying the accuracy of an output posture of a posture estimation model, the device comprising:

[0090] Conversion module 301: used to obtain an image sequence to be estimated, and convert the target object in the image sequence to be estimated into a 2D mask image sequence based on a mask conversion model, wherein the 2D mask image sequence corresponds one-to-one with the image sequence to be estimated;

[0091] In a specific implementation process of the present invention, the target object in the image sequence to be estimated is converted into a 2D mask image sequence based on a mask conversion model, including: inputting the image sequence to be estimated into the mask conversion model, and selecting corresponding foreground points and background points in the first frame of the image sequence to be estimated by a point selection method, wherein the mask conversion model is a SAM2 model; the mask conversion model generates a 2D mask of the target object according to the foreground points in the first frame of the image to be estimated; the mask conversion model uses a timing module to automatically perform mask generation processing for the images to be estimated in subsequent frames in the image sequence to be estimated according to the inter-frame changes between the image sequence to be estimated and the 2D mask formed by the previous frame to form a 2D mask image sequence.

[0092] Specifically, the first step is to obtain the image sequence to be estimated and input the image sequence to be estimated into the mask conversion model, which is the SAM2 (Segment Anything Model 2) model; it supports universal segmentation of images and videos, and has core capabilities such as zero-sample generalization, interactive prompts, and real-time processing; by optimizing the architecture and introducing a spatiotemporal memory mechanism, compared with the previous generation SAM model, the segmentation accuracy is improved and the processing speed is accelerated by 6 times, making it an important tool in the fields of autonomous driving, film and television production, AR / VR, etc.; at this time, it is necessary to use a point selection method to select the corresponding foreground and background points in the first frame of the image sequence to be estimated; the SAM2 model will generate a 2D mask of the target object in the image based on the foreground point; then, through SAM2's timing module, combined with the inter-frame changes and the previous frame mask, it automatically generates a series of 2D masks for the target objects in the subsequent frames of the image sequence to be estimated, until the 2D mask images generated after processing the last input image are saved in sequence, and the name is consistent with the name of its input image, so that the mask and image correspond one to one.

[0093] Loading module 302: used to load the CAD model corresponding to the target object in the image sequence to be estimated, and obtain the border vertices of the CAD model;

[0094] In a specific implementation of the present invention, the step of loading the CAD model corresponding to the target object in the image sequence to be estimated and obtaining the border vertices of the CAD model includes: loading the CAD model corresponding to the target object in the image sequence to be estimated, and using a border extraction function library to obtain the 3D border corresponding to the CAD model; and obtaining the border vertices of the CAD model based on the 3D border corresponding to the CAD model.

[0095] Specifically, first load the CAD model corresponding to the target object in the image sequence to be estimated, then use the border extraction function library to obtain the 3D border corresponding to the CAD model; then obtain the border vertices of the CAD model through the 3D border corresponding to the CAD; in this embodiment, there are 8 border vertices.

[0096] Projection operation module 303: configured to perform a projection operation based on the bounding box vertices and the camera intrinsic parameters corresponding to the camera that acquires the image sequence to be estimated, to obtain a posture mask corresponding to the target object in each image to be estimated in the image sequence to be estimated;

[0097] In a specific implementation of the present invention, the projection operation based on the bounding box vertices and the camera intrinsic parameters corresponding to the camera that collects the image sequence to be estimated includes: obtaining a pose matrix of the CAD model, and obtaining an estimated pose corresponding to the target object in each frame of the image to be estimated in the image sequence to be estimated by inputting the image sequence to be estimated into a pose estimation model; performing calculation processing based on the estimated pose corresponding to the target object in each frame of the image to be estimated in the image sequence to be estimated and the pose matrix to obtain a pose representation of the target object in each frame of the image to be estimated in the image sequence to be estimated in the camera coordinate system; performing calculation based on the pose representation of the target object in each frame of the image to be estimated in the image sequence to be estimated in the camera coordinate system to obtain a position of each bounding box vertex in the camera coordinate system; performing a projection operation on the position of each bounding box vertex in the camera coordinate system to the image coordinate system to obtain each bounding box vertex projected into the image coordinate system; and performing a convex hull operation on each bounding box vertex projected into the image coordinate system to obtain a pose mask corresponding to the target object in each frame of the image to be estimated in the image sequence to be estimated, wherein the pose mask is the area enclosed by the convex hull operation.

[0098] Furthermore, the calculation formula for projecting the position of each border vertex in the camera coordinate system to the image coordinate system is as follows:

[0099]

[0100] Among them, Z x , Z y , Z c are the coordinates of the bounding box vertices of the target object in the camera coordinate system; K is the intrinsic parameter of the camera; u and v are the position points in the image coordinate system.

[0101] Furthermore, the formula for the convex hull operation is as follows:

[0102]

[0103] Among them, P i is the i-th border vertex; n=1,2,3,…,8;λ i is the weight coefficient corresponding to the i-th border vertex; CH(P i ) is convex and contains i bounding box vertices in pixel coordinates.

[0104] Specifically, we first need to obtain the estimated pose and pose matrix Pose of the CAD model origin, the pose matrix is ​​multiplied by the pose estimation model output result to obtain the pose representation of the target object in each picture in the camera coordinate system. The output result of the pose estimation model here is to input the image sequence to be estimated into the pose estimation model, thereby outputting the estimated pose corresponding to the target object in each frame of the image sequence to be estimated; specifically expressed as: The pose here est Pose is the estimated pose of the target object in each frame of the image to be estimated output by the estimation model. cam It represents the object's pose in the camera coordinate system.

[0105] The estimated pose of the target object in each image to be estimated in the camera coordinate system is recorded as M j , border vertex P i =(x i ,y i , z i , 1) T , then the position L of the border vertex in the camera coordinate system j =M j P i Then, project the bounding box vertices in the camera coordinate system to the image coordinate system to obtain their 2D positions. By performing a convex hull operation on the eight bounding box vertices projected to the image coordinate system, we obtain a pose mask containing pose information. The pose mask is the area enclosed by the convex hull operation. The projection process is as follows:

[0106]

[0107] Among them, Z x , Z y , Z c are the coordinates of the bounding box vertices of the target object in the camera coordinate system; K is the intrinsic parameter of the camera; u and v are the position points in the image coordinate system.

[0108] The convex hull process is expressed as:

[0109]

[0110] Among them, P i is the i-th border vertex; n=1,2,3,…,8;λ i is the weight coefficient corresponding to the i-th border vertex; CH(P i ) is convex and contains i bounding box vertices in pixel coordinates.

[0111] After obtaining the posture of the target object, the relationship between the posture of the target object and its corresponding posture mask can be referred to Figure 5 .

[0112] Verification module 304 is configured to perform posture accuracy verification processing based on the relationship between each 2D mask in the 2D mask sequence and the corresponding posture mask.

[0113] In the specific implementation process of the present invention, the posture accuracy verification processing is performed based on the relationship between each 2D mask in the 2D mask sequence and the corresponding posture mask, including: calculating the intersection between each 2D mask in the 2D mask sequence and the corresponding posture mask, and obtaining the intersection calculation result between each 2D mask and the corresponding posture mask; calculating the area ratio between each 2D mask and the corresponding posture mask based on the intersection calculation result between each 2D mask and the corresponding 2D mask, and performing the posture accuracy verification processing according to the calculated area ratio.

[0114] Specifically, we first need to obtain the positional relationship between the 2D mask and the pose mask (refer to Figure 6 ), calculate the intersection of the 2D mask and the pose mask, and calculate the ratio between its area and the 2D mask area. This value is the pose verification score, which is expressed as

[0115] Among them S pose is the pose mask, S mask is a 2D mask; the higher the score, the better the quality of the current estimated posture. The verification threshold is set to 0.8. When the verification result is greater than 0.8, the estimated posture is considered to be correct and usable; when the verification result is less than 0.8, the current output posture quality is considered to be poor, and the posture estimation process is repeated for the input image at this time. The specific motion sequence posture verification score can be referred to Figure 7 .

[0116] In an embodiment of the present invention, it is possible to evaluate the quality of the posture results output by the posture estimation model without a real posture reference, and distinguish between good and bad postures in the output results; it can be independent of a specific model and can be quickly integrated into any posture estimation model to verify its results; and it can be verified while the model outputs the results. For the posture estimation model, it can be given the ability to detect posture estimation errors in the output posture in real time, and then perform additional processing on the erroneous posture; therefore, the accuracy of the output results of the posture estimation model can be improved; the output posture is verified using mask information, which can be used in the actual use stage without labeled posture comparison, to provide a basic evaluation method for the output posture quality, thereby being able to distinguish the quality of the output posture.

[0117] An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the output posture accuracy verification method of any one of the above embodiments is implemented. The computer-readable storage medium includes, but is not limited to, any type of disk (including floppy disk, hard disk, optical disk, CD-ROM, and magneto-optical disk), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic card or optical card. In other words, the storage device includes any medium that can store or transmit information in a readable form by a device (for example, a computer, a mobile phone), which can be a read-only memory, a disk or an optical disk, etc.

[0118] An embodiment of the present invention further provides a computer application program that runs on a computer and is used to execute the output posture accuracy verification method of any one of the above embodiments.

[0119] also, Figure 4 It is a schematic diagram of the structure of an electronic device in an embodiment of the present invention.

[0120] The embodiment of the present invention further provides an electronic device, such as Figure 4 The electronic device includes a processor 402, a memory 403, an input unit 404, a display unit 405 and other components. Those skilled in the art will understand that Figure 4 The structural components of the electronic device shown do not constitute a limitation on all devices, and may include more or fewer components than shown, or combine certain components. The memory 403 can be used to store the application 401 and various functional modules, and the processor 402 runs the application 401 stored in the memory 403, thereby executing various functional applications and data processing of the device. The memory can be an internal memory or an external memory, or include both internal and external memories. The internal memory may include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, or a random access memory. The external memory may include a hard disk, a floppy disk, a ZIP disk, a USB flash drive, a magnetic tape, etc. The memory disclosed in the present invention includes but is not limited to these types of memories. The memory disclosed in the present invention is only an example and not a limitation.

[0121] The input unit 404 is used to receive input signals and keywords entered by the user. The input unit 404 may include a touch panel and other input devices. The touch panel can collect user touch operations on or near it (such as operations performed by the user using a finger, stylus, or any other suitable object or accessory on or near the touch panel) and drive the corresponding connected device according to a pre-set program; other input devices may include, but are not limited to, one or more of a physical keyboard, function keys (such as playback control keys, on / off keys, etc.), a trackball, a mouse, a joystick, etc. The display unit 405 can be used to display information entered by the user or information provided to the user, as well as various menus of the terminal device. The display unit 405 can be in the form of a liquid crystal display, an organic light-emitting diode, etc. The processor 402 is the control center of the terminal device, connecting the various parts of the entire device using various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 403 and calling data stored in the memory.

[0122] As an embodiment, the electronic device includes: one or more processors 402, a memory 403, and one or more applications 401, wherein the one or more applications 401 are stored in the memory 403 and are configured to be executed by the one or more processors 402, and the one or more applications 401 are configured to execute the corresponding output posture accuracy verification method in any one of the above embodiments.

[0123] In an embodiment of the present invention, it is possible to evaluate the quality of the posture results output by the posture estimation model without a real posture reference, and distinguish between good and bad postures in the output results; it can be independent of a specific model and can be quickly integrated into any posture estimation model to verify its results; and it can be verified while the model outputs the results. For the posture estimation model, it can be given the ability to detect posture estimation errors in the output posture in real time, and then perform additional processing on the erroneous posture; therefore, the accuracy of the output results of the posture estimation model can be improved; the output posture is verified using mask information, which can be used in the actual use stage without labeled posture comparison, to provide a basic evaluation method for the output posture quality, thereby being able to distinguish the quality of the output posture.

[0124] In addition, the above is a detailed introduction to the output posture accuracy verification method and related devices of a posture estimation model provided by an embodiment of the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for verifying the accuracy of an output posture of a posture estimation model, characterized in that: The method comprises: Obtaining an image sequence to be estimated, and converting a target object in the image sequence to be estimated into a 2D mask image sequence based on a mask conversion model, wherein the 2D mask image sequence corresponds one-to-one to the image sequence to be estimated; Loading a CAD model corresponding to the target object in the image sequence to be estimated, and obtaining the bounding box vertices of the CAD model; Performing a projection operation based on the bounding box vertices and the camera intrinsic parameters corresponding to the camera that collected the image sequence to be estimated, to obtain a posture mask corresponding to the target object in each image to be estimated in the image sequence to be estimated; Performing posture accuracy verification processing according to the relationship between each 2D mask in the 2D mask sequence and the corresponding posture mask.

2. The method for verifying the accuracy of the output posture according to claim 1, wherein: The step of converting the target object in the image sequence to be estimated into a 2D mask image sequence based on a mask conversion model includes: Inputting the image sequence to be estimated into the mask transfer model, and selecting corresponding foreground points and background points in the first frame of the image to be estimated in the image sequence to be estimated by a point selection method, wherein the mask transfer model is a SAM2 model; The mask conversion model generates a 2D mask of the target object according to the foreground point in the first frame of the image to be estimated; The mask conversion model utilizes a timing module to automatically perform mask generation processing for the images to be estimated in subsequent frames in the image sequence to be estimated according to the inter-frame changes between the image sequence to be estimated and the 2D mask formed in the previous frame, thereby forming a 2D mask image sequence.

3. The method for verifying the accuracy of the output posture according to claim 1, wherein: The step of loading a CAD model corresponding to the target object in the image sequence to be estimated and obtaining border vertices of the CAD model includes: Loading the CAD model corresponding to the target object in the image sequence to be estimated, and using a border extraction function library to obtain the 3D border corresponding to the CAD model; The border vertices of the CAD model are obtained based on the 3D border corresponding to the CAD model.

4. The method for verifying the accuracy of the output posture according to claim 1, wherein: The performing of a projection operation based on the bounding box vertices and the camera intrinsic parameters corresponding to the camera that acquires the image sequence to be estimated includes: Obtaining a pose matrix of the CAD model, and obtaining an estimated pose corresponding to a target object in each frame of the image to be estimated in the image sequence to be estimated by inputting the image sequence to be estimated into a pose estimation model; Performing calculations based on the estimated pose corresponding to the target object in each frame of the image to be estimated in the image sequence to be estimated and the pose matrix to obtain a pose representation of the target object in each frame of the image to be estimated in the image sequence to be estimated in the camera coordinate system; Calculate the position of each bounding box vertex in the camera coordinate system based on the pose representation of the target object in each frame of the image to be estimated in the image sequence to be estimated; Projecting the position of each border vertex in the camera coordinate system to the image coordinate system to obtain each border vertex projected to the image coordinate system; By performing a convex hull operation on each vertex of the bounding box projected into the image coordinate system, a posture mask corresponding to the target object in each image to be estimated in the image sequence to be estimated is obtained, and the posture mask is the area enclosed by the convex hull operation.

5. The method for verifying the accuracy of the output posture according to claim 4, wherein: The calculation formula for projecting the position of each border vertex in the camera coordinate system to the image coordinate system is as follows: Among them, Z x , Z y , Z c are the coordinates of the bounding box vertices of the target object in the camera coordinate system; K is the intrinsic parameter of the camera; u and v are the position points in the image coordinate system.

6. The method for verifying the accuracy of the output posture according to claim 4, characterized in that: The formula for the convex hull operation is as follows: Among them, P i is the i-th border vertex; n=1,2,3,…,8;λ i is the weight coefficient corresponding to the i-th border vertex; CH(P i ) is convex and contains i bounding box vertices in pixel coordinates.

7. The method for verifying the accuracy of the output posture according to claim 1, wherein: The performing posture accuracy verification processing according to the relationship between each 2D mask in the 2D mask sequence and the corresponding posture mask includes: Calculate the intersection between each 2D mask in the 2D mask sequence and the corresponding pose mask, and obtain the intersection calculation result between each 2D mask and the corresponding pose mask; Based on the intersection calculation result between each 2D mask and the corresponding posture mask, the area ratio between the mask and the corresponding 2D mask is calculated, and the posture accuracy verification processing is performed according to the calculated area ratio.

8. A device for verifying the accuracy of an output posture of a posture estimation model, characterized in that: The device comprises: A conversion module is used to obtain an image sequence to be estimated, and convert the target object in the image sequence to be estimated into a 2D mask image sequence based on a mask conversion model, wherein the 2D mask image sequence corresponds one-to-one with the image sequence to be estimated; A loading module is used to load the CAD model corresponding to the target object in the image sequence to be estimated and obtain the border vertices of the CAD model; A projection operation module is configured to perform a projection operation based on the bounding box vertices and the camera intrinsic parameters corresponding to the camera that acquires the image sequence to be estimated, so as to obtain a posture mask corresponding to the target object in each image to be estimated in the image sequence to be estimated; Verification module: used for performing posture accuracy verification processing according to the relationship between each 2D mask in the 2D mask sequence and the corresponding posture mask.

9. An electronic device comprising a processor and a memory, characterized in that: The processor runs the computer program or code stored in the memory to implement the output posture accuracy verification method according to any one of claims 1 to 7.

10. A computer-readable storage medium for storing a computer program or code, characterized in that: When the computer program or code is executed by a processor, the output gesture accuracy verification method according to any one of claims 1 to 7 is implemented.

Images