Pose estimation method, system, terminal and storage medium

Abstract

The application provides a pose estimation method, system, terminal and storage medium, the method comprises the following steps: training a feature point extraction network according to a current frame image and a last frame image of a sample object; training a pose estimation subnetwork according to image feature points and sample images; inputting an object image to be estimated into the feature point extraction network and the pose estimation subnetwork for pose estimation to obtain a pose estimation result, wherein the object image comprises a current frame image and a last frame image of a target object at any time. The pose estimation subnetwork can effectively learn the geometric relationship between adjacent frame images of the object, so that the trained pose estimation subnetwork is not only limited to the change of the object itself, but also can estimate the three-dimensional pose of the target object from the information around the object, thereby avoiding the problem of low pose estimation accuracy caused by pose estimation based on texture features.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to an attitude estimation method, system, terminal, and storage medium. Background Technology

[0002] With the development of the times, 3D vision technology has also made rapid progress, especially in fields such as robotics and VR / AR. This has placed higher demands on the 6-DoF pose estimation problem. Rigid body 6-DoF pose estimation refers to estimating the 6D pose of an object in the camera coordinate system, i.e., 3D position and 3D pose. The original object's coordinate system can be considered as the world coordinate system. The 3D position is the estimated position of the object in the world coordinate system, and the 3D pose describes the rotation of the camera relative to the origin of the world coordinate system, i.e., obtaining the RT transformation from the original object's world frame to the camera frame. A rigid body is an object that does not bend or deform and has no moving joints. The significance of rigid body 6D pose estimation lies in obtaining the precise pose of the object, supporting fine manipulation of the object, and is mainly applied in the fields of robotic grasping and augmented reality. In robotic grasping, the mainstream method is to estimate the 6D pose of a known object, and then obtain the target 6D grasping pose of the grasper. In augmented reality, virtual elements can be superimposed on objects, maintaining their relative pose as the object moves. With the maturity of technologies such as Simultaneous Localization and Mapping (SLAM), robots are already able to perform localization well in space. However, if they want to interact with objects in the environment, 6D pose estimation of the objects is a necessary technology.

[0003] Existing pose estimation methods generally rely on the texture features of an object's surface for pose estimation. However, when the surface texture features of the object to be fixed are limited, the accuracy of pose estimation is low. Summary of the Invention

[0004] The purpose of this invention is to provide an attitude estimation method, system, terminal, and storage medium, aiming to solve the problem of low accuracy in existing attitude estimation methods.

[0005] The present invention is implemented as follows: a pose estimation method, the method comprising:

[0006] The feature point extraction network is trained based on the sample images until the feature point extraction network converges. The sample images include the current frame image and the previous frame image of the sample object.

[0007] The converged feature point extraction network is used to extract features from the sample image to obtain image feature points. Then, the pose estimation sub-network is trained based on the image feature points and the sample image until the pose estimation sub-network converges.

[0008] The pose estimation result of the object image to be estimated is obtained by inputting the converged feature point extraction network and the pose estimation sub-network into the object image to be estimated. The object image to be estimated includes the current frame image and the previous frame image of the target object at any time.

[0009] Preferably, the step of training the feature point extraction network based on the sample images includes:

[0010] The current frame image and the previous frame image of the sample object are respectively input into the feature point extraction network for feature extraction to obtain the first feature point information and the second feature point information. The feature point information includes the feature point coordinates and the feature point code.

[0011] Based on the first feature point information and the second feature point information, feature point matching is performed on the current frame image and the previous frame image of the sample object to obtain a feature matching value, which is used to characterize the degree of matching between feature points;

[0012] The network loss of the feature point extraction network is determined based on the feature matching value to obtain a first loss value, and the parameters of the feature point extraction network are updated based on the first loss value.

[0013] Preferably, the step of performing feature point matching on the current frame image and the previous frame image of the sample object based on the first feature point information and the second feature point information to obtain feature matching values ​​includes:

[0014] Calculate the positional error between each feature point and the true feature point in the current frame image and the previous frame image of the sample object, respectively, and filter each feature point according to the positional error;

[0015] The matching relationship of feature points between the current frame image and the previous frame image of the sample object is determined based on the filtered feature points.

[0016] Based on the matching relationship of the feature points, feature points are matched between the current frame image and the previous frame image of the sample object to obtain the feature matching value.

[0017] Preferably, the formula used for feature point matching of the current frame image and the previous frame image of the sample object based on the feature point matching relationship includes:

[0018]

[0019]

[0020] L pull It is the first sub-matching value, L pulsh It is the second sub-matching value, p tk It is the feature encoding of the current frame image of the sample object, p bk It is the feature encoding of the previous frame image of the sample object, p k It is p tk With p bk The mean between the two, and the number of feature points in the current frame image and the previous frame image of the sample object is N;

[0021] Calculate the first sub-match value between the current frame image and the previous frame image;

[0022] Calculate the second sub-matching value between the current frame image and the previous frame image;

[0023] The first and second sub-matching values ​​are used as loss values ​​to train the feature point extraction network.

[0024] Preferably, training the pose estimation subnetwork based on the image feature points and the sample images includes:

[0025] The current frame image and the previous frame image of the sample object are respectively input into the pose estimation sub-network for feature extraction to obtain a feature map, and the image feature points are superimposed on the feature map.

[0026] Based on the feature map after feature overlay, a matrix transformation is performed between the current frame image and the previous frame image of the sample object to obtain a first model matrix, a second model matrix, and a third model matrix. The model matrix includes rotational and translational attitudes.

[0027] Based on the first model matrix and the second model matrix, the projection loss of the current frame image and the previous frame image of the sample object is calculated to obtain the projection loss value;

[0028] Based on the first model matrix, the second model matrix, and the third model matrix, the inverse consistency of the current frame image of the sample object is calculated to obtain the inverse value. The inverse value is used to characterize the consistency of the labeled points and predicted points between the current frame image of the sample object and the previous frame image of the sample object after the inverse transformation by the third model matrix.

[0029] The network loss of the attitude estimation subnetwork is determined based on the projection loss value and the inverse value, a second loss value is obtained, and the parameters of the attitude estimation subnetwork are updated based on the second loss value.

[0030] Preferably, the formula used to calculate the projection loss between the current frame image and the previous frame image of the sample object includes:

[0031] L cp1 =smooth_mse(view_port*project*view*(R1*P 3D +T1), p gt1 )

[0032] L cp2 =smooth_mse(view_port*project*view*(R2*P 3D +T2), p gt2 )

[0033] `view_port × project × view` represents the preset image projection process, where `view_port` is the viewport matrix, `project` is the projection matrix, and `view` is the view matrix. P 3D It is related to setting the 3D coordinates of the object, p gt1 p represents the annotation point of the current frame image of the sample object. gt2 The annotation points represent the previous frame image of the sample object, R1 is the rotational pose in the first model matrix, T1 is the translational pose in the first model matrix, R2 is the rotational pose in the second model matrix, T2 is the translational pose in the second model matrix, and L... cp1 This is the first sub-projection loss value, L cp2 It is the second sub-projection loss value;

[0034] The projection loss value includes the first sub-projection loss value and the second sub-projection loss value.

[0035] Preferably, the formula used to calculate the inverse consistency of the current frame image of the sample object and obtain the inverse value includes:

[0036] L consist1 =smooth_mse(view_port*project*view*M 12 *(R1*P 3D +T1), p gt2 )

[0037] L consist2 =smooth_mse(view_port*project*view*M 21 (R2*P 3D +T2), p gt1 )

[0038] M 12 =R 12 T 12 M 21 =M 12 -1

[0039] L consistl It is the first sub-inverse value, which is used to characterize the consistency of the annotation points between the current frame image of the sample object after inverse transformation by the third model matrix and the previous frame image of the sample object. L consist1 This is the second sub-inverse value, which characterizes the consistency of the annotation points between the current frame image of the sample object and the current frame image of the sample object after the inverse transformation by the third model matrix. R 12 It is the rotational attitude in the third model matrix, T 12 It is the translational attitude in the third model matrix;

[0040] L consist12 =smooth_mse(p predict12 , view_port*project*view*(R2*P 3D +T2))

[0041] p predict12 =view_port*project*view*M 12 *(R1*P 3D +T1)

[0042] L consist21 =smooth_mse(p predict21 , view_port*project*view*(R1*P 3D +T1))

[0043] p predict21 =view_port*project*view*M 21 (R2*P 3D +T2)

[0044] L consist12 It is the third sub-inverse value, which characterizes the consistency of predicted points between the current frame image of the sample object after inverse transformation by the third model matrix and the previous frame image of the sample object. consistl It is the fourth sub-inverse value, which is used to characterize the consistency of the predicted points between the current frame image of the sample object and the current frame image of the sample object after the inverse transformation by the third model matrix.

[0045] The inverter value includes the first sub-inverter value, the second sub-inverter value, the third sub-inverter value, and the fourth sub-inverter value.

[0046] Another objective of this invention is to provide an attitude estimation system, the system comprising:

[0047] The first training module is used to train a feature point extraction network based on sample images until the feature point extraction network converges. The sample images include the current frame image and the previous frame image of the sample object.

[0048] The second training module is used to extract features from the sample image based on the converged feature point extraction network to obtain image feature points, and to train the pose estimation sub-network based on the image feature points and the sample image until the pose estimation sub-network converges.

[0049] The pose estimation module is used to input the image of the object to be estimated into the converged feature point extraction network and the pose estimation sub-network to perform pose estimation, and obtain the pose estimation result of the image of the object to be estimated. The image of the object to be estimated includes the current frame image and the previous frame image of the target object at any time.

[0050] Another objective of this invention is to provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0051] Another objective of this invention is to provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0052] In this embodiment of the invention, a pose estimation sub-network is trained using image feature points and sample images. This allows the training of the pose estimation sub-network to be more focused on the regions where feature points are located, improving the accuracy of the trained network. By training the pose estimation sub-network based on the current frame image and the previous frame image of the sample object, the network can effectively learn the geometric relationships between adjacent frames. This means that the trained pose estimation sub-network is not limited to changes in the object itself, but can also estimate the three-dimensional pose of the target object based on information from its surroundings. This avoids the problem of low pose estimation accuracy caused by pose estimation based on texture features. Attached Figure Description

[0053] Figure 1 This is a flowchart of the attitude estimation method provided in the first embodiment of the present invention;

[0054] Figure 2 This is a flowchart of the attitude estimation method provided in the second embodiment of the present invention;

[0055] Figure 3 This is a schematic diagram of the attitude estimation system provided in the third embodiment of the present invention;

[0056] Figure 4 This is a schematic diagram of the structure of the terminal device provided in the fourth embodiment of the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0058] To illustrate the technical solution described in this invention, specific embodiments are described below.

[0059] Example 1

[0060] Please see Figure 1 This is a flowchart of the attitude estimation method provided in the first embodiment of the present invention. The attitude estimation method can be applied to any terminal device or system. The attitude estimation method includes the following steps:

[0061] Step S10: Train the feature point extraction network based on the sample images until the feature point extraction network converges;

[0062] The sample images include the current frame image and the previous frame image of the sample object. By training the feature point extraction network, the subsequent pose estimation sub-network training is effectively facilitated, making the network training more focused on the region where the feature points are located, thus improving the accuracy of the pose estimation sub-network after training.

[0063] Optionally, in this step, training the feature point extraction network based on the sample images includes:

[0064] The current frame image and the previous frame image of the sample object are respectively input into the feature point extraction network for feature extraction to obtain the first feature point information and the second feature point information; wherein, the feature point information includes feature point coordinates, feature point encoding and feature encoding similarity. There are N feature points in the current frame image and the previous frame image of the sample object. In this step, the feature point information corresponding to the N feature points is obtained respectively. Preferably, the feature point coordinates are normalized coordinate values.

[0065] Based on the first feature point information and the second feature point information, feature point matching is performed on the current frame image and the previous frame image of the sample object to obtain a feature matching value; wherein, the feature matching value is used to characterize the degree of matching between feature points;

[0066] The network loss of the feature point extraction network is determined based on the feature matching value to obtain a first loss value, and the parameters of the feature point extraction network are updated based on the first loss value.

[0067] Further, in this step, the step of performing feature point matching on the current frame image and the previous frame image of the sample object based on the first feature point information and the second feature point information to obtain feature matching values ​​includes:

[0068] Calculate the positional error between each feature point and the true feature point in the current frame image and the previous frame image of the sample object, respectively, and filter each feature point according to the positional error;

[0069] The formula used to calculate the positional error between each feature point and the true feature point in the current frame image and the previous frame image of the sample object is as follows:

[0070]

[0071] Among them, l k This represents the position of the predicted feature point in the current frame image and the previous frame image of the sample object, l k ’ L represents the position of the true feature points in the current frame and the previous frame of the sample object. p This refers to the positional error. In this step, if any positional error exceeds the error threshold, the feature point corresponding to that positional error will be deleted. The error threshold can be set as a parameter value according to requirements.

[0072] The matching relationship of feature points between the current frame image and the previous frame image of the sample object is determined based on the filtered feature points.

[0073] Based on the matching relationship of the feature points, feature points are matched between the current frame image and the previous frame image of the sample object to obtain the feature matching value.

[0074] Furthermore, the formula used for feature point matching of the current frame image and the previous frame image of the sample object based on the feature point matching relationship includes:

[0075]

[0076]

[0077] Lpull It is the first sub-matching value, L pulsh It is the second sub-matching value, p tk It is the feature encoding of the current frame image of the sample object, p bk It is the feature encoding of the previous frame image of the sample object, p k It is p tk With p bk The mean between the two, and the number of feature points in the current frame image and the previous frame image of the sample object is N;

[0078] Calculate the first sub-match value between the current frame image and the previous frame image;

[0079] Calculate the second sub-matching value between the current frame image and the previous frame image;

[0080] The first and second sub-matching values ​​are used as loss values ​​to train the feature point extraction network;

[0081] In this step, when feature points of the sample object in the current frame image and the previous frame image are matched, L... pull To minimize the distance between their feature codes, when the feature points of the current frame image and the previous frame image of the sample object do not match, L... pulsh Make the distance between their feature codes as large as possible.

[0082] Optionally, in this step, via L loss =αL H +βL pull +δL push The loss function is calculated for the feature point sub-network, and the parameters of the feature point extraction network are updated by gradient backpropagation until the feature point extraction network converges. α, β, and δ represent the adjustment coefficients of each loss.

[0083] Step S20: Extract features from the sample image using the converged feature point extraction network to obtain image feature points, and train a pose estimation sub-network based on the image feature points and the sample image until the pose estimation sub-network converges.

[0084] In this process, the current frame and the previous frame of the sample image are input into the feature point extraction network and the pose estimation sub-network. The feature point extraction network obtains image feature points, and then the image feature points are weighted onto the feature map of the pose estimation sub-network. This allows the pose estimation sub-network to pay more attention to the information of the location of the feature points, thereby improving the accuracy of the pose estimation sub-network.

[0085] Step S30: Input the image of the object to be estimated into the converged feature point extraction network and the pose estimation sub-network to perform pose estimation, and obtain the pose estimation result of the image of the object to be estimated.

[0086] The image of the object to be estimated includes the current frame image and the previous frame image of the target object at any given time. In this step, the user only needs to input the RGB image of the relationship between the two frames of the target object to accurately estimate the pose change of the target object from the previous frame to the image of the object to be estimated, and obtain the pose estimation result.

[0087] In this embodiment, the pose estimation sub-network is trained using image feature points and sample images. This allows the training process to effectively focus the network training on the regions where feature points are located, improving the accuracy of the pose estimation sub-network after training. By training the pose estimation sub-network based on the current frame image and the previous frame image of the sample object, the pose estimation sub-network can effectively learn the geometric relationship between the object in adjacent frame images. This means that the trained pose estimation sub-network is not limited to the changes of the object itself, but can also estimate the three-dimensional pose of the target object based on the information around the object, avoiding the problem of low pose estimation accuracy caused by pose estimation based on texture features.

[0088] Example 2

[0089] Please see Figure 2 This is a flowchart of the attitude estimation method provided in the second embodiment of the present invention. This embodiment is used to further refine step S20 in the first embodiment, including the following steps:

[0090] Step S21: Input the current frame image and the previous frame image of the sample object into the pose estimation sub-network for feature extraction to obtain a feature map, and then superimpose the image feature points onto the feature map.

[0091] In this process, the current frame and the previous frame of the sample image are input into the feature point extraction network and the pose estimation sub-network. The feature point extraction network obtains the image feature points, and then the image feature points are superimposed on the feature map of the pose estimation sub-network. This allows the pose estimation sub-network to pay more attention to the information of the location of the feature points, thereby improving the accuracy of the pose estimation sub-network.

[0092] Step S22: Based on the feature map after feature overlay, perform matrix transformation between the current frame image and the previous frame image of the sample object to obtain the first model matrix, the second model matrix and the third model matrix;

[0093] In the pose estimation subnetwork, the 3D coordinates of any 3D object are predefined as (0, 0, 0), (1, 1, 1), (1, 1, -1), (1, -1, 1), (1, -1, -1), (-1, -1, -1), (-1, -1, 1), (-1, 1, -1), (-1, 1, -1), (-1, 1, 1). The image projection process is defined as p = view_port × project × view × model, where view_port is the viewport matrix, project is the projection matrix, view is the view matrix, and model is the model matrix. The image is input into the predefined deep neural network structure, and the output is the model matrix, i.e., the rotation pose and translation pose. The rotation of the model matrix is ​​represented by quaternions, and the value range is [-1, 1]. The pose estimation subnetwork regresses the rotation pose R1 and translation pose T1 of the current frame image, the rotation pose R2 and translation pose T2 of the previous frame image, and the rotation transformation R of the sample object from the previous frame image to the current frame image. 21 Translation T 21 ;

[0094] Step S23: Based on the first model matrix and the second model matrix, calculate the projection loss of the current frame image and the previous frame image of the sample object to obtain the projection loss value;

[0095] Optionally, in this step, the formula used to calculate the projection loss of the current frame image and the previous frame image of the sample object includes:

[0096] L cp1 =smooth_mse(view_port*project*view*(R1*P 3D +T1), p gt1 )

[0097] L cp2 =smooth_mse(view_pot*project*view*(R2*P 3D +T2), p gt2 )

[0098] `view_port × project × view` represents the preset image projection process, where `view_port` is the viewport matrix, `project` is the projection matrix, and `view` is the view matrix. P 3D It is related to setting the 3D coordinates of the object, p gt1 p represents the annotation point of the current frame image of the sample object. gt2The annotation points represent the previous frame image of the sample object, R1 is the rotational pose in the first model matrix, T1 is the translational pose in the first model matrix, R2 is the rotational pose in the second model matrix, T2 is the translational pose in the second model matrix, and L... cp1 This is the first sub-projection loss value, L cp2 It is the second sub-projection loss value. smooth_mse indicates that the smoothed mse is used to solve the problem of differences between manually marked points and actual projected points. The projection loss value includes the first sub-projection loss value and the second sub-projection loss value.

[0099] Step S24: Calculate the inverse consistency of the current frame image of the sample object based on the first model matrix, the second model matrix, and the third model matrix to obtain the inverse value;

[0100] Among them, the inverse value is used to characterize the consistency of the labeled points and predicted points between the current frame image of the sample object and the current frame image and the previous frame image of the sample object after the inverse transformation by the third model matrix.

[0101] Optionally, in this step, the formula used to calculate the inverse consistency of the current frame image of the sample object and obtain the inverse value includes:

[0102] L consist1 =smooth_mse(view_port*project*view*M 12 *(R1*P 3D +T1), p gt2 )

[0103] L consist2 =smooth_mse(view_port*project*view*M 21 (R2*P 3D +T2), p gt1 )

[0104] M 12 =R 12 T 12 M 21 =M 12 -1

[0105] L consistl It is the first sub-inverse value, which is used to characterize the consistency of the annotation points between the current frame image of the sample object after inverse transformation by the third model matrix and the previous frame image of the sample object. L consist1This is the second sub-inverse value, which characterizes the consistency of the annotation points between the current frame image of the sample object and the current frame image of the sample object after the inverse transformation by the third model matrix. R 12 It is the rotational attitude in the third model matrix, T 12 It is the translational attitude in the third model matrix;

[0106] L consist12 =smooth_mse(p predict12 , view_port*project*view*(R2*P 3D +T2))

[0107] p predict12 =view_port*project*view*M 12 *(R1*P 3D +T1)

[0108] L consist21 =smooth_mse(p predict21 , view_port*project*view*(R1*P 3D +T1))

[0109] p predict21 =view_port*project*view*M 21 (R2*P 3D +T2)

[0110] L consist12 It is the third sub-inverse value, which characterizes the consistency of predicted points between the current frame image of the sample object after inverse transformation by the third model matrix and the previous frame image of the sample object. consistl It is the fourth sub-inverse value, which is used to characterize the consistency of the predicted points between the current frame image of the sample object and the current frame image of the sample object after the inverse transformation by the third model matrix.

[0111] The inverter value includes the first sub-inverter value, the second sub-inverter value, the third sub-inverter value, and the fourth sub-inverter value.

[0112] Step S25: Determine the network loss of the attitude estimation subnetwork based on the projection loss value and the inverse value to obtain a second loss value, and update the parameters of the attitude estimation subnetwork based on the second loss value;

[0113] Optionally, in this step, the formula used to determine the network loss of the attitude estimation subnetwork based on the projection loss value and the inverse value is:

[0114] L loss =α(L cp1 +L cp2 )+β(L consist1 +L consist2 )+δ(L consist12 +L consist21 )

[0115] In this embodiment, by superimposing image feature points onto the feature map, the pose estimation subnetwork can pay more attention to the information of the location of feature points, thereby improving the accuracy of the pose estimation subnetwork. By performing matrix transformation between the current frame image and the previous frame image of the sample object through the feature map after feature superposition, the first model matrix, the second model matrix, and the third model matrix can be effectively obtained. Based on the first model matrix, the second model matrix, and the third model matrix, the projection loss value and the inverse value can be effectively calculated. Based on the calculated projection loss value and the inverse value, the network loss of the pose estimation subnetwork can be effectively calculated, thereby improving the accuracy of the pose estimation subnetwork training.

[0116] Example 3

[0117] Please see Figure 3 This is a schematic diagram of the structure of the attitude estimation system 100 provided in the third embodiment of the present invention, including: a first training module 10, a second training module 11, and an attitude estimation module 12, wherein:

[0118] The first training module 10 is used to train a feature point extraction network based on sample images until the feature point extraction network converges. The sample images include the current frame image and the previous frame image of the sample object.

[0119] Optionally, the first training module 10 is further configured to: input the current frame image and the previous frame image of the sample object into the feature point extraction network for feature extraction, and obtain first feature point information and second feature point information, wherein the feature point information includes feature point coordinates and feature point encoding;

[0120] Based on the first feature point information and the second feature point information, feature point matching is performed on the current frame image and the previous frame image of the sample object to obtain a feature matching value, which is used to characterize the degree of matching between feature points;

[0121] The network loss of the feature point extraction network is determined based on the feature matching value to obtain a first loss value, and the parameters of the feature point extraction network are updated based on the first loss value.

[0122] Furthermore, the first training module 10 is also used to: calculate the positional error between each feature point and the real feature point in the current frame image and the previous frame image of the sample object, respectively, and filter each feature point according to the positional error;

[0123] The matching relationship of feature points between the current frame image and the previous frame image of the sample object is determined based on the filtered feature points.

[0124] Based on the matching relationship of the feature points, feature points are matched between the current frame image and the previous frame image of the sample object to obtain the feature matching value.

[0125] Furthermore, the formula used for feature point matching of the current frame image and the previous frame image of the sample object based on the feature point matching relationship includes:

[0126]

[0127]

[0128] L pull It is the first sub-matching value, L pulsh It is the second sub-matching value, p tk It is the feature encoding of the current frame image of the sample object, p bk It is the feature encoding of the previous frame image of the sample object, p k It is p tk With p bk The mean between the two, and the number of feature points in the current frame image and the previous frame image of the sample object is N;

[0129] Calculate the first sub-match value between the current frame image and the previous frame image;

[0130] Calculate the second sub-matching value between the current frame image and the previous frame image;

[0131] The first and second sub-matching values ​​are used as loss values ​​to train the feature point extraction network.

[0132] The second training module 11 is used to extract features from the sample image based on the converged feature point extraction network to obtain image feature points, and to train a pose estimation sub-network based on the image feature points and the sample image until the pose estimation sub-network converges.

[0133] Optionally, the second training module 12 is further configured to: input the current frame image and the previous frame image of the sample object into the pose estimation sub-network for feature extraction to obtain a feature map, and superimpose the image feature points onto the feature map;

[0134] Based on the feature map after feature overlay, a matrix transformation is performed between the current frame image and the previous frame image of the sample object to obtain a first model matrix, a second model matrix, and a third model matrix. The model matrix includes rotational and translational attitudes.

[0135] Based on the first model matrix and the second model matrix, the projection loss of the current frame image and the previous frame image of the sample object is calculated to obtain the projection loss value;

[0136] Based on the first model matrix, the second model matrix, and the third model matrix, the inverse consistency of the current frame image of the sample object is calculated to obtain the inverse value. The inverse value is used to characterize the consistency of the labeled points and predicted points between the current frame image of the sample object and the previous frame image of the sample object after the inverse transformation by the third model matrix.

[0137] The network loss of the attitude estimation subnetwork is determined based on the projection loss value and the inverse value, a second loss value is obtained, and the parameters of the attitude estimation subnetwork are updated based on the second loss value.

[0138] Furthermore, the formula used to calculate the projection loss of the current frame image and the previous frame image of the sample object includes:

[0139] L cp1 =smooth_mse(view_port*project*view*(R1*P 3D +T1), p gt1 )

[0140] L cp2 =smooth_mse(view_port*project*view*(R2*P 3D +T2), p gt2 )

[0141] `view_port × project × view` represents the preset image projection process, where `view_port` is the viewport matrix, `project` is the projection matrix, and `view` is the view matrix. P 3D It is related to setting the 3D coordinates of the object, p gt1 p represents the annotation point of the current frame image of the sample object. gt2 The annotation points represent the previous frame image of the sample object, R1 is the rotational pose in the first model matrix, T1 is the translational pose in the first model matrix, R2 is the rotational pose in the second model matrix, T2 is the translational pose in the second model matrix, and L... cp1This is the first sub-projection loss value, L cp2 It is the second sub-projection loss value;

[0142] The projection loss value includes the first sub-projection loss value and the second sub-projection loss value.

[0143] Furthermore, the formula used to calculate the inverse consistency of the current frame image of the sample object and obtain the inverse value includes:

[0144] L consist1 =smooth_mse(view_port*project*view*M 12 *(R1*P 3D +T1), p gt2 )

[0145] L consist2 =smooth_mse(view_port*project*view*M 21 (R2*P 3D +T2), p gt1 )

[0146] M 12 =R 12 T 12 M 21 =M 12 -1

[0147] L consist1 It is the first sub-inverse value, which is used to characterize the consistency of the annotation points between the current frame image of the sample object after inverse transformation by the third model matrix and the previous frame image of the sample object. L consistl This is the second sub-inverse value, which characterizes the consistency of the annotation points between the current frame image of the sample object and the current frame image of the sample object after the inverse transformation by the third model matrix. R 12 It is the rotational attitude in the third model matrix, T 12 It is the translational attitude in the third model matrix;

[0148] L consist12 =smooth_mse(p predict12 , view_port*project*view*(R2*P 3D +T2))

[0149] p predict12 =view_port*project*view*M 12 *(R1*P 3D +T1)

[0150] L consist21 =smooth_mse(p predict21 , view_port*project*view*(R1*P 3D +T1))

[0151] p predict21 =view_port*project*view*M 21 (R2*P 3D +T2)

[0152] L consist12 It is the third sub-inverse value, which characterizes the consistency of predicted points between the current frame image of the sample object after inverse transformation by the third model matrix and the previous frame image of the sample object. consistl It is the fourth sub-inverse value, which is used to characterize the consistency of the predicted points between the current frame image of the sample object and the current frame image of the sample object after the inverse transformation by the third model matrix.

[0153] The inverter value includes the first sub-inverter value, the second sub-inverter value, the third sub-inverter value, and the fourth sub-inverter value.

[0154] The pose estimation module 12 is used to input the image of the object to be estimated into the converged feature point extraction network and the pose estimation sub-network to perform pose estimation, and obtain the pose estimation result of the image of the object to be estimated. The image of the object to be estimated includes the current frame image and the previous frame image of the target object at any time.

[0155] In this embodiment, the pose estimation sub-network is trained using image feature points and sample images. This allows the training process to effectively focus the network training on the regions where feature points are located, improving the accuracy of the pose estimation sub-network after training. By training the pose estimation sub-network based on the current frame image and the previous frame image of the sample object, the pose estimation sub-network can effectively learn the geometric relationship between the object in adjacent frame images. This means that the trained pose estimation sub-network is not limited to the changes of the object itself, but can also estimate the three-dimensional pose of the target object based on the information around the object, avoiding the problem of low pose estimation accuracy caused by pose estimation based on texture features.

[0156] Example 4

[0157] Figure 4 This is a structural block diagram of a terminal device 2 provided in the fourth embodiment of this application. For example... Figure 4As shown, the terminal device 2 in this embodiment includes a processor 20, a memory 21, and a computer program 22 stored in the memory 21 and executable on the processor 20, such as a program for an attitude estimation method. When the processor 20 executes the computer program 22, it implements the steps in the various embodiments of the attitude estimation methods described above.

[0158] For example, the computer program 22 may be divided into one or more modules, which are stored in the memory 21 and executed by the processor 20 to complete this application. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 22 in the terminal device 2. The terminal device may include, but is not limited to, the processor 20 and the memory 21.

[0159] The processor 20 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0160] The memory 21 can be an internal storage unit of the terminal device 2, such as a hard drive or memory of the terminal device 2. The memory 21 can also be an external storage device of the terminal device 2, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the terminal device 2. Furthermore, the memory 21 can include both internal and external storage units of the terminal device 2. The memory 21 is used to store the computer program and other programs and data required by the terminal device. The memory 21 can also be used to temporarily store data that has been output or will be output.

[0161] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0162] If an integrated module is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. This computer-readable storage medium can be non-volatile or volatile. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable storage medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the contents of a computer-readable storage medium may be appropriately added to or subtracted from the contents as required by the legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, a computer-readable storage medium may not include electrical carrier signals and telecommunication signals.

[0163] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A pose estimation method, characterized in that, The method includes: The feature point extraction network is trained based on the sample images until the feature point extraction network converges. The sample images include the current frame image and the previous frame image of the sample object. The converged feature point extraction network is used to extract features from the sample image to obtain image feature points. Then, the pose estimation sub-network is trained based on the image feature points and the sample image until the pose estimation sub-network converges. After the input of the image of the object to be estimated converges, the feature point extraction network and the pose estimation sub-network perform pose estimation to obtain the pose estimation result of the image of the object to be estimated. The image of the object to be estimated includes the current frame image and the previous frame image of the target object at any time. The step of training a feature point extraction network based on sample images includes: The current frame image and the previous frame image of the sample object are respectively input into the feature point extraction network for feature extraction to obtain the first feature point information and the second feature point information. The feature point information includes the feature point coordinates and the feature point code. Based on the first feature point information and the second feature point information, feature point matching is performed on the current frame image and the previous frame image of the sample object to obtain a feature matching value, which is used to characterize the degree of matching between feature points; The network loss of the feature point extraction network is determined based on the feature matching value to obtain a first loss value, and the parameters of the feature point extraction network are updated based on the first loss value. The step of training a pose estimation subnetwork based on the image feature points and the sample images includes: The current frame image and the previous frame image of the sample object are respectively input into the pose estimation sub-network for feature extraction to obtain a feature map, and the image feature points are superimposed on the feature map. Based on the feature map after feature overlay, a matrix transformation is performed between the current frame image and the previous frame image of the sample object to obtain a first model matrix, a second model matrix, and a third model matrix. The model matrix includes rotational and translational attitudes. Based on the first model matrix and the second model matrix, the projection loss of the current frame image and the previous frame image of the sample object is calculated to obtain the projection loss value; Based on the first model matrix, the second model matrix, and the third model matrix, the inverse consistency of the current frame image of the sample object is calculated to obtain the inverse value. The inverse value is used to characterize the consistency of the labeled points and predicted points between the current frame image of the sample object and the previous frame image of the sample object after the inverse transformation by the third model matrix. The network loss of the attitude estimation subnetwork is determined based on the projection loss value and the inverse value, a second loss value is obtained, and the parameters of the attitude estimation subnetwork are updated based on the second loss value.

2. The attitude estimation method as described in claim 1, characterized in that, The step of performing feature point matching on the current frame image and the previous frame image of the sample object based on the first feature point information and the second feature point information to obtain feature matching values ​​includes: Calculate the positional error between each feature point and the true feature point in the current frame image and the previous frame image of the sample object, and filter each feature point according to the positional error; The matching relationship of feature points between the current frame image and the previous frame image of the sample object is determined based on the filtered feature points. Based on the matching relationship of the feature points, feature points are matched between the current frame image and the previous frame image of the sample object to obtain the feature matching value.

3. The attitude estimation method as described in claim 2, characterized in that, The formula used to perform feature point matching between the current frame image and the previous frame image of the sample object based on the feature point matching relationship includes: L pull It is the first sub-match value. L push It is the second sub-match value. p tk It is the feature encoding of the current frame image of the sample object. p bk It is the feature encoding of the previous frame image of the sample object. p k yes p tk and p bk The mean between the two, and the number of feature points in the current frame image and the previous frame image of the sample object is N; Calculate the first sub-match value between the current frame image and the previous frame image; Calculate the second sub-matching value between the current frame image and the previous frame image; The first and second sub-matching values ​​are used as loss values ​​to train the feature point extraction network.

4. The attitude estimation method as described in claim 1, characterized in that, The formula used to calculate the projection loss of the current frame image and the previous frame image of the sample object includes: `view_port×project×view` represents the preset image projection process, where `view_port` is the viewport matrix, `project` is the projection matrix, and `view` is the view matrix. P 3D It is related to setting the 3D coordinates of an object. p gt1 The annotation points represent the current frame image of the sample object. p gt2 The annotation points represent the previous frame image of the sample object, R1 is the rotational pose in the first model matrix, T1 is the translational pose in the first model matrix, R2 is the rotational pose in the second model matrix, and T2 is the translational pose in the second model matrix. L cp1 This is the first sub-projection loss value. L cp2 It is the second sub-projection loss value; The projection loss value includes the first sub-projection loss value and the second sub-projection loss value.

5. The attitude estimation method as described in claim 4, characterized in that, The formula used to calculate the inverse consistency of the current frame image of the sample object and obtain the inverse value includes: M 12 =R 12 T 12 ,M 21 =M 12 -1 L consist1 This is the first sub-inverse value, which characterizes the consistency of the annotation points between the current frame image of the sample object after inverse transformation by the third model matrix and the previous frame image of the sample object. L consist2 This is the second sub-inverse value, which characterizes the consistency of the annotation points between the current frame image of the sample object and the current frame image of the sample object after the inverse transformation by the third model matrix. R 12 It is the rotational attitude in the third model matrix, T 12 It is the translational attitude in the third model matrix; L consist12 This is the third sub-inverse value, which characterizes the consistency of predicted points between the current frame image of the sample object after inverse transformation by the third model matrix and the previous frame image of the sample object. L consist21 It is the fourth sub-inverse value, which is used to characterize the consistency of the predicted points between the current frame image of the sample object and the current frame image of the sample object after the inverse transformation by the third model matrix. The inverter value includes the first sub-inverter value, the second sub-inverter value, the third sub-inverter value, and the fourth sub-inverter value.

6. An attitude estimation system, characterized in that, The system includes: The first training module is used to train a feature point extraction network based on sample images until the feature point extraction network converges. The sample images include the current frame image and the previous frame image of the sample object. The second training module is used to extract features from the sample image based on the converged feature point extraction network to obtain image feature points, and to train the pose estimation sub-network based on the image feature points and the sample image until the pose estimation sub-network converges. The pose estimation module is used to input the image of the object to be estimated into the converged feature point extraction network and the pose estimation sub-network to perform pose estimation, and obtain the pose estimation result of the image of the object to be estimated. The image of the object to be estimated includes the current frame image and the previous frame image of the target object at any time. The first training module is also used to: input the current frame image and the previous frame image of the sample object into the feature point extraction network for feature extraction, and obtain the first feature point information and the second feature point information, wherein the feature point information includes feature point coordinates and feature point encoding; Based on the first feature point information and the second feature point information, feature point matching is performed on the current frame image and the previous frame image of the sample object to obtain a feature matching value, which is used to characterize the degree of matching between feature points; The network loss of the feature point extraction network is determined based on the feature matching value to obtain a first loss value, and the parameters of the feature point extraction network are updated based on the first loss value. The second training module is also used to: input the current frame image and the previous frame image of the sample object into the pose estimation sub-network for feature extraction to obtain a feature map, and then superimpose the image feature points onto the feature map; Based on the feature map after feature overlay, a matrix transformation is performed between the current frame image and the previous frame image of the sample object to obtain a first model matrix, a second model matrix, and a third model matrix. The model matrix includes rotational and translational attitudes. Based on the first model matrix and the second model matrix, the projection loss of the current frame image and the previous frame image of the sample object is calculated to obtain the projection loss value; Based on the first model matrix, the second model matrix, and the third model matrix, the inverse consistency of the current frame image of the sample object is calculated to obtain the inverse value. The inverse value is used to characterize the consistency of the labeled points and predicted points between the current frame image of the sample object and the previous frame image of the sample object after the inverse transformation by the third model matrix. The network loss of the attitude estimation subnetwork is determined based on the projection loss value and the inverse value, a second loss value is obtained, and the parameters of the attitude estimation subnetwork are updated based on the second loss value.

7. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 5.

Images