Pose estimation method and related model training method, device, equipment and medium

Abstract

The application discloses a pose estimation method and a related model training method, device, equipment and medium. The pose estimation model training method comprises the following steps: obtaining a sample image containing an object to be positioned, wherein the sample image comprises a sample color image and a sample depth image corresponding to the sample color image; processing the sample color image by using a pose estimation model to obtain a sample initial pose of the object to be positioned; optimizing the sample initial pose based on depth information of the object to be positioned in the sample depth image to obtain an optimized pose of the object to be positioned; and adjusting network parameters in the pose estimation model based on a difference between the optimized pose and the sample initial pose. The above scheme can improve the efficiency of model training.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a pose estimation method and a training method, apparatus, device, and medium for related models. Background Technology

[0002] With the development of technology, it is possible to use a camera to capture images of objects that need to be located, and then use a network model to process the captured images to obtain the location of the object. Currently, a large number of labeled sample images are required to train the network model. The process of labeling sample images is very labor-intensive and time-consuming, resulting in a long training time and low training efficiency. Summary of the Invention

[0003] This application provides at least one pose estimation method and a training method, apparatus, device, and medium for related models.

[0004] This application provides a method for training a pose estimation model, comprising: acquiring a sample image containing an object to be localized, wherein the sample image includes a sample color image and a sample depth image corresponding to the sample color image; processing the sample color image using a pose estimation model to obtain an initial sample pose of the object to be localized; optimizing the initial sample pose based on the depth information of the object to be localized in the sample depth image to obtain an optimized pose of the object to be localized; and adjusting the network parameters in the pose estimation model based on the difference between the optimized pose and the initial sample pose.

[0005] Therefore, by processing the sample color image using the pose estimation model to obtain the initial pose of the object to be localized, and then optimizing the initial pose using the sample depth image, the optimized pose of the object to be localized becomes more accurate. Furthermore, by using the difference between the optimized pose and the sample pose to adjust the network parameters in the pose estimation model, there is no need to annotate the sample color image, reducing the annotation workload and improving the training efficiency of the pose estimation model.

[0006] Specifically, based on the depth information of the object to be located in the sample depth image, the initial pose of the sample is optimized to obtain the optimized pose of the object to be located. This includes: determining a rendered depth map of the object to be located based on the initial pose of the sample and the preset 3D model corresponding to the object; determining optimization terms by using the difference between the rendered depth map and the sample depth image; adjusting the initial pose of the sample so that the optimization terms meet the preset requirements, and using the adjusted initial pose of the sample as the optimized pose.

[0007] Therefore, by determining the rendering depth map of the object to be located based on the initial pose of the sample and the preset 3D model corresponding to the object to be located, and then constructing an optimization term based on the difference between the rendering depth map and the sample depth image, the initial pose of the sample is adjusted using the optimization term, so that the adjusted initial pose of the sample is more accurate.

[0008] The preset requirement is to minimize the optimization term; and / or, the method further includes: determining the normal map of the object to be located based on the initial pose of the sample and the preset 3D model; and determining the optimization term by utilizing the difference between the rendered depth map and the sample depth image, including: back-projecting the rendered depth map and the sample depth image respectively to obtain a first point cloud corresponding to the rendered depth map and a second point cloud corresponding to the sample depth image, wherein the first point cloud includes a first 3D point corresponding to several object pixels, and the second point cloud includes a second 3D point corresponding to each object pixel, and the object pixels are the pixels in the sample color image that belong to the object to be located; for each object pixel, determining the deviation representation value corresponding to the object pixel, wherein the deviation representation is the product between the target pose difference corresponding to the object pixel and the normal direction corresponding to the object pixel in the normal map, wherein the target pose difference is the pose difference between the first 3D point and the corresponding second 3D point corresponding to the object pixel; and determining the optimization term by combining the deviation representation value corresponding to each object pixel.

[0009] Therefore, by back-projecting and rendering depth maps and sample depth images to obtain the first point cloud and the second point cloud, and by combining the normal directions of each point based on the differences between the three-dimensional points in the first point cloud and the three-dimensional points in the second point cloud, the determined deviation characterization value is more accurate.

[0010] Before adjusting the network parameters in the pose estimation model based on the difference between the optimized pose and the initial pose of the sample, the method further includes: determining whether the optimized pose is a preset incorrect estimated pose; and in response that the optimized pose is not a preset incorrect estimated pose, performing the step of adjusting the network parameters in the pose estimation model based on the difference between the optimized pose and the initial pose of the sample.

[0011] Therefore, by using the difference between the optimized pose and the initial pose of the sample, without using the preset incorrect estimated pose, it is more reasonable to adjust the network parameters in the pose estimation model, which can reduce the disturbance of the pose estimation model by the incorrect estimation.

[0012] The determination of whether the optimized pose is a preset incorrect estimated pose includes: obtaining the central tendency value among the deviation values ​​corresponding to each object pixel, wherein the object pixel is a pixel in the sample color image belonging to the object to be located, the deviation value corresponding to the object pixel is the product of the target pose difference corresponding to the object pixel and the normal direction corresponding to the object pixel, the target pose difference is the pose difference between the first 3D point and the corresponding second 3D point corresponding to the object pixel, the first 3D point is a 3D point in the first point cloud corresponding to the rendered depth map, and the second 3D point is a 3D point in the second point cloud corresponding to the sample depth image; determining whether the central tendency value is less than or equal to a preset size, the preset size being related to the size of the object to be located in the physical world; and determining that the optimized pose is not a preset incorrect estimated pose in response to the central tendency value being less than or equal to the preset size.

[0013] Therefore, by assuming that the central tendency representation value is not greater than the preset size, the optimized pose is not a preset incorrectly estimated pose, and the optimized pose can be filtered based on the physical size of the object to be located.

[0014] The process of processing the sample color image using a pose estimation model to obtain the initial pose of the object to be located includes: using the pose estimation model to determine the projection positions of several 3D key points of the object to be located on the sample color image; and determining the initial pose of the object to be located based on the projection positions of each 3D key point on the sample color image and the intrinsic parameters of the target camera.

[0015] Therefore, by using the pose estimation model, the projection positions of the 3D key points of the object to be located on the sample color image can be determined, and the initial pose of the object to be located can be obtained based on the determined projection positions of the 3D key points and the internal parameters of the target camera.

[0016] The process involves using a pose estimation model to determine the projection positions of several 3D key points of the object to be located on a sample color image. This includes: using the pose estimation model to predict the direction vector from each object pixel to each projection position, where the object pixel is a pixel in the sample color image belonging to the object to be located; for each projection position, determining a preset number of direction vectors from several direction vectors corresponding to the projection position to generate candidate projection positions corresponding to each direction vector; determining the score of each candidate projection position based on the positional relationship between them; and using the candidate projection positions whose scores meet preset requirements as the projection positions.

[0017] Therefore, by determining several candidate projection positions based on the direction vector of each object pixel with respect to the projection position, and then selecting the candidate projection position that meets the requirements as the final projection position, the determined projection position is more accurate.

[0018] The process includes: determining a preset number of direction vectors from several direction vectors corresponding to the projection position, and generating candidate projection positions corresponding to each direction vector; summing the position of each object pixel with the direction vector corresponding to the object pixel to obtain the candidate projection positions corresponding to each object pixel; determining the score of each candidate projection position based on the positional relationship between each candidate projection position; determining the number of target distances between each candidate projection position and other candidate projection positions for each candidate projection position, and using the number of target distances as the score, where the target distance is less than or equal to a preset distance; and using the candidate projection positions whose scores meet the preset requirements as projection positions; including using the candidate projection position corresponding to the maximum score as the projection position.

[0019] Therefore, by determining the distance between each candidate projection position, the final projection position is obtained, making the determined projection position more accurate.

[0020] The process of processing the sample color image using a pose estimation model to obtain the initial pose of the object to be located includes: performing target detection on the sample color image using the pose estimation model to obtain the position of the object to be located; cropping the sample color image based on the position of the object to be located to obtain a local image containing the object to be located; and processing the local image to obtain the initial pose of the object to be located.

[0021] Therefore, by first performing target detection on the sample color image to obtain the position of the object to be located, the sample color image is then cropped to obtain a local image containing the object to be located. The initial pose of the sample object is obtained by processing the local image, which can reduce background interference and thus improve the accuracy of the initial pose of the sample.

[0022] This application provides a pose estimation method, comprising: acquiring a target image containing an object to be located, the target image including a target color image and a target depth image corresponding to the target color image; processing the target color image using a pose estimation model to obtain the target initial pose of the object to be located; optimizing the target initial pose based on the depth information of the object to be located in the target depth image to obtain the target pose of the object to be located; wherein, the image detection model is trained using the training method of the above pose estimation model.

[0023] Therefore, by processing the target color image using a pose estimation model to obtain the initial pose of the object to be located, and then optimizing the initial pose using the target depth image, the optimized target pose of the object to be located becomes more accurate.

[0024] This application provides a training device for a pose estimation model, comprising: a sample image acquisition module for acquiring a sample image containing an object to be localized, the sample image including a sample color image and a sample depth image corresponding to the sample color image; a sample pose estimation module for processing the sample color image using the pose estimation model to obtain an initial sample pose of the object to be localized; a sample pose optimization module for optimizing the initial sample pose based on the depth information of the object to be localized in the sample depth image to obtain an optimized pose of the object to be localized; and a parameter adjustment module for adjusting the network parameters in the pose estimation model based on the difference between the optimized pose and the initial sample pose.

[0025] This application provides a pose estimation device, comprising: a target image acquisition module for acquiring a target image containing an object to be located, the target image including a target color image and a target depth image corresponding to the target color image; a target pose estimation module for processing the target color image using a pose estimation model to obtain the target initial pose of the object to be located; and a target pose optimization module for optimizing the target initial pose based on the depth information of the object to be located in the target depth image to obtain the target pose of the object to be located; wherein, the image detection model is trained using the training device of the above-mentioned pose estimation model.

[0026] This application provides an electronic device including a memory and a processor coupled to each other. The processor is used to execute program instructions stored in the memory to implement the above-described pose estimation model training method or the above-described image detection method.

[0027] This application provides a computer-readable storage medium storing program instructions thereon, which, when executed by a processor, implement the above-described pose estimation model training method or the above-described image detection method.

[0028] The above scheme obtains the initial pose of the object to be located by processing the sample color image using a pose estimation model, and then optimizes the initial pose using the sample depth image, resulting in a more accurate optimized pose. Furthermore, by using the difference between the optimized pose and the sample pose, the network parameters in the pose estimation model are adjusted, eliminating the need for annotation of the sample color image, reducing annotation workload, and improving the training efficiency of the pose estimation model.

[0029] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0030] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0031] Figure 1 This is a flowchart illustrating an embodiment of the training method for the pose estimation model of this application;

[0032] Figure 2 This is a schematic diagram of a sub-process of step S13 in an embodiment of the training method for the pose estimation model of this application;

[0033] Figure 3 This is another flowchart illustrating an embodiment of the training method for the pose estimation model of this application;

[0034] Figure 4 This is a flowchart illustrating an embodiment of the pose estimation method of this application;

[0035] Figure 5 This is a schematic diagram of the structure of an embodiment of the training device for the pose estimation model of this application;

[0036] Figure 6 This is a schematic diagram of the structure of an embodiment of the pose estimation device of this application;

[0037] Figure 7 This is a schematic diagram of the structure of an embodiment of the electronic device of this application;

[0038] Figure 8 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0039] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0040] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0041] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0042] The execution entity of the pose estimation model training method provided in this disclosure can be a pose estimation model training device. This device can be any terminal device, server, or other processing device capable of executing the method embodiments of this application. The terminal device can be a visual positioning device, user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, vehicle-mounted device, wearable device, etc. In some possible implementations, the pose estimation model training method can be implemented by a processor calling computer-readable instructions stored in memory.

[0043] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the training method for the pose estimation model of this application. Specifically, it may include the following steps:

[0044] Step S11: Obtain a sample image containing the object to be located. The sample image includes a sample color image and a sample depth image corresponding to the sample color image.

[0045] The sample images can be real images or synthetic images. In some application scenarios, the sample images may include a portion of real images and a portion of synthetic images. When the sample images are real images, the sample images containing the object to be located can be obtained by the execution device that performs the training method of the pose estimation model provided in the embodiments of this disclosure, which captures images of the object to be located; or, other devices can capture images of the object to be located and transmit them to the execution device via a communication connection. In some embodiments, publicly available image datasets for pose estimation can be used as sample images.

[0046] The pixel value of each pixel in the sample depth image is used to represent the depth value of the corresponding pixel in the sample color image. Specifically, this depth can be the distance between the 3D point corresponding to that pixel and the imaging device.

[0047] Step S12: Process the sample color image using the pose estimation model to obtain the initial pose of the object to be located.

[0048] The pose estimation model can be a pre-trained model or an unpre-trained model. It can be a ResNet network or any other network structure. Alternatively, the pose estimation model can directly output the initial pose of the sample object to be localized, or it can be based on an intermediate result from the pose estimation model, which is then further processed to obtain the initial pose of the sample object.

[0049] The initial pose of the sample can be a six-degree-of-freedom pose, that is, the initial pose of the sample includes the position and orientation of the object to be located in the camera coordinate system.

[0050] Step S13: Based on the depth information of the object to be located in the sample depth image, optimize the initial pose of the sample to obtain the optimized pose of the object to be located.

[0051] As mentioned above, the pixel value of each pixel in the sample depth image is used to represent the depth value of the corresponding pixel in the sample color image. Specifically, this depth can be the distance between the 3D point corresponding to that pixel and the imaging device. Because a single sample color image cannot accurately represent the distance between the object to be located and the imaging device, the resulting initial pose may be inaccurate. Therefore, by combining the depth information of the object to be located in the sample depth image, the initial pose is optimized, resulting in a more accurate optimized pose.

[0052] Step S14: Adjust the network parameters in the pose estimation model based on the difference between the optimized pose and the initial pose of the sample.

[0053] Alternatively, the loss can be determined based on the difference between the optimized pose and the initial pose of the sample, and then the network parameters in the pose estimation model can be adjusted using the loss.

[0054] The above scheme obtains the initial pose of the object to be located by processing the sample color image using a pose estimation model, and then optimizes the initial pose using the sample depth image, resulting in a more accurate optimized pose. Furthermore, by using the difference between the optimized pose and the sample pose, the network parameters in the pose estimation model are adjusted, eliminating the need for annotation of the sample color image, reducing annotation workload, and improving the training efficiency of the pose estimation model.

[0055] In some disclosed embodiments, step S12 above may include the following steps:

[0056] A pose estimation model is used to detect objects in a sample color image to obtain the location of the object to be located. For example, the pose estimation model includes an object detection sub-network, which is used to detect objects in the sample color image to obtain the location of the object to be located within the sample color image. In other disclosed embodiments, the object detection sub-network and the pose estimation model can be separate and independent; that is, the object detection network is first used to detect objects in the sample color image to obtain the location of the object to be located, and then the pose estimation model processes the sample color image based on the detection results of the object detection network.

[0057] Then, based on the location of the object to be located, the sample color image is cropped to obtain a local image containing the object to be located. For example, the cropping method can be to crop by extending the area where the object to be located is located on the sample color image outward by a preset scale, and the cropped portion containing the object to be located is used as the local image.

[0058] Next, the local image is processed to obtain the initial pose of the object to be located. By first detecting the position of the object and cropping the sample color image, the interference of the background of the merged image on the processing result can be reduced, thereby improving the accuracy of the initial pose of the sample. For example, the background and foreground of the synthesized image may have significant differences. If the synthesized image is processed directly, the initial pose of the obtained sample may be inaccurate.

[0059] By first performing target detection on the sample color image to obtain the location of the object to be located, the sample color image is then cropped to obtain a local image containing the object to be located. The initial pose of the sample object is obtained by processing the local image, thereby reducing background interference and improving the accuracy of the initial pose of the sample.

[0060] In some disclosed embodiments, step S12 may further include the following steps:

[0061] Using a pose estimation model, the projection positions of several 3D key points of the object to be located on a sample color image are determined. As mentioned above, the sample color image can be cropped to obtain a local image containing the object to be located. Specifically, determining the projection positions of the several 3D key points of the object on the sample color image can involve determining the projection positions of each 3D key point on that local image. The several 3D key points of the object to be located can be extracted from a preset 3D model corresponding to the object. For example, the several 3D key points can be obtained from a set of 3D points on the preset 3D model using a farthest point sampling algorithm.

[0062] Based on the projection positions of each 3D keypoint on the sample color image and the intrinsic parameters of the target camera, the initial pose of the object to be localized is determined. Specifically, after determining the projection positions of each 3D keypoint on the local image, the initial pose of the object to be localized can be determined using a point-and-shoot (PNP) approach, combined with the intrinsic parameters of the target camera. For example, the intrinsic parameters of the target camera may include parameters such as focal length. The specific method of obtaining the initial pose of the object to be localized using PNP will not be described in detail here. The target camera can be a camera that acquires sample images, similar to the aforementioned imaging devices.

[0063] By using the pose estimation model, the projection positions of the 3D key points of the object to be located on the sample color image can be determined. Based on the determined projection positions of the 3D key points and the internal parameters of the target camera, the initial pose of the sample object to be located can be obtained.

[0064] The method described above for determining the projection positions of several 3D key points of the object to be located on the sample color image using the pose estimation model can be as follows:

[0065] Using a pose estimation model, the direction vector from each object pixel to each projected position is predicted. Here, object pixels are the pixels in the sample color image that belong to the object to be located. As described above, the pose estimation model can perform target detection on the sample color image to obtain the position of the object to be located. The semantic labels of pixels belonging to the object to be located can be set to a first preset value (e.g., the first preset value can be 1), and the semantic labels of other pixels not belonging to the object to be located can be set to a second preset value (e.g., the second preset value can be 0). Pixels with the semantic label of the first preset value are considered object pixels. The direction vector can be a two-dimensional vector, with one dimension being the vector along the x-axis of the sample color image and the other dimension being the vector along the y-axis of the sample color image.

[0066] For example, the direction vector from the object pixel to the projected position can be referred to formula (1).

[0067] v k (p)=x k -p Formula (1);

[0068] Among them, v k (p) represents the direction vector from pixel p of the object to the k-th projection position, x k Let represent the k-th projection position, and p represent the position of pixel p.

[0069] For each projection position, a preset number of direction vectors are determined from several direction vectors corresponding to that projection position, generating candidate projection positions corresponding to each direction vector. For example, the position of each object pixel is summed with its corresponding direction vector to obtain the candidate projection positions for each object pixel. For example, if there are 10 object pixels belonging to the object to be located in the sample color image, and the preset number is 5, then 5 direction vectors can be selected from the 10 direction vectors. Each direction vector corresponds to an object pixel, and the candidate projection positions corresponding to that object pixel are obtained by summing the object pixel with its corresponding direction vector; thus, 5 candidate projection positions are obtained. In some application scenarios, the input to the pose estimation model is several sample color images, and the output is the semantic label corresponding to each pixel and the direction vector corresponding to each pixel. The semantic label indicates whether the pixel belongs to the object to be located. In some application scenarios, the input sample color images may include several types of objects to be located, and the output includes the semantic label corresponding to each pixel, which may specifically be a label indicating which type of object the pixel belongs to. For example, the objects to be located may include cups, tables, stools, etc. That is, the pose estimation model obtained by the training method of the pose estimation model provided in this embodiment of the present disclosure can simultaneously estimate the pose of multiple objects to be located, and obtain the target pose of each object to be located.

[0070] Among them, the method for determining the corresponding candidate projection position based on the direction vector and the corresponding object pixel can be referred to formula (2).

[0071] h k，i =p+v k (p) Formula (2);

[0072] Among them, {h k，i |i = 1, 2, ..., N}, where N is the number of candidate projection positions. p represents the object pixel p, v k (p) represents the direction vector from the object pixel p to the k-th projection position.

[0073] Then, based on the positional relationships between the candidate projection positions, a score is determined for each candidate projection position. The candidate projection positions whose scores meet preset requirements are then designated as the projection positions.

[0074] Optionally, the method for determining the score of each candidate projection position based on the positional relationship between each candidate projection position can be as follows: for each candidate projection position, determine the target distance between that candidate projection position and other candidate projection positions, and use the target distance as the score. Here, the target distance is a distance less than or equal to a preset distance. For example, the difference between the current candidate position and other candidate projection positions is calculated to obtain the distance between the current candidate projection position and other candidate projection positions. The preset distance can be adjusted during the training of the pose estimation model to determine the final preset distance.

[0075] For example, the score for each candidate projection position can be calculated by referring to formula (3).

[0076] w k，i =∑I(|h k，i -pv k (p)|≤θ) (3);

[0077] Among them, w k，i Let I represent the score of the i-th candidate projection position, where I is an indicator function, set to 1 if the condition is met and 0 if the condition is not met. θ is a preset distance, for example, θ can take the value 1.

[0078] Specifically, the method of using candidate projection positions that meet the preset requirements can be as follows: the candidate projection position corresponding to the highest score can be used as the projection position. The method for determining the projection positions of other 3D keypoints can be referenced above and will not be repeated here.

[0079] In some application scenarios, the training method for the pose estimation model provided in this disclosure may further include a pre-training step for the pose estimation model. The specific steps of pre-training may include: acquiring several sample images, which may be the same as or different from the sample images acquired in step S11; obtaining the sample semantic labels and sample projection positions for each pixel in the sample images; obtaining a first loss between the semantic labels output by the pose estimation model and the sample semantic labels, and determining the projection position based on the direction vectors of each object pixel output by the pose estimation model; obtaining a second loss between the projection position and the sample projection position; and adjusting the network parameters in the pose estimation model by combining the first and second losses. In the pre-training, the initial learning rate may be set to 1e. －3 The learning rate is halved after every predetermined number of iterations. After pre-training, the learning rate can be adjusted to 5e. －4 The learning rate is halved after every second predetermined number of iterations. Optionally, the first predetermined number of iterations is twice the second predetermined number of iterations.

[0080] By determining several candidate projection positions based on the direction vector of each object pixel with respect to the projection position, and then selecting the candidate projection position that meets the requirements as the final projection position, the determined projection position is more accurate.

[0081] Please also see Figure 2 , Figure 2 This is a schematic diagram of a sub-process of step S13 in an embodiment of the pose estimation model training method of this application. Figure 2 As described above, step S13 may include the following steps:

[0082] Step S131: Based on the initial pose of the sample and the preset 3D model corresponding to the object to be located, determine the rendering depth map of the object to be located.

[0083] The preset 3D model can be drawn using drawing software, or it can be constructed in 3D using a modeling network with several images containing the corresponding images to be located.

[0084] The initial pose of the sample can be considered as the pose of the preset 3D model corresponding to the object to be localized in the camera coordinate system. The method described above for determining the rendering depth map of the object to be localized based on the initial pose of the sample and the preset 3D model corresponding to the object to be localized can be to project the preset 3D model onto the camera plane based on the initial pose of the sample to obtain the rendering depth map.

[0085] Step S132: Determine optimization terms by utilizing the difference between the rendered depth map and the sample depth image.

[0086] The training method for the pose estimation model may further include the following steps: determining the normal map of the object to be located based on the initial pose of the sample and a preset 3D model. For example, the preset 3D model may be composed of several planes (e.g., triangular mesh surfaces), each plane corresponding to a pixel value of a pixel in the normal map, which can be used to represent the normal direction of the plane.

[0087] One method for determining optimization terms by utilizing the difference between the rendered depth map and the sample depth image is:

[0088] The rendered depth map and the sample depth image are back-projected separately to obtain the rendered depth map and the corresponding first point cloud, and the sample depth image and the corresponding second point cloud. The first point cloud includes first 3D points corresponding to several object pixels, and the second point cloud includes second 3D points corresponding to each object pixel. As mentioned above, object pixels are pixels in the sample color image belonging to the object to be located. Specifically, the rendered depth map is back-projected using the initial pose of the sample object to be located to obtain the first point cloud; the sample depth image is back-projected using the initial pose of the sample object to be located to obtain the second point cloud.

[0089] Then, for each object pixel, the deviation representation value corresponding to the object pixel is determined. The deviation representation value can be a residual. Here, the deviation representation value corresponding to the object pixel is the product of the target pose difference corresponding to the object pixel and the normal direction corresponding to the object pixel in the normal map. The target pose difference is the pose difference between the first 3D point and the corresponding second 3D point corresponding to the object pixel.

[0090] For the object pixel point p, the method to determine its deviation characterization value L(p) can be referred to formula (4).

[0091] L(p) = ||(π) -1 (D r (p))-π -1 (D(p)))N r (p)||2 Formula (4);

[0092] Where, π -1 It is the back projection function, D r (p) represents the depth value of object pixel p in the rendered depth map, D(p) represents the depth value of object pixel p in the sample depth map, and N r (p) represents the normal direction of pixel p in the normal map. π -1 (D r (p))-π -1 (D(p)) represents the pose difference between the first three-dimensional point and the corresponding second three-dimensional point.

[0093] The deviation representation value of an object pixel can be the minimum distance from the second 3D point to the first 3D point on a plane defined by the first 3D point and its normal.

[0094] Next, by combining the deviation representation values ​​corresponding to each object pixel, optimization terms are determined. For example, the sum of all deviation representation values, the average of all deviation representation values, or the maximum value among all deviation representation values ​​are used as optimization terms. These residual terms are minimized using gradient descent to obtain the desired optimized pose. In some applications, to overcome the convergence of results to local minima, a set of hypothetical initial sample poses can be generated by perturbing the initial sample poses. These poses are then optimized to obtain a more accurate optimized pose.

[0095] Step S133: Adjust the initial pose of the sample to make the optimization terms meet the preset requirements, and use the adjusted initial pose of the sample as the optimized pose.

[0096] Optionally, the preset requirement can be minimizing the optimization items.

[0097] By determining the rendered depth map of the object to be located based on the initial pose of the sample and the preset 3D model corresponding to the object to be located, an optimization term is constructed based on the difference between the rendered depth map and the sample depth image. The initial pose of the sample is adjusted using the optimization term, so that the adjusted initial pose of the sample is more accurate.

[0098] In addition, by back-projecting and rendering depth maps and sample depth images, the first point cloud and the second point cloud are obtained. Based on the difference between the three-dimensional points in the first point cloud and the three-dimensional points in the second point cloud, the normal directions of each point are set, making the determined deviation characterization value more accurate.

[0099] In some disclosed embodiments, the following steps may be performed before performing step S14 above:

[0100] Determine whether the optimized pose is a preset incorrect estimated pose. If the optimized pose is not a preset incorrect estimated pose, perform the step of adjusting the network parameters in the pose estimation model based on the difference between the optimized pose and the initial sample pose.

[0101] Optionally, in response to the optimized pose being a preset incorrect estimated pose, the optimized pose is discarded, and the step of adjusting the network parameters in the pose estimation model based on the difference between the optimized pose and the sample initial pose is not performed.

[0102] By using the difference between the optimized pose and the initial pose of the sample when the optimized pose is not the preset erroneous estimated pose, it is more reasonable to adjust the network parameters in the pose estimation model, which can reduce the disturbance of the pose estimation model by erroneous estimation.

[0103] One possible way to determine whether the optimized pose is a preset incorrectly estimated pose is:

[0104] Obtain the central tendency of the deviation values ​​corresponding to each object pixel. As mentioned above, the object pixel is the pixel in the sample color image that belongs to the object to be located. The deviation value corresponding to the object pixel is the product of the target pose difference of the object pixel and the normal direction of the object pixel. The target pose difference is the pose difference between the first 3D point and the second 3D point corresponding to the object pixel. The first 3D point is a 3D point in the first point cloud corresponding to the rendered depth map, and the second 3D point is a 3D point in the second point cloud corresponding to the sample depth image. The first point cloud is obtained by backprojection from the rendered depth map, and the second point cloud is obtained by backprojection from the sample depth image.

[0105] Then, it is determined whether the central tendency characteristic value is less than or equal to a preset size. The preset size is related to the physical dimensions of the object to be located. For example, the preset size could be 0.2 times the length of the object to be located.

[0106] If the central tendency representation value is less than or equal to a preset size, the optimized pose is determined to be a preset incorrectly estimated pose. If the central tendency representation value is less than or equal to a preset size, the optimized pose is determined to be a preset incorrectly estimated pose.

[0107] By assuming that the central tendency representation value is not greater than the preset size, the optimized pose is not a preset incorrectly estimated pose, and the optimized pose can be filtered based on the physical size of the object to be located.

[0108] To better understand the training method of the pose estimation model provided in the embodiments of this disclosure, please refer to... Figure 3 , Figure 3 This is another flowchart illustrating an embodiment of the training method for the pose estimation model of this application. Figure 3 As shown, given a set of unannotated sample image data, the pose estimation model first predicts the initial pose of the object to be localized on these sample color images. Then, the estimated initial poses of these samples are optimized using depth information. The pose optimization can be performed iteratively. Next, each optimized pose is evaluated, i.e., it is determined whether each optimized pose is a pre-defined incorrect estimate, and the optimized poses are filtered based on the judgment results, discarding failed estimates. Finally, the pose estimation model is adjusted by the difference between the retained optimized poses and the sample initial poses. In the pose estimation process, our method can predict the 6D pose of the object to be localized from only a single sample color image using the pose estimation model.

[0109] The pose estimation model training method provided in this disclosure can be applied to the field of augmented reality applications.

[0110] Please see Figure 4 , Figure 4This is a schematic flowchart of an embodiment of the pose estimation method of this application. Figure 4 As shown, the pose estimation method provided in this embodiment may include the following steps:

[0111] Step S21: Obtain a target image containing the object to be located. The target image includes a target color image and a target depth image corresponding to the target color image.

[0112] The target image containing the object to be located can be captured by the device executing the pose estimation method, or it can be captured by a device that has established a communication connection with the execution device.

[0113] Step S22: Process the target color image using the pose estimation model to obtain the initial pose of the object to be located.

[0114] The method for obtaining the initial pose of the target can refer to the method for obtaining the initial pose of the sample in the above-described embodiment of the pose estimation model training method, and will not be repeated here. The pose estimation model is trained using the training method provided in the above-described embodiment of the pose estimation model training method.

[0115] Step S23: Based on the depth information of the object to be located in the target depth image, optimize the initial pose of the target to obtain the target pose of the object to be located.

[0116] The method for obtaining the target pose of the object to be located can refer to the method for obtaining the optimized pose in the above-mentioned training method embodiment of the pose estimation model, which will not be repeated here.

[0117] The above scheme obtains the initial pose of the target object by processing the target color image using a pose estimation model, and then optimizes the initial pose of the target object using the target depth image, making the optimized target pose of the target object more accurate.

[0118] The pose estimation method provided in this disclosure can be applied to the field of augmented reality applications.

[0119] The pose estimation method provided in this disclosure can be executed by a pose estimation device, which can be any terminal device, server, or other processing device capable of executing the method embodiments of this application. The terminal device can be an augmented reality display device, a visual positioning device, a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, etc. In some possible implementations, the pose estimation method can be implemented by a processor calling computer-readable instructions stored in memory.

[0120] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0121] Please see Figure 5 , Figure 5 This is a schematic diagram of an embodiment of the training device for the pose estimation model of this application. The training device 50 for the pose estimation model includes a sample image acquisition module 51, a sample pose estimation module 52, a sample pose optimization module 53, and a parameter adjustment module 54. The sample image acquisition module 51 is used to acquire a sample image containing the object to be localized, the sample image including a sample color image and a sample depth image corresponding to the sample color image; the sample pose estimation module 52 is used to process the sample color image using the pose estimation model to obtain the initial pose of the object to be localized; the sample pose optimization module 53 is used to optimize the initial pose of the sample based on the depth information of the object to be localized in the sample depth image to obtain the optimized pose of the object to be localized; the parameter adjustment module 54 is used to adjust the network parameters in the pose estimation model based on the difference between the optimized pose and the initial pose of the sample.

[0122] The above scheme obtains the initial pose of the object to be located by processing the sample color image using a pose estimation model, and then optimizes the initial pose using the sample depth image, resulting in a more accurate optimized pose. Furthermore, by using the difference between the optimized pose and the sample pose, the network parameters in the pose estimation model are adjusted, eliminating the need for annotation of the sample color image, reducing annotation workload, and improving the training efficiency of the pose estimation model.

[0123] In some disclosed embodiments, the sample pose optimization module 53 optimizes the initial pose of the sample based on the depth information of the object to be located in the sample depth image to obtain the optimized pose of the object to be located. This includes: determining a rendered depth map of the object to be located based on the initial pose of the sample and a preset 3D model corresponding to the object to be located; determining optimization terms by utilizing the difference between the rendered depth map and the sample depth image; adjusting the initial pose of the sample so that the optimization terms meet preset requirements, and using the adjusted initial pose of the sample as the optimized pose.

[0124] The above scheme determines the rendering depth map of the object to be located based on the initial pose of the sample and the preset 3D model corresponding to the object to be located. Then, based on the difference between the rendering depth map and the sample depth image, an optimization term is constructed. The optimization term is used to adjust the initial pose of the sample, so that the adjusted initial pose of the sample is more accurate.

[0125] In some disclosed embodiments, the preset requirement is to minimize the optimization term; and / or, the sample pose optimization module 53 is further configured to: determine the normal map of the object to be located based on the initial pose of the sample and the preset three-dimensional model; and, the sample pose optimization module 53 determines the optimization term by utilizing the difference between the rendered depth map and the sample depth image, including: back-projecting the rendered depth map and the sample depth image respectively to obtain a first point cloud corresponding to the rendered depth map and a second point cloud corresponding to the sample depth image, wherein the first point cloud includes a number of first three-dimensional points corresponding to object pixels, and the second point cloud includes second three-dimensional points corresponding to each object pixel, and the object pixels are pixels in the sample color image that belong to the object to be located; for each object pixel, determining the deviation representation value corresponding to the object pixel, wherein the deviation representation is the product between the target pose difference corresponding to the object pixel and the normal direction corresponding to the object pixel in the normal map, wherein the target pose difference is the pose difference between the first three-dimensional point and the corresponding second three-dimensional point corresponding to the object pixel; and combining the deviation representation value corresponding to each object pixel to determine the optimization term.

[0126] The above scheme obtains the first point cloud and the second point cloud by back-projecting and rendering the depth map and the sample depth image. Based on the difference between the three-dimensional points in the first point cloud and the three-dimensional points in the second point cloud, the normal direction of each point is set, so that the determined deviation characterization value is more accurate.

[0127] In some disclosed embodiments, before adjusting the network parameters in the pose estimation model based on the difference between the optimized pose and the initial sample pose, the adjustment module 54 is further configured to: determine whether the optimized pose is a preset incorrect estimated pose; and in response to the optimized pose not being a preset incorrect estimated pose, perform the step of adjusting the network parameters in the pose estimation model based on the difference between the optimized pose and the initial sample pose.

[0128] The above scheme, by using the difference between the optimized pose and the initial pose of the sample when the optimized pose is not the preset erroneous estimated pose, makes it more reasonable to adjust the network parameters in the pose estimation model, which can reduce the disturbance of the pose estimation model by erroneous estimation.

[0129] In some disclosed embodiments, the adjustment module 54 determines whether the optimized pose is a preset incorrect estimated pose, including: obtaining the central tendency representation value among the deviation representation values ​​corresponding to each object pixel, wherein the object pixel is a pixel in the sample color image belonging to the object to be located, the deviation representation value corresponding to the object pixel is the product between the target pose difference corresponding to the object pixel and the normal direction corresponding to the object pixel, the target pose difference is the pose difference between the first three-dimensional point corresponding to the object pixel and the corresponding second three-dimensional point, the first three-dimensional point is a three-dimensional point in the first point cloud corresponding to the rendered depth map, and the second three-dimensional point is a three-dimensional point in the second point cloud corresponding to the sample depth image; determining whether the central tendency representation value is less than or equal to a preset size, the preset size being related to the size of the object to be located in the physical world; and determining that the optimized pose is not a preset incorrect estimated pose in response to the central tendency representation value being less than or equal to the preset size.

[0130] The above scheme, by assuming that the optimized pose is not a preset erroneous pose when the central tendency representation value is not greater than the preset size, can filter the optimized pose based on the physical size of the object to be located.

[0131] In some disclosed embodiments, the sample pose estimation module 52 processes the sample color image using a pose estimation model to obtain the initial pose of the object to be located, including: using the pose estimation model to determine the projection positions of several three-dimensional key points of the object to be located on the sample color image; and determining the initial pose of the object to be located based on the projection positions of each three-dimensional key point on the sample color image and the intrinsic parameters of the target camera.

[0132] The above scheme, through the pose estimation model, can determine the projection position of the three-dimensional key points of the object to be located on the sample color image, and thus obtain the initial pose of the sample object to be located based on the determined projection position of the three-dimensional key points and the internal parameters of the target camera.

[0133] In some disclosed embodiments, the sample pose estimation module 52 uses a pose estimation model to determine the projection positions of several 3D key points of the object to be located on a sample color image, including: using the pose estimation model to predict the direction vector from each object pixel to each projection position, where the object pixel is a pixel in the sample color image belonging to the object to be located; for each projection position, determining a preset number of direction vectors from several direction vectors corresponding to the projection position to generate candidate projection positions corresponding to each direction vector; determining the score of each candidate projection position based on the positional relationship between each candidate projection position; and using the candidate projection positions whose scores meet preset requirements as projection positions.

[0134] The above scheme determines several candidate projection positions based on the direction vector of each pixel with respect to the projection position, and then selects the candidate projection position that meets the requirements as the final projection position, making the determined projection position more accurate.

[0135] In some disclosed embodiments, the sample pose estimation module 52 determines a preset number of direction vectors from a plurality of direction vectors corresponding to the projection position, and generates candidate projection positions corresponding to each direction vector, including: summing the position of each object pixel with the direction vector corresponding to the object pixel to obtain the candidate projection position corresponding to each object pixel; determining the score of each candidate projection position based on the positional relationship between each candidate projection position, including: for each candidate projection position, determining the number of target distances between the candidate projection position and other candidate projection positions, and using the number of target distances as the score, wherein the target distance is a distance less than or equal to a preset distance; and using the candidate projection position whose score meets the preset requirements as the projection position, including: using the candidate projection position corresponding to the maximum score as the projection position.

[0136] The above scheme determines the final projection position by determining the distance between each candidate projection position, thus making the determined projection position more accurate.

[0137] In some disclosed embodiments, the sample pose estimation module 52 processes the sample color image using a pose estimation model to obtain the initial pose of the object to be located, including: performing target detection on the sample color image using the pose estimation model to obtain the position of the object to be located; cropping the sample color image based on the position of the object to be located to obtain a local image containing the object to be located; and processing the local image to obtain the initial pose of the object to be located.

[0138] The above scheme first performs target detection on the sample color image to obtain the position of the object to be located, then crops the sample color image to obtain a local image containing the object to be located, and processes the local image to obtain the initial pose of the sample object to be located. This reduces background interference and improves the accuracy of the initial pose of the sample.

[0139] Please see Figure 6 , Figure 6 This is a schematic diagram of an embodiment of the pose estimation device of this application. The pose estimation device 60 includes a target image acquisition module 61, a target pose estimation module 62, and a target pose optimization module 63. The target image acquisition module 61 is used to acquire a target image containing an object to be located, the target image including a target color image and a target depth image corresponding to the target color image; the target pose estimation module 62 is used to process the target color image using a pose estimation model to obtain the target initial pose of the object to be located; the target pose optimization module 63 is used to optimize the target initial pose based on the depth information of the object to be located in the target depth image to obtain the target pose of the object to be located; wherein, the image detection model is trained by the pose estimation model training device provided in the above-described pose estimation model training device embodiment.

[0140] The above scheme obtains the initial pose of the target object by processing the target color image using a pose estimation model, and then optimizes the initial pose of the target object using the target depth image, making the optimized target pose of the target object more accurate.

[0141] Please see Figure 7 , Figure 7 This is a schematic diagram of an embodiment of the electronic device of this application. The electronic device 70 includes a memory 71 and a processor 72 coupled to each other. The processor 72 is used to execute program instructions stored in the memory 71 to implement the steps of any of the above-described pose estimation model training method embodiments, or to implement the steps of any of the above-described pose estimation method embodiments. In a specific implementation scenario, the electronic device 70 may include, but is not limited to, a microcomputer, a server, etc. In addition, the electronic device 70 may also include mobile devices such as laptops and tablets, which are not limited here.

[0142] Specifically, processor 72 controls itself and memory 71 to implement the steps of any of the above-described pose estimation model training method embodiments, or to implement the steps of any of the above-described pose estimation method embodiments. Processor 72 can also be referred to as a CPU (Central Processing Unit). Processor 72 may be an integrated circuit chip with signal processing capabilities. Processor 72 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 72 can be implemented using integrated circuit chips.

[0143] The above scheme obtains the initial pose of the object to be located by processing the sample color image using a pose estimation model, and then optimizes the initial pose using the sample depth image, resulting in a more accurate optimized pose. Furthermore, by using the difference between the optimized pose and the sample pose, the network parameters in the pose estimation model are adjusted, eliminating the need for annotation of the sample color image, reducing annotation workload, and improving the training efficiency of the pose estimation model.

[0144] Please see Figure 8 , Figure 8 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application. The computer-readable storage medium 80 stores program instructions 801 that can be executed by a processor. The program instructions 801 are used to implement the steps of any of the above-described pose estimation model training method embodiments, or to implement the steps of any of the above-described pose estimation method embodiments.

[0145] The above scheme obtains the initial pose of the object to be located by processing the sample color image using a pose estimation model, and then optimizes the initial pose using the sample depth image, resulting in a more accurate optimized pose. Furthermore, by using the difference between the optimized pose and the sample pose, the network parameters in the pose estimation model are adjusted, eliminating the need for annotation of the sample color image, reducing annotation workload, and improving the training efficiency of the pose estimation model.

[0146] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0147] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0148] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0149] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0150] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0151] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the user through pop-up information or by asking the user to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

Claims

1. A method for training a pose estimation model, characterized in that, include: Obtain a sample image containing the object to be located, wherein the sample image includes a sample color image and a sample depth image corresponding to the sample color image; The initial pose of the object to be located is obtained by processing the sample color image using a pose estimation model. Based on the depth information of the object to be located in the sample depth image, the initial pose of the sample is optimized to obtain the optimized pose of the object to be located. This includes: determining a rendered depth map of the object to be located based on the initial pose of the sample and a preset 3D model corresponding to the object to be located; determining optimization terms using the difference between the rendered depth map and the sample depth image; adjusting the initial pose of the sample so that the optimization terms meet preset requirements, and using the adjusted initial pose of the sample as the optimized pose. Based on the difference between the optimized pose and the initial pose of the sample, the network parameters in the pose estimation model are adjusted.

2. The method according to claim 1, characterized in that, The preset requirement is to minimize the optimization term; And / or, The method further includes: determining a normal map of the object to be located based on the initial pose of the sample and the preset 3D model; and determining optimization terms by utilizing the difference between the rendered depth map and the sample depth image, including: Back-projecting the rendered depth map and the sample depth image respectively yields a first point cloud corresponding to the rendered depth map and a second point cloud corresponding to the sample depth image. The first point cloud includes a first three-dimensional point corresponding to several object pixels, and the second point cloud includes a second three-dimensional point corresponding to each of the object pixels. The object pixels are the pixels in the sample color image that belong to the object to be located. For each object pixel, a deviation characterization value corresponding to the object pixel is determined. The deviation characterization is the product between the target pose difference corresponding to the object pixel and the normal direction corresponding to the object pixel in the normal map. The target pose difference is the pose difference between the first three-dimensional point and the corresponding second three-dimensional point of the object pixel. The optimization term is determined by combining the deviation characterization value corresponding to each of the object pixels.

3. The method according to claim 1, characterized in that, Before adjusting the network parameters in the pose estimation model based on the difference between the optimized pose and the initial pose of the sample, the method further includes: Determine whether the optimized pose is a preset incorrectly estimated pose; In response to the optimized pose not being the preset incorrect estimated pose, the step of adjusting the network parameters in the pose estimation model based on the difference between the optimized pose and the initial sample pose is performed.

4. The method according to claim 3, characterized in that, The step of determining whether the optimized pose is a preset incorrect estimated pose includes: Obtain the central tendency representation value among the deviation representation values ​​corresponding to each object pixel, wherein the object pixel is the pixel in the sample color image that belongs to the object to be located, the deviation representation value corresponding to the object pixel is the product of the target pose difference corresponding to the object pixel and the normal direction corresponding to the object pixel, the target pose difference is the pose difference between the first three-dimensional point and the corresponding second three-dimensional point corresponding to the object pixel, the first three-dimensional point is a three-dimensional point in the first point cloud corresponding to the rendered depth map, and the second three-dimensional point is a three-dimensional point in the second point cloud corresponding to the sample depth image; Determine whether the central tendency characterization value is less than or equal to a preset size, the preset size being related to the size of the object to be located in the physical world; In response to the central tendency representation value being less than or equal to the preset size, it is determined that the optimized pose is not the preset incorrectly estimated pose.

5. The method according to any one of claims 1-4, characterized in that, The step of processing the sample color image using a pose estimation model to obtain the initial pose of the object to be located includes: Using the pose estimation model, the projection positions of several three-dimensional key points of the object to be located on the sample color image are determined; Based on the projection positions of each of the three-dimensional key points on the sample color image and the intrinsic parameters of the target camera, the initial pose of the sample object to be located is determined.

6. The method according to claim 5, characterized in that, The step of using the pose estimation model to determine the projection positions of several 3D key points of the object to be located on the sample color image includes: Using the pose estimation model, the direction vector from each object pixel to each projection position is predicted, where the object pixel is the pixel in the sample color image that belongs to the object to be located. For each projection position, a preset number of direction vectors are determined from a plurality of direction vectors corresponding to the projection position, and candidate projection positions corresponding to each direction vector are generated. Based on the positional relationship between the candidate projection positions, the score of each candidate projection position is determined; The candidate projection positions that meet the preset requirements are taken as the projection positions.

7. The method according to claim 6, characterized in that, The step of determining a preset number of direction vectors from a plurality of direction vectors corresponding to the projection position, and generating candidate projection positions corresponding to each direction vector, includes: The candidate projection position corresponding to each object pixel is obtained by summing the position of each object pixel with the direction vector corresponding to the object pixel. The step of determining the score of each candidate projection position based on the positional relationship between each candidate projection position includes: For each candidate projection position, determine the number of target distances between the candidate projection position and other candidate projection positions, and use the number of target distances as the score, wherein the target distance is a distance less than or equal to a preset distance; The step of using candidate projection positions that satisfy preset requirements as the projection positions includes: The candidate projection position corresponding to the maximum score is taken as the projection position.

8. The method according to any one of claims 1-4, characterized in that, The step of processing the sample color image using a pose estimation model to obtain the initial pose of the object to be located includes: The position of the object to be located is obtained by using the pose estimation model to perform target detection on the sample color image; Based on the location of the object to be located, the sample color image is cropped to obtain a partial image containing the object to be located; The local image is processed to obtain the initial pose of the sample of the object to be located.

9. A pose estimation method, characterized in that, include: Obtain a target image containing the object to be located, the target image including a target color image and a target depth image corresponding to the target color image; The target color image is processed using a pose estimation model to obtain the initial pose of the object to be located; Based on the depth information of the object to be located in the target depth image, the initial pose of the target is optimized to obtain the target pose of the object to be located. The pose estimation model is trained using the pose estimation model training method described in any one of claims 1 to 8.

10. A training device for a pose estimation model, characterized in that, include: The sample image acquisition module is used to acquire a sample image containing the object to be located, wherein the sample image includes a sample color image and a sample depth image corresponding to the sample color image; The sample pose estimation module is used to process the sample color image using a pose estimation model to obtain the initial pose of the sample object to be located. The sample pose optimization module is used to optimize the initial pose of the sample based on the depth information of the object to be located in the sample depth image, to obtain the optimized pose of the object to be located. This includes: determining a rendered depth map of the object to be located based on the initial pose and a preset 3D model corresponding to the object; determining optimization terms using the difference between the rendered depth map and the sample depth image; adjusting the initial pose of the sample to make the optimization terms meet preset requirements, and using the adjusted initial pose of the sample as the optimized pose. The parameter adjustment module is used to adjust the network parameters in the pose estimation model based on the difference between the optimized pose and the initial pose of the sample.

11. A pose estimation device, characterized in that, include: The target image acquisition module is used to acquire a target image containing the object to be located, wherein the target image includes a target color image and a target depth image corresponding to the target color image; The target pose estimation module is used to process the target color image using a pose estimation model to obtain the initial pose of the object to be located. The target pose optimization module is used to optimize the initial pose of the target based on the depth information of the object to be located in the target depth image, so as to obtain the target pose of the object to be located. The pose estimation model is trained using the pose estimation model training device described in claim 10.

12. An electronic device, characterized in that, It includes a memory and a processor that are coupled to each other, the processor being used to execute program instructions stored in the memory to implement the training method of the pose estimation model according to any one of claims 1 to 8, or to implement the pose estimation method according to claim 9.

13. A computer-readable storage medium having program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, they implement the training method of the pose estimation model according to any one of claims 1 to 8, or the pose estimation method according to claim 9.

Images