Object Pose Estimation Method, Apparatus, Device, and Storage Medium

Abstract

The present application discloses a method, apparatus, device and storage medium for estimating the pose of an object. The method comprises: capturing a first color depth image of the object and extracting feature points of the first color depth image; based on the feature points of the first color depth image, selecting a second color depth image that matches the first color depth image from a pre-stored file corresponding to the object; wherein the pre-stored file includes multiple color depth images of the object and feature points of each color depth image; calculating the initial pose of the object based on the feature points matched between the first color depth image and the second color depth image; iteratively optimizing the initial pose based on the color mode used to reconstruct the three-dimensional model of the object in the pre-stored file and the confidence of using the color mode to obtain the final pose of the object. The object pose estimation method provided by the embodiment of the present application can accurately and quickly estimate the pose of an unfamiliar object with good performance.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method, device, equipment and storage medium for estimating the pose of an object. Background Art

[0002] Estimating the 6D pose of an object involves estimating its 6D pose in the camera coordinate system, including its rotation and translation. This involves obtaining the rotation and translation transformation from the object's coordinate system to the camera coordinate system. Estimating the 6D pose of an object from an image is a fundamental problem in computer vision. It has many important applications, such as robotic grasping, autonomous navigation, and augmented reality.

[0003] Currently, existing methods for estimating the 6D pose of objects include using deep learning methods to extract 3D-2D keypoint correspondences for preliminary pose estimation, then iteratively optimizing the preliminary pose estimate by minimizing the collinearity error in the object space to obtain a refined pose. However, this method is unable to estimate the pose of unfamiliar objects. Other methods also include using deep learning to estimate the target pose from RGBD images, moving the camera multiple times to estimate and iteratively optimize the pose. However, this method still cannot estimate the pose of unfamiliar objects, and RGBD cameras lose depth information when imaging transparent objects, affecting pose estimation performance. Summary of the Invention

[0004] The embodiments of the present application provide a method, apparatus, device, and storage medium for estimating the pose of an object. To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is provided below. This summary is not intended to be a comprehensive review, identify key / important components, or delineate the scope of protection of these embodiments. Its sole purpose is to present some concepts in a simplified form as a prelude to the detailed description that follows.

[0005] In a first aspect, an embodiment of the present application provides a method for estimating the pose of an object, comprising:

[0006] capturing a first color depth image of an object, and extracting feature points of the first color depth image;

[0007] Based on the feature points of the first color depth image, selecting a second color depth image that matches the first color depth image from a pre-stored file corresponding to the object; wherein the pre-stored file includes multiple color depth images of the object and the feature points of each color depth image;

[0008] Calculating an initial pose of the object based on feature points matched between the first color depth image and the second color depth image;

[0009] The initial pose is iteratively optimized according to the color mode used to reconstruct the three-dimensional model of the object in the pre-stored file and the confidence of using the color mode to obtain the final pose of the object.

[0010] In one embodiment, calculating an initial pose of an object based on feature points matched between the first color depth image and the second color depth image includes:

[0011] Obtaining point cloud data and pose corresponding to the second color depth image from a pre-stored file of the object;

[0012] Acquire a relative pose between the first color depth image and the second color depth image according to the matched feature points between the first color depth image and the second color depth image and the point cloud data;

[0013] The product of the relative pose and the pose corresponding to the second color depth image is calculated to obtain an initial pose of the object.

[0014] In one embodiment, the initial pose is iteratively optimized based on the color mode used to reconstruct the three-dimensional model of the object in the pre-stored file and the confidence of the color mode to obtain the final pose of the object, including:

[0015] Obtaining a color mode used to reconstruct a three-dimensional model of the object and a confidence level of the color mode from a pre-stored file of the object;

[0016] When the confidence level is greater than or equal to a preset first threshold, iteratively optimizing the initial pose using an optimization algorithm corresponding to the color mode to obtain a final pose of the object;

[0017] When the confidence level is greater than or equal to a preset second threshold and less than the preset first threshold, the initial pose is iteratively optimized using the optimization algorithms corresponding to the RGB mode and the RGBD mode, respectively, to obtain the first pose and the second pose of the object; based on the first pose and the second pose, the final pose of the object is calculated; wherein the color mode in the pre-stored file is one of the RGB mode and the RGBD mode.

[0018] In one embodiment, iteratively optimizing the initial pose using an optimization algorithm corresponding to the color mode to obtain a final pose of the object includes:

[0019] When the color mode is RGB mode, obtaining an RGB image corresponding to the first color depth image, and iteratively optimizing the initial pose using a first optimization algorithm corresponding to the RGB mode based on the RGB image and the three-dimensional model of the object in the pre-stored file to obtain a final pose of the object;

[0020] When the color mode is RGBD mode, obtain point cloud data corresponding to the first color depth image, and based on the point cloud data and the three-dimensional model of the object in the pre-stored file, use the second optimization algorithm corresponding to the RGBD mode to iteratively optimize the initial pose to obtain the final pose of the object.

[0021] In one embodiment, calculating the final pose of the object according to the first pose and the second pose includes:

[0022] Determining a weight of the first posture and a weight of the second posture respectively according to the color mode and the confidence level in the pre-stored file;

[0023] According to the weight of the first posture and the weight of the second posture, the first posture and the second posture are weighted and summed to obtain a final posture of the object.

[0024] In one embodiment, based on feature points of the first color depth image, selecting a second color depth image that matches the first color depth image from a pre-stored file corresponding to the object includes:

[0025] Determining, based on the feature points of the first color depth image and the feature points of each color depth image in the pre-stored file corresponding to the object, feature points that match the first color depth image; wherein the matched feature points are feature points whose matching degree calculated by a preset matching algorithm is greater than a preset threshold;

[0026] The color depth image with the largest number of matched feature points is used as the second color depth image that matches the first color depth image.

[0027] In one embodiment, before selecting a second color depth image that matches the first color depth image from a pre-stored file corresponding to the object, the method further includes:

[0028] photographing the object from multiple angles to obtain multiple color depth images of the object;

[0029] Inputting the plurality of color depth images into a preset color mode selection model to obtain a selected color mode and a confidence level;

[0030] Reconstruct the object in three dimensions using a reconstruction algorithm corresponding to the selected color mode according to the multiple color depth images to obtain a pose and a three-dimensional model corresponding to each color depth image;

[0031] Extract feature points of each color depth image;

[0032] Based on each color depth image, the posture and feature points corresponding to each color depth image, the reconstructed three-dimensional model, the selected color mode and the confidence level, a pre-stored file of the object is generated, and the pre-stored file is stored.

[0033] In one embodiment, inputting the plurality of color depth images into a preset color mode selection model to obtain a selected color mode and a confidence level includes:

[0034] Inputting the plurality of color depth images into a preset color mode selection model, and obtaining a mode selection result and a confidence level corresponding to each color image;

[0035] Based on the number of mode selection results indicating the RGB mode being greater than the number of mode selection results indicating the RGBD mode, selecting the RGB mode; calculating an average of the confidence levels corresponding to all mode selection results indicating the RGB mode, and using the average as the confidence level for selecting the RGB mode;

[0036] Based on the fact that the number of mode selection results indicating the RGB mode is less than or equal to the number of mode selection results indicating the RGBD mode, the RGBD mode is selected; and an average of the confidences corresponding to all mode selection results indicating the RGBD mode is calculated, and the average is used as the confidence for selecting the RGBD mode.

[0037] In one embodiment, based on the multiple color depth images, a reconstruction algorithm corresponding to the selected color mode is used to perform three-dimensional reconstruction of the object to obtain a pose and a three-dimensional model corresponding to each color depth image, including:

[0038] When the selected color mode is RGB mode, obtain an RGB image corresponding to each color depth image, and perform three-dimensional reconstruction on the obtained multiple RGB images using a reconstruction algorithm corresponding to the RGB mode to obtain a pose and a three-dimensional model corresponding to each color depth image;

[0039] When the selected color mode is the RGBD mode, the multiple color depth images are three-dimensionally reconstructed using a reconstruction algorithm corresponding to the RGBD mode to obtain a posture and a three-dimensional model corresponding to each color depth image.

[0040] In a second aspect, an embodiment of the present application provides a device for estimating the pose of an object, comprising:

[0041] a feature extraction module, configured to capture a first color depth image of an object and extract feature points from the first color depth image;

[0042] a feature matching module, configured to select, from a pre-stored file corresponding to the object, a second color depth image that matches the first color depth image based on feature points of the first color depth image; wherein the pre-stored file includes multiple color depth images of the object and feature points of each color depth image;

[0043] a posture estimation module, configured to calculate an initial posture of the object based on feature points matched between the first color depth image and the second color depth image;

[0044] The posture optimization module is used to iteratively optimize the initial posture according to the color mode used to reconstruct the three-dimensional model of the object in the pre-stored file and the confidence of the color mode to obtain the final posture of the object.

[0045] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor and a memory storing program instructions, wherein the processor is configured to execute the object pose estimation method provided in the above embodiment when executing the program instructions.

[0046] In a fourth aspect, an embodiment of the present application provides a computer-readable medium having computer-readable instructions stored thereon, and the computer-readable instructions are executed by a processor to implement a method for estimating the pose of an object provided in the above embodiment.

[0047] The technical solutions provided by the embodiments of the present application may have the following beneficial effects:

[0048] The object pose estimation method provided in the embodiments of this application does not require the construction of a CAD model of the unfamiliar object. Instead, it only requires the pre-registration of multiple color depth images of the object from different angles, along with information such as color patterns, confidence levels, and feature points. By matching these features, a pre-stored file corresponding to the unfamiliar object can be obtained, thereby enabling pose estimation of the unfamiliar object. This solution also performs iterative optimization based on the object's different color patterns to avoid the loss of depth information when imaging transparent objects with an RGBD camera, which can affect pose estimation performance. Therefore, this solution is adaptable to a wider range of application scenarios and offers improved pose estimation performance.

[0049] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0051] Figure 1 is a schematic diagram of a method for estimating the pose of an object according to an exemplary embodiment;

[0052] Figure 2 is a schematic diagram of a method for pre-registering an object according to an exemplary embodiment;

[0053] Figure 3 is a schematic diagram of a method for object registration and pose estimation according to an exemplary embodiment;

[0054] Figure 4 is a schematic diagram showing a method for photographing an object according to an exemplary embodiment;

[0055] Figure 5 is a schematic structural diagram of a device for estimating the position and posture of an object according to an exemplary embodiment;

[0056] Figure 6 is a schematic structural diagram of an electronic device according to an exemplary embodiment;

[0057] Figure 7 It is a schematic diagram of a computer storage medium according to an exemplary embodiment. DETAILED DESCRIPTION

[0058] The following description and the drawings sufficiently illustrate specific embodiments of the invention to enable those skilled in the art to practice them.

[0059] It should be understood that the embodiments described are only a portion of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative work are within the scope of protection of the present invention.

[0060] In the following description, unless otherwise indicated, identical numbers in different figures represent identical or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of systems and methods consistent with certain aspects of the present invention, as detailed in the appended claims.

[0061] The following is a detailed description of the object pose estimation method provided by the embodiment of the present application in conjunction with the accompanying drawings. Figure 1 , the method specifically includes the following steps.

[0062] S101 : Capture a first color depth image of an object, and extract feature points of the first color depth image.

[0063] In one possible implementation, an arbitrarily placed object is photographed and imaged by an RGBD camera to obtain a first color depth image of the object, wherein the photographed object may be an unfamiliar object.

[0064] Furthermore, feature points of the captured first color depth image are extracted. Optionally, feature points of the first color depth image are extracted using a Super Point algorithm. Optionally, feature points of the first color depth image are extracted using a SIFT algorithm. Optionally, feature points of the first color depth image are extracted using an ORB algorithm. The embodiments of the present application do not limit the specific feature point extraction method, and the method can be selected according to actual needs.

[0065] S102. Based on the feature points of the first color depth image, select a second color depth image that matches the first color depth image from a pre-stored file corresponding to the object; wherein the pre-stored file includes multiple color depth images of the object and feature points of each color depth image.

[0066] The object pose estimation method provided in the embodiment of the present application includes a registration phase and a pose estimation phase. The object needs to be registered first and the pre-stored file corresponding to the object is stored in a database.

[0067] Figure 2 FIG. 1 is a schematic diagram showing a method for pre-registering an object according to an exemplary embodiment. Figure 2 As shown, the following steps are included.

[0068] S201 photographs an object from multiple angles to obtain multiple color depth images of the object.

[0069] When registering an object, the RGBD camera is used to capture color and depth images of the object from multiple angles. Figure 4 As shown, the object is placed at a specified location, and the RGBD camera is moved using a robotic arm or other device. It can move along a fixed route, and the N position points where it stays during the movement are evenly distributed on the upper hemisphere of the object. The stop position points are as follows: Figure 4 The camera is oriented toward the spherical surface normal during the pause. An RGBD image is captured at each pause point, resulting in color depth images of the object captured from different angles.

[0070] S202 inputs a plurality of color depth images into a preset color mode selection model to obtain a selected color mode and confidence level.

[0071] Specifically, multiple color depth images are input into a preset color mode selection model to obtain the mode selection results and confidence levels corresponding to each color image. The input of the color mode selection model is a 4-channel image, which is composed of a 224x224 central area of the RGB image and a 224x224 central area of the depth image connected in series. The main structure of the deep learning network is resnet50. A large number of public data sets and synthetic data are used during training to ensure strong generalization capabilities. The model output is 0 or 1 and a confidence level, where 0 and 1 represent the selected color mode, for example, 0 represents RGB mode and 1 represents RGBD mode. The confidence level indicates the reliability of the mode selection result, which is a probability value between 0 and 1. The larger the probability value, the higher the confidence level. The confidence threshold is 0.5, and the results with a confidence level greater than or equal to 0.5 are saved.

[0072] After inputting multiple color depth images into the color mode selection model, the RGB mode is selected based on the number of mode selection results indicating RGB mode being greater than the number of mode selection results indicating RGBD mode. The confidence levels corresponding to all mode selection results indicating RGB mode are averaged and used as the confidence level for selecting the RGB mode. For example, if the number of results indicating 0 is greater than the number of results indicating 1, the RGB mode is selected.

[0073] If the number of mode selection results indicating the RGB mode is less than or equal to the number of mode selection results indicating the RGBD mode, the RGBD mode is selected. The confidence levels corresponding to all mode selection results indicating the RGBD mode are averaged, and this average is used as the confidence level for selecting the RGBD mode. For example, if the number of results indicating a 0 is less than or equal to the number of results indicating a 1, the RGBD mode is selected.

[0074] S203 performs three-dimensional reconstruction of the object based on the multiple color depth images using a reconstruction algorithm corresponding to the selected color mode to obtain a posture and a three-dimensional model corresponding to each color depth image.

[0075] Specifically, when the selected color mode is RGB, an RGB image corresponding to each color depth image is obtained. Three-dimensional reconstruction is performed on these multiple RGB images using a reconstruction algorithm corresponding to the RGB mode, obtaining the pose and 3D model corresponding to each color depth image. For example, when the RGB mode lacks image depth information, the Colmap algorithm is used to perform dense 3D reconstruction of the object on the N collected RGB images, obtaining the pose and 3D model corresponding to each color depth image. Those skilled in the art may also employ other algorithms for 3D reconstruction based solely on color images.

[0076] When the selected color mode is RGBD, the 3D reconstruction algorithm corresponding to the RGBD mode is used to reconstruct multiple color depth images, obtaining the pose and 3D model corresponding to each color depth image. For example, when the RGBD mode is selected, the Bundle Fusion algorithm is used to perform dense 3D reconstruction of the object on N RGBD images, obtaining the pose and 3D model corresponding to each color depth image.

[0077] The method provided in the embodiments of this application can intelligently select a mode based on a pre-trained model to determine the appropriate mode for the object. This allows for the use of different 3D reconstruction algorithms based on the different modes. This prevents the loss of depth information when imaging small, thin, or transparent objects with an RGBD camera, which can affect 3D reconstruction performance.

[0078] S204 extracts feature points from each color depth image.

[0079] Optionally, feature points of multiple color depth images are extracted using a Super Point algorithm. Optionally, feature points of multiple color depth images are extracted using a SIFT algorithm. Optionally, feature points of multiple color depth images are extracted using an ORB algorithm. The embodiments of this application do not limit the specific feature point extraction method, and the method can be selected according to actual needs.

[0080] S205 generates a pre-stored file of the object based on each color depth image, the posture and feature points corresponding to each color depth image, the reconstructed three-dimensional model, the selected color mode and the confidence level, and stores the pre-stored file.

[0081] Based on the collected color depth images, the pose and feature points corresponding to each color depth image, the reconstructed 3D model, the selected color mode and confidence, a pre-stored file of the object is generated and stored in the database. The storage format is a json file, which includes the storage path of the 3D model, the mode of 3D model reconstruction, the confidence of the mode selection, the storage path list of the images corresponding to the 3D model, and the file storage path list of the corresponding image feature points.

[0082] After the object registration is completed, based on the feature points of the first color depth image, a second color depth image that matches the first color depth image is selected from a pre-stored file corresponding to the object.

[0083] Specifically, based on the feature points of the first color depth image and the feature points of each color depth image in the pre-stored file corresponding to the object, feature points of each color depth image that match the first color depth image are determined. Matched feature points are feature points for which a matching degree calculated by a preset matching algorithm is greater than a preset threshold. For example, when performing feature point matching using the Superglue feature point matching algorithm, the preset threshold may be 0.3, 0.5, or the like. This application does not limit the specific value of the preset threshold.

[0084] Furthermore, the color depth image with the largest number of matched feature points is used as the second color depth image that matches the first color depth image.

[0085] S103 : Calculate an initial pose of the object based on the matched feature points between the first color depth image and the second color depth image.

[0086] In one possible implementation, point cloud data and pose corresponding to the second color depth image are obtained from a pre-stored file of the object, and the relative pose between the first color depth image and the second color depth image is obtained based on the matched feature points and point cloud data between the first color depth image and the second color depth image.

[0087] When the point cloud data corresponding to the second RGBD image and the matching feature points between the first RGBD image and the second RGBD image are known, the relative pose TAB between the first RGBD image and the second RGBD image can be obtained through the PNP algorithm. Furthermore, the product of the relative pose and the pose TB corresponding to the second color depth image is calculated to obtain the camera pose TA corresponding to the first color depth image. b Since the second color depth image is acquired in a discrete pose, there are mismatched points between the first and second color depth images. Therefore, the currently acquired camera pose TAb is used as the initial pose of the object to reduce the search space for subsequent pose optimization. The camera pose includes the 3D spatial position xyz and the rotation angle.

[0088] S104 , iteratively optimizing the initial pose according to the color mode used to reconstruct the three-dimensional model of the object in the pre-stored file and the confidence of the color mode, to obtain the final pose of the object.

[0089] Specifically, the color mode used to reconstruct the three-dimensional model of the object and the confidence of the color mode are obtained from the pre-stored file of the object. When the confidence is greater than or equal to a preset first threshold, the optimization algorithm corresponding to the color mode is used to iteratively optimize the initial pose to obtain the final pose of the object.

[0090] For example, if the confidence level is greater than or equal to 0.8, the retrieved color mode is trusted, and the initial pose is iteratively optimized using the optimization algorithm corresponding to the color mode. If the color mode is RGB, an RGB image corresponding to the first color depth image is obtained. Based on the RGB image and the three-dimensional model of the object in the pre-stored file, the initial pose is iteratively optimized using the first optimization algorithm corresponding to the RGB mode, such as the DeepIM algorithm, to obtain the final pose of the object. RGB images do not contain depth information, so a pose estimation algorithm based on color images is used. Those skilled in the art may also use other pose estimation algorithms that do not use depth information to perform iterative pose optimization.

[0091] When the color mode is RGBD, point cloud data corresponding to the first color depth image is obtained. Based on the point cloud data and the 3D model of the object in a pre-stored file, a second optimization algorithm corresponding to the RGBD mode, such as the ICP algorithm, is used to iteratively optimize the initial pose to obtain the final pose of the object. Because RGBD images contain depth information, an ICP algorithm based on point cloud registration can be used for pose optimization. This embodiment of the present application does not limit the specific optimization algorithm, and other pose optimization algorithms based on point cloud registration may also be used.

[0092] When the confidence level is greater than or equal to the preset second threshold and less than the preset first threshold, for example, the confidence level is greater than or equal to 0.5 and less than 0.8, the initial pose is iteratively optimized using the optimization algorithms corresponding to the RGB mode and the RGBD mode, respectively, to obtain the first pose and the second pose of the object; based on the first pose and the second pose, the final pose of the object is calculated; wherein the color mode in the pre-stored file is one of the RGB mode and the RGBD mode.

[0093] In one embodiment, the final pose of the object is calculated based on the first pose and the second pose, including: determining the weight of the first pose and the weight of the second pose according to the color mode and confidence in the pre-stored file, and taking a weighted sum of the first pose and the second pose according to the weight of the first pose and the weight of the second pose to obtain the final pose of the object.

[0094] Specifically, after running the DeepIM and ICP algorithms, the optimized poses TA are obtained respectively. RGB and TA D If the retrieved mode is RGB mode, the final pose of the object is TA = P × TA RGB +(1-P)×TA D , if the retrieved mode is RGBD mode, the final pose of the object TA = (1-P) × TA RGB +P×TA DThe weight value P can be set according to the actual situation and is not limited in this application.

[0095] The pose iterative optimization method provided in the embodiment of the present application can select different optimization algorithms according to different color modes, so as to adopt the most appropriate and accurate algorithm for pose estimation. When the confidence of the selected mode is low, the pose estimation results under different modes can also be fused to improve the accuracy of pose estimation. Through the intelligent switching of RGB and RGBD, it can adapt to more application scenarios and have better performance.

[0096] In order to facilitate understanding of the object pose estimation method provided in the embodiment of the present application, the following Figure 3 Provide explanation.

[0097] The overall process of the 6D pose estimation algorithm for unfamiliar objects proposed in this application is as follows Figure 3 As shown in Figure 1, the process is divided into the registration phase and the pose estimation phase. If the database does not contain a 3D model of an unfamiliar object, the registration phase is required. Once registration is complete, the pose estimation phase completes the object's pose.

[0098] The registration phase includes: placing the object at a specified location, using a robotic arm containing an RGBD camera to perform motion imaging around the upper hemisphere of the object, and obtaining multiple RGBD images taken from different angles. The multiple RGBD images taken are input into a pre-trained color mode selection model for intelligent mode selection to obtain the mode selection result and the corresponding confidence level. Different 3D reconstruction algorithms are used to perform 3D reconstruction based on the mode selection results. For example, when the selected mode is RGB mode, the colmap algorithm is used for 3D reconstruction to obtain a 3D model. When the selected mode is RGBD mode, the bundlefusion algorithm is used for 3D reconstruction to obtain a 3D model. Feature points of the multiple RGBD images collected are extracted, and the collected images, extracted feature points, reconstructed 3D models, reconstructed selected modes, and mode selection confidence levels are stored in a database in the form of a json file.

[0099] The pose estimation stage involves capturing an object image A at any pose using an RGBD camera, extracting feature points from image A, and searching for image B with the most matching points from a pre-stored file using the feature point matching algorithm Superglue. Given the point cloud data of image B and the locations of the matching feature points between images A and B, the relative pose between images A and B is determined using the PNP algorithm. The initial pose of the object is then determined by multiplying the relative pose by the pose of image B. Finally, the color mode used to reconstruct the object's 3D model and the confidence level of the color mode are retrieved from the pre-stored file. If the confidence level is greater than or equal to a preset first threshold, the initial pose is iteratively optimized using the optimization algorithm corresponding to the color mode to obtain the final pose of the object. If the color mode is RGB, the DeepIM algorithm is used to iteratively optimize the initial pose to obtain the final pose. If the color mode is RGB, the ICP algorithm is used to iteratively optimize the initial pose to obtain the final pose.

[0100] When the confidence level is greater than or equal to the preset second threshold and less than the preset first threshold, the initial pose is iteratively optimized using the optimization algorithms corresponding to the RGB mode and RGBD mode respectively to obtain the first pose and second pose of the object; based on the first pose and the second pose, the final pose of the object is calculated.

[0101] The object pose estimation method provided in the embodiments of the present application does not require the construction of a CAD model of the unfamiliar object. Instead, it only requires the pre-registration of multiple color depth images, color modes, confidence levels, feature points, and other information of the object from different angles. By matching the feature points, a pre-stored file corresponding to the unfamiliar object can be obtained, thereby achieving pose estimation of the unfamiliar object. The solution also performs iterative optimization based on the different color modes of the object to avoid the loss of depth information when the RGBD camera images transparent objects, which affects the pose estimation performance. When the confidence level of the selected mode is low, the pose estimation results of different modes can be fused to improve the accuracy of the pose estimation. The intelligent switching between RGB and RGBD can adapt to more application scenarios and achieve better pose estimation performance.

[0102] The present application also provides an object posture estimation device, which is used to perform the object posture estimation method of the above embodiment, such as Figure 5 As shown, the device includes:

[0103] A feature extraction module 501 is configured to capture a first color depth image of an object and extract feature points from the first color depth image;

[0104] A feature matching module 502 is configured to select, based on feature points of the first color depth image, a second color depth image that matches the first color depth image from a pre-stored file corresponding to the object; wherein the pre-stored file includes multiple color depth images of the object and feature points of each color depth image;

[0105] a posture estimation module 503, configured to calculate an initial posture of the object based on feature points matched between the first color depth image and the second color depth image;

[0106] The posture optimization module 504 is used to iteratively optimize the initial posture according to the color mode used to reconstruct the three-dimensional model of the object in the pre-stored file and the confidence of the color mode to obtain the final posture of the object.

[0107] It should be noted that the object pose estimation device provided in the above embodiment only uses the division of the above-mentioned functional modules as an example when executing the object pose estimation method. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the object pose estimation device provided in the above embodiment and the object pose estimation method embodiment are of the same concept. The implementation process is detailed in the method embodiment and will not be repeated here.

[0108] An embodiment of the present application further provides an electronic device corresponding to the object pose estimation method provided in the aforementioned embodiment, so as to execute the aforementioned object pose estimation method.

[0109] Please refer to Figure 6 , which shows a schematic diagram of an electronic device provided by some embodiments of the present application. Figure 6 As shown, the electronic device includes: a processor 600, a memory 601, a bus 602 and a communication interface 603, and the processor 600, the communication interface 603 and the memory 601 are connected via the bus 602; the memory 601 stores a computer program that can be run on the processor 600, and when the processor 600 runs the computer program, it executes the object pose estimation method provided in any of the aforementioned embodiments of the present application.

[0110] The memory 601 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage. The system network element communicates with at least one other network element via at least one communication interface 603 (which may be wired or wireless), and may utilize the Internet, a wide area network, a local area network, a metropolitan area network, or the like.

[0111] Bus 602 may be an ISA bus, a PCI bus, or an EISA bus. Buses may be classified as address buses, data buses, and control buses. Memory 601 is used to store programs, and processor 600 executes the programs upon receiving execution instructions. The object pose estimation method disclosed in any of the aforementioned embodiments of the present application may be applied to or implemented by processor 600.

[0112] The processor 600 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor 600 or by software instructions. The above processor 600 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 601 , and the processor 600 reads the information in the memory 601 and completes the steps of the above method in combination with its hardware.

[0113] The electronic device provided in the embodiment of the present application and the method for estimating the pose of an object provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented therein.

[0114] The present application also provides a computer-readable storage medium corresponding to the method for estimating the pose of an object provided in the above embodiment. Figure 7 The computer-readable storage medium shown is a CD 700, on which a computer program (i.e., a program product) is stored. When the computer program is run by the processor, it will execute the object pose estimation method provided by any of the aforementioned embodiments.

[0115] It should be noted that examples of computer-readable storage media may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0116] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the object pose estimation method provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0117] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0118] The above embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A method for estimating the pose of an object, characterized in that: include: capturing a first color depth image of an object, and extracting feature points of the first color depth image; Based on the feature points of the first color depth image, selecting a second color depth image that matches the first color depth image from a pre-stored file corresponding to the object; wherein the pre-stored file includes multiple color depth images of the object and the feature points of each color depth image; Calculating an initial pose of the object based on feature points matched between the first color depth image and the second color depth image; The initial pose is iteratively optimized according to the color mode used to reconstruct the three-dimensional model of the object in the pre-stored file and the confidence of using the color mode to obtain the final pose of the object.

2. The method according to claim 1, characterized in that Calculating an initial pose of the object based on feature points matched between the first color depth image and the second color depth image, including: Obtaining point cloud data and pose corresponding to the second color depth image from a pre-stored file of the object; Acquire a relative pose between the first color depth image and the second color depth image according to the matched feature points between the first color depth image and the second color depth image and the point cloud data; The product of the relative pose and the pose corresponding to the second color depth image is calculated to obtain an initial pose of the object.

3. The method according to claim 1, characterized in that Iteratively optimizing the initial pose according to the color mode used to reconstruct the three-dimensional model of the object in the pre-stored file and the confidence of the color mode to obtain the final pose of the object, including: Obtaining a color mode used to reconstruct a three-dimensional model of the object and a confidence level of the color mode from a pre-stored file of the object; When the confidence level is greater than or equal to a preset first threshold, iteratively optimizing the initial pose using an optimization algorithm corresponding to the color mode to obtain a final pose of the object; When the confidence level is greater than or equal to a preset second threshold and less than the preset first threshold, the initial pose is iteratively optimized using the optimization algorithms corresponding to the RGB mode and the RGBD mode, respectively, to obtain the first pose and the second pose of the object; based on the first pose and the second pose, the final pose of the object is calculated; wherein the color mode in the pre-stored file is one of the RGB mode and the RGBD mode.

4. The method according to claim 3, characterized in that Iteratively optimizing the initial pose using an optimization algorithm corresponding to the color mode to obtain a final pose of the object includes: When the color mode is RGB mode, obtaining an RGB image corresponding to the first color depth image, and iteratively optimizing the initial pose using a first optimization algorithm corresponding to the RGB mode based on the RGB image and the three-dimensional model of the object in the pre-stored file to obtain a final pose of the object; When the color mode is RGBD mode, obtain point cloud data corresponding to the first color depth image, and based on the point cloud data and the three-dimensional model of the object in the pre-stored file, use the second optimization algorithm corresponding to the RGBD mode to iteratively optimize the initial pose to obtain the final pose of the object.

5. The method according to claim 3, characterized in that Calculating the final pose of the object according to the first pose and the second pose includes: Determining a weight of the first posture and a weight of the second posture respectively according to the color mode and the confidence level in the pre-stored file; The first posture and the second posture are weighted and summed according to the weight of the first posture and the weight of the second posture to obtain a final posture of the object.

6. The method according to claim 1, characterized in that Selecting, based on feature points of the first color depth image, a second color depth image that matches the first color depth image from a pre-stored file corresponding to the object, comprising: Determining, based on the feature points of the first color depth image and the feature points of each color depth image in the pre-stored file corresponding to the object, feature points that match the first color depth image; wherein the matched feature points are feature points whose matching degree calculated by a preset matching algorithm is greater than a preset threshold; The color depth image with the largest number of matched feature points is used as the second color depth image that matches the first color depth image.

7. The method according to claim 1, characterized in that Before selecting a second color depth image that matches the first color depth image from a pre-stored file corresponding to the object, the method further includes: photographing the object from multiple angles to obtain multiple color depth images of the object; Inputting the plurality of color depth images into a preset color mode selection model to obtain a selected color mode and a confidence level; Reconstruct the object in three dimensions using a reconstruction algorithm corresponding to the selected color mode according to the multiple color depth images to obtain a pose and a three-dimensional model corresponding to each color depth image; Extract feature points of each color depth image; Based on each color depth image, the posture and feature points corresponding to each color depth image, the reconstructed three-dimensional model, the selected color mode and the confidence level, a pre-stored file of the object is generated, and the pre-stored file is stored.

8. The method according to claim 7, characterized in that Inputting the plurality of color depth images into a preset color mode selection model to obtain a selected color mode and a confidence level, comprising: Inputting the plurality of color depth images into a preset color mode selection model, and obtaining a mode selection result and a confidence level corresponding to each color image; Based on the number of mode selection results indicating the RGB mode being greater than the number of mode selection results indicating the RGBD mode, selecting the RGB mode; calculating an average of the confidence levels corresponding to all mode selection results indicating the RGB mode, and using the average as the confidence level for selecting the RGB mode; Based on the fact that the number of mode selection results indicating the RGB mode is less than or equal to the number of mode selection results indicating the RGBD mode, the RGBD mode is selected; and an average of the confidences corresponding to all mode selection results indicating the RGBD mode is calculated, and the average is used as the confidence for selecting the RGBD mode.

9. The method according to claim 7, characterized in that Performing a three-dimensional reconstruction of the object according to the multiple color depth images using a reconstruction algorithm corresponding to the selected color mode to obtain a pose and a three-dimensional model corresponding to each color depth image, including: When the selected color mode is RGB mode, obtain an RGB image corresponding to each color depth image, and perform three-dimensional reconstruction on the obtained multiple RGB images using a reconstruction algorithm corresponding to the RGB mode to obtain a pose and a three-dimensional model corresponding to each color depth image; When the selected color mode is the RGBD mode, the multiple color depth images are three-dimensionally reconstructed using a reconstruction algorithm corresponding to the RGBD mode to obtain a posture and a three-dimensional model corresponding to each color depth image.

10. A device for estimating the position and posture of an object, characterized in that: include: a feature extraction module, configured to capture a first color depth image of an object and extract feature points from the first color depth image; a feature matching module, configured to select, from a pre-stored file corresponding to the object, a second color depth image that matches the first color depth image based on feature points of the first color depth image; wherein the pre-stored file includes multiple color depth images of the object and feature points of each color depth image; a posture estimation module, configured to calculate an initial posture of the object based on feature points matched between the first color depth image and the second color depth image; The posture optimization module is used to iteratively optimize the initial posture according to the color mode used to reconstruct the three-dimensional model of the object in the pre-stored file and the confidence of the color mode to obtain the final posture of the object.

11. An electronic device, characterized in that: The method comprises a processor and a memory storing program instructions, wherein the processor is configured to execute the method for estimating the pose of an object according to any one of claims 1 to 9 when executing the program instructions.

12. A computer-readable medium, characterized in that Computer-readable instructions are stored thereon, and the computer-readable instructions are executed by a processor to implement a method for estimating the pose of an object as described in any one of claims 1 to 9.

Images