6D pose estimation method and device and electronic equipment

Abstract

The invention provides a 6D pose estimation method and device and electronic equipment, and relates to the technical field of computers. The method comprises the steps that a scene graph is obtained, a mask graph corresponding to the scene graph is obtained, the scene graph comprises a target object, and the scene graph comprises a scene RGB graph and a scene depth graph; a plurality of first poses are determined, the first poses are 6D poses, and depth related information in the first poses is determined according to the mask image and the scene depth image; obtaining an image group corresponding to each first pose, wherein the image group comprises a first RGBD image of a target three-dimensional model corresponding to the target object under a visual angle corresponding to the first pose, and a second RGBD image obtained by cutting the scene image according to the first pose; and inputting each image group into a trained pose estimation network to obtain a target 6D pose of the target object, and obtaining the target 6D pose by the pose estimation network based on the matching degree of the images in the image groups. Therefore, the 6D pose of the object can be quickly and accurately estimated.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more specifically, to a 6D pose estimation method, device and electronic device. Background Art

[0002] The disordered grasping of objects in deep baskets is an important process in automated production, which is generally achieved in the following way: first, use a camera to take pictures to obtain the necessary image or point cloud information; then, each independent object in the image is segmented by an instance segmentation algorithm, and the pose of each object is estimated by a 6D pose (i.e., 3D position and 3D direction) estimation algorithm; finally, the estimation result is sent to the robot, and the robot performs the grasping action. In this process, 6D pose estimation is the most important link, and its accuracy will directly affect the success rate of grasping. Therefore, how to quickly and accurately estimate the 6D pose has become a technical problem that technicians in this field need to solve urgently. Summary of the invention

[0003] The embodiments of the present application provide a 6D pose estimation method, device, electronic device and readable storage medium, which can quickly and accurately estimate the 6D pose of an object.

[0004] The embodiments of the present application can be implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides a 6D pose estimation method, the method comprising:

[0006] Obtain a scene graph and obtain a mask graph corresponding to the scene graph, wherein the scene graph includes a target object, and the scene graph includes a scene RGB graph and a scene depth graph;

[0007] Determine a plurality of first poses, wherein the first poses are 6D poses, and depth-related information in the first poses is determined based on the mask image and the scene depth map;

[0008] Obtaining an image group corresponding to each first pose, wherein the image group includes a first RGBD image of a target three-dimensional model corresponding to the target object at a viewing angle corresponding to the first pose, and a second RGBD image obtained by cropping the scene image according to the first pose;

[0009] Each image group is input into a trained pose estimation network to obtain a target 6D pose of the target object, wherein the pose estimation network obtains the target 6D pose based on the matching degree of images in the image group.

[0010] In a second aspect, an embodiment of the present application provides a 6D pose estimation device, the device comprising:

[0011] A scene information acquisition module, used to obtain a scene graph and a mask graph corresponding to the scene graph, wherein the scene graph includes a target object, and the scene graph includes a scene RGB graph and a scene depth graph;

[0012] A determination module, configured to determine a plurality of first poses, wherein the first poses are 6D poses, and depth-related information in the first poses is determined according to the mask image and the scene depth image;

[0013] A processing module, used to obtain an image group corresponding to each first pose, wherein the image group includes a first RGBD image of a target three-dimensional model corresponding to the target object at a viewing angle corresponding to the first pose, and a second RGBD image obtained by cropping the scene image according to the first pose;

[0014] The estimation module is used to input each image group into a trained pose estimation network to obtain a target 6D pose of the target object, wherein the pose estimation network obtains the target 6D pose based on the matching degree of images in the image group.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor can execute the machine executable instructions to implement the 6D pose estimation method described in the aforementioned embodiment.

[0016] In a fourth aspect, an embodiment of the present application provides a readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the 6D pose estimation method as described in the aforementioned embodiment is implemented.

[0017] The 6D pose estimation method, device, electronic device and readable storage medium provided in the embodiments of the present application first obtain a scene graph including a target object and a mask graph corresponding to the scene graph, then determine multiple first poses in combination with the scene graph, and obtain image groups corresponding to each first pose, and finally input the above image groups into a trained pose estimation network to obtain the target 6D pose of the target object; the above scene graph includes a scene RGB graph and a scene depth graph, the first pose is a 6D pose, the depth-related information in the first pose is determined according to the mask graph and the scene depth graph, the image group includes a first RGBD graph of the target three-dimensional model corresponding to the target object at the viewing angle corresponding to the first pose, and a second RGBD graph obtained by cropping the scene graph according to the first pose, and the pose estimation network obtains the target 6D pose based on the matching degree of the images in the image group. In this way, the 6D pose of the object can be estimated quickly and accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A block diagram of an electronic device provided in an embodiment of the present application;

[0020] Figure 2 One of the flowcharts of the 6D pose estimation method provided in the embodiment of the present application;

[0021] Figure 3 for Figure 2 A schematic flow chart of the sub-steps included in step S300;

[0022] Figure 4 A block diagram of the 6D pose estimation method provided in the embodiment of the present application;

[0023] Figure 5 for Figure 2 A schematic flow chart of the sub-steps included in step S500;

[0024] Figure 6 for Figure 5 A schematic flow chart of the sub-steps included in sub-step S520;

[0025] Figure 7 for Figure 6 A schematic flow chart of the sub-steps included in sub-step S522;

[0026] Figure 8 The second flowchart of the 6D pose estimation method provided in the embodiment of the present application;

[0027] Fig. 9 for Figure 8 A schematic flow chart of the sub-steps included in step S110;

[0028] Fig.10 A schematic diagram of the process of obtaining a model to be used provided by the embodiment itself;

[0029] Fig.11 One of the block diagrams of the 6D pose estimation device provided in an embodiment of the present application;

[0030] Fig.12 The second block diagram of the 6D pose estimation device provided in an embodiment of the present application.

[0031] Icon: 100 - electronic device; 110 - memory; 120 - processor; 130 - communication unit; 200 - 6D pose estimation device; 210 - training module; 220 - scene information acquisition module; 230 - determination module; 240 - processing module; 250 - estimation module. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0033] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0034] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0035] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0036] Please refer to Figure 1 , Figure 1A block diagram of an electronic device 100 provided in an embodiment of the present application. The electronic device 100 may be, but is not limited to, a computer, a server, etc. The electronic device 100 may include a memory 110, a processor 120, and a communication unit 130. The memory 110, the processor 120, and the communication unit 130 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components may be electrically connected to each other via one or more communication buses or signal lines.

[0037] The memory 110 is used to store programs or data. The memory 110 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc.

[0038] The processor 120 is used to read / write data or programs stored in the memory 110 and perform corresponding functions. For example, the memory 110 stores a 6D pose estimation device 200, and the 6D pose estimation device 200 includes at least one software function module that can be stored in the memory 110 in the form of software or firmware. The processor 120 executes various functional applications and data processing by running software programs and modules stored in the memory 110, such as the 6D pose estimation device 200 in the embodiment of the present application, that is, implementing the 6D pose estimation method in the embodiment of the present application.

[0039] The communication unit 130 is used to establish a communication connection between the electronic device 100 and other communication terminals through a network, and to send and receive data through the network.

[0040] It should be understood that Figure 1 The structure shown is only a schematic diagram of the structure of the electronic device 100. The electronic device 100 may also include Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown. Figure 1 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0041] Please refer to Figure 2 , Figure 2This is one of the flow diagrams of the 6D pose estimation method provided in the embodiment of the present application. The method can be applied to the above-mentioned electronic device 100. The specific process of the 6D pose estimation method is described in detail below. In this embodiment, the method may include steps S200 to S500.

[0042] Step S200: obtaining a scene graph, and obtaining a mask graph corresponding to the scene graph.

[0043] In this embodiment, a 3D camera can be used to take pictures in a real scene, and the obtained image can be used as a scene graph. The scene graph includes a target object, and the target object is an object that needs to be estimated in 6D pose. The scene graph includes a scene RGB graph and a scene depth graph, that is, the scene graph can be an RGBD graph. The target object can be identified by the scene RGB graph in the scene graph, thereby obtaining a mask graph. Optionally, the scene RGB graph can be analyzed by an instance segmentation algorithm to obtain the mask graph, or the mask graph can be obtained by other means. The mask graph is used to indicate the position of the target object in the scene RGB graph and the scene depth graph.

[0044] Step S300, determining multiple first poses.

[0045] In this embodiment, multiple first poses can be determined by combining the mask image and the scene depth map in any manner. The first pose is a 6D pose, and the depth-related information in the first pose is determined based on the mask image and the scene depth map. The first pose is a 6D pose of a hypothetical target object in the scene map.

[0046] Step S400, obtaining image groups corresponding to each first pose.

[0047] For a first pose, an image group corresponding to the first pose can be obtained according to the target three-dimensional model corresponding to the target object and the scene graph. The image group corresponding to a first pose includes a first RGBD image of the target three-dimensional model corresponding to the target object at the viewing angle corresponding to the first pose, and a second RGBD image obtained by cropping the scene graph according to the first pose. The above processing is performed for multiple first poses respectively, and multiple image groups corresponding to each of the first poses can be obtained. The number of image groups obtained is the same as the number of determined first poses.

[0048] Step S500, input each image group into a trained pose estimation network to obtain a target 6D pose of the target object.

[0049] In this embodiment, a pre-trained pose estimation network is stored in the electronic device, and the pose estimation network may be pre-trained by the electronic device, or may be pre-trained by the electronic device. The pose estimation network is trained according to a training data set, and a training data may include a sample image group corresponding to each of a plurality of first sample poses corresponding to a sample scene graph, and a real pose corresponding to the sample scene graph. The pose estimation network may be an end-to-end network based on deep learning.

[0050] When the above image group is obtained, the above image group can be input into a trained pose estimation network, and the pose estimation network obtains the target 6D pose of the target object based on the matching degree of the images in the image group. In this way, the target 6D pose can be quickly and accurately estimated. The target 6D pose includes the position and rotation matrix of the target object in the camera coordinate system.

[0051] As a possible implementation, Figure 3 As shown, multiple first poses are determined. Figure 3 , Figure 3 for Figure 2 Schematic diagram of the flow of sub-steps included in step S300. In this embodiment, step S300 may include sub-steps S310 to S330.

[0052] Sub-step S310, obtaining two-dimensional images of the target three-dimensional model at different viewing angles.

[0053] Sub-step S320, obtaining a target depth value corresponding to a target object in the scene image according to the mask image and the scene depth image.

[0054] Sub-step S330, for each two-dimensional image, according to the plane pose of the target object indicated by the two-dimensional image and the target depth value, obtain a first pose corresponding to the two-dimensional image.

[0055] In this embodiment, two-dimensional images of the target three-dimensional model at different viewing angles can be obtained in the simulation environment. Each two-dimensional image indicates a plane posture of the target object. Optionally, the specific viewing angle selection method can be determined in combination with actual needs and is not specifically limited here. As a possible implementation method, Figure 4 As shown, polyhedron sampling is performed on the target three-dimensional model to obtain two-dimensional images under different viewing angles, and then to obtain image groups corresponding to multiple first poses.

[0056] The method of obtaining multiple two-dimensional images by polyhedron sampling can be as follows. First, a regular polyhedron is generated for the target three-dimensional model, and the target three-dimensional model is located inside the regular polyhedron; then, for each face of the regular polyhedron, assuming that there is an observation point (also called a camera, viewing angle) on the face, a two-dimensional image can be firstly collected based on the observation point toward the center of the regular polyhedron to obtain a two-dimensional image, and then the observation direction can be adjusted multiple times while the position of the observation point remains unchanged and the two-dimensional image is collected to obtain multiple two-dimensional images. Optionally, in the above process, uniform sampling can be performed to cover as many different viewing angles of the target object as possible.

[0057] The depth values ​​of each position of the target object in the scene map can be obtained according to the mask map and the scene depth map, and a value can be determined as the target depth value according to the obtained depth value. For example, the mode value, average value, etc. of the depth values ​​of each position of the target object in the scene map can be used as the target depth value. Optionally, in this embodiment, the median value of the depth values ​​of each position of the target object in the scene map is used as the target depth value.

[0058] Each 2D image indicates a plane pose of the target object, and combined with the target depth value, a 6D pose as the first pose can be obtained. It can be understood that the plane image includes x and y coordinates, and the target depth value is the z coordinate, and the combination can obtain the 3D position of the object.

[0059] For a first pose, a first RGBD image of the target three-dimensional model at the viewing angle corresponding to the first pose can be obtained, that is, the target three-dimensional model is converted into an RGB image and a depth image at a specific viewing angle corresponding to the first pose, and a second RGBD image is obtained by cropping the scene image according to the first pose, so as to obtain an image group corresponding to the first pose.

[0060] Optionally, when the target three-dimensional model is a model including texture, single-view rendering can be performed based on a first pose, that is, the target three-dimensional model of the target object is converted into an RGB image and a depth image at a specific viewing angle corresponding to the first pose.

[0061] A two-dimensional image corresponding to a first pose indicates the location area of ​​the target object in the two-dimensional image. According to the area where the target object is located in the two-dimensional image corresponding to the first pose, the scene RGB image and the scene depth image of the scene graph are cropped to obtain a second RGBD image corresponding to the first RGBD image corresponding to the first pose. That is, the position of the second RGBD image in an image group in the scene graph is the same as the position of the target object in a two-dimensional image. In this way, an image group corresponding to a first pose can be obtained.

[0062] Repeat the above process for the remaining first poses, and then the image groups corresponding to the determined first poses can be obtained.

[0063] Optionally, in this embodiment, if Figure 4 As shown, the pose estimation network includes a pose optimization network and an estimation network. In this case, Figure 5 The target 6D pose is obtained in the manner shown. Figure 5 , Figure 5 for Figure 2 Schematic diagram of the flow of sub-steps included in step S500. In this embodiment, step S500 may include sub-steps S510 to S520.

[0064] Sub-step S510: For each image group, the pose optimization network is used to optimize the first pose corresponding to the image group based on the image group to obtain a second pose.

[0065] Sub-step S520, obtaining the target 6D pose according to the second pose corresponding to each image group.

[0066] In this embodiment, the pose optimization network improves the matching degree of two images in the image by optimizing the 6D pose indicated by the first RGBD image in the image group. When multiple image groups corresponding to each of the first poses are determined, the multiple image groups can be input into the pose optimization network.

[0067] For an image group, the posture optimization network optimizes the first posture corresponding to the first image group, that is, adjusts the first RGBD image in the image group so that the matching degree between the adjusted first RGBD image and the second RGBD image in the image is improved (that is, there are more similar parts). It can be seen that the matching degree of the image group corresponding to the optimized posture is greater than or equal to the matching degree of the image group corresponding to the optimized posture before optimization, and the optimized image group includes the second RGBD image and the first RGBD image adjusted based on the optimized posture. The output of the posture optimization network can be used as the second posture obtained after optimizing the first posture corresponding to the image group. Repeat the above process to obtain the second posture corresponding to each image group.

[0068] Optionally, when the second pose corresponding to each image group is obtained, the estimation network can be used to obtain the target 6D pose based on the obtained multiple second poses in any manner, and the specific manner can be determined in combination with actual needs.

[0069] As a possible implementation, Figure 6 The target 6D pose is obtained in the manner shown. Figure 6 , Figure 6 for Figure 5 Schematic diagram of the flow of sub-steps included in sub-step S520. In this embodiment, sub-step S520 may include sub-steps S521 and S522.

[0070] Sub-step S521, for each second posture, according to the optimized image group under the second posture, calculate the matching score corresponding to the second posture.

[0071] Sub-step S522, obtaining the target 6D posture according to each second posture and the matching score corresponding to each second posture.

[0072] In this embodiment, by performing posture optimization based on the image groups corresponding to the first postures, the optimized image groups corresponding to the second postures can be obtained. It can be understood that the optimized image group corresponding to a second posture includes the second RGBD image in the image group corresponding to the first posture corresponding to the second posture, and the adjusted first RGBD image obtained by performing posture optimization on the first RGBD image in the image.

[0073] For each second posture, the matching score of the optimized image group can be calculated according to the optimized image group corresponding to the second posture. Optionally, the optimized image group corresponding to the second posture can be feature-compared through an encoder and a multi-head self-attention mechanism to generate a feature embedding vector (feature embedding) for evaluating the matching degree of the second posture, and the matching degree can be used as the matching score of the second posture.

[0074] The target 6D posture may be obtained according to the matching scores of the second postures and the obtained multiple second postures. For example, the second posture corresponding to the highest matching score is used as the target 6D posture.

[0075] Optionally, you can Figure 7 The target 6D pose is obtained in the manner shown. Figure 7 , Figure 7 for Figure 6 Schematic diagram of the flow of sub-steps included in sub-step S522. In this embodiment, sub-step S522 may include sub-steps S5221 to S5222.

[0076] Sub-step S5221, sorting the determined multiple third postures according to the matching scores to obtain a sorting result.

[0077] Sub-step S5222, based on the sorting results, a global score of each third posture is calculated through a multi-head self-attention mechanism.

[0078] Sub-step S5223, taking the third posture corresponding to the highest global score among the obtained global scores as the target 6D posture.

[0079] In this embodiment, the multiple third postures selected from the obtained second postures can be sorted in ascending or descending order according to the corresponding matching scores to obtain the sorting result. Optionally, the multiple second postures obtained can be directly sorted in ascending or descending order according to the corresponding matching scores to obtain the sorting result. Alternatively, a second posture can be selected from the obtained second postures as the third posture according to the matching scores corresponding to each second posture, wherein the matching score of the selected second posture is greater than or equal to the matching score of the unselected second posture. For example, the second posture with a matching score greater than a threshold can be used as the selected third posture, or the obtained second postures can be sorted in order from large to small according to the matching scores, and then a preset number of second postures in the front position can be used as the selected third posture.

[0080] According to the ranking results obtained based on the multiple third postures, a context association analysis can be performed in the multiple third postures through a multi-head self-attention mechanism, so as to calculate the global score of each third posture. Finally, among the global scores of each third posture, the third posture corresponding to the highest global score is used as the target 6D posture.

[0081] In this way, the estimation network uses a hierarchical comparison strategy, first performing a local comparison to calculate the matching score corresponding to each second position, and then screening out the third pose; then, a global comparison is performed on the multiple screened third poses, in which the global score of each third pose is calculated, and the third pose with the highest global score is used as the estimated target 6D pose of the target object.

[0082] The target 6D pose of the target object can be used for downstream tasks such as robot grasping. Optionally, in a general precision grasping scenario, the target 6D pose can be used as a direct pose estimation result, that is, the object can be grasped directly according to the target 6D pose.

[0083] In the high-precision grasping scenario, the above-mentioned 6D pose of the target can be used as the initial estimated pose, and then combined with the scene graph and the target 3D model for point cloud registration, so as to obtain a pose that can be used for object grasping. In this way, an initial estimated pose with a small error can be obtained, avoiding poor precision registration results due to large errors in the initial estimated pose.

[0084] Please refer to Figure 8 , Figure 8The second flowchart of the 6D pose estimation method provided in the embodiment of the present application is as follows. In the present embodiment, before step S200, the method may further include steps S110 to S130.

[0085] Step S110, obtaining a three-dimensional model to be used.

[0086] In this embodiment, the three-dimensional model to be used can be obtained in any manner, and the three-dimensional model to be used is used to obtain multiple sample data for training the pose estimation network. The three-dimensional model to be used may include model shape information, or may include model shape information and texture information. The three-dimensional model to be used may be one or more; all of them may have textures, or none of them may have textures, or some of them may have textures and some may not, which can be determined according to actual needs.

[0087] In the current deep learning-based 6D pose estimation of workpieces, a large number of images with different backgrounds, lighting conditions, stacking conditions, and viewing angles are generally collected for each specific workpiece that needs to be estimated, and the obtained images are annotated to train a model for pose estimation. This series of processes requires a lot of manpower, material resources, and time costs, and the trained model only has a good pose estimation effect on the object data used for training, and it is difficult to generalize to new objects.

[0088] In order to improve the generalization ability, in this embodiment, Fig. 9 The three-dimensional model to be used is obtained in the manner shown. Fig. 9 , Fig. 9 for Figure 8 Schematic diagram of the flow of sub-steps included in step S110. In this embodiment, step S110 may include sub-steps S111 to S112.

[0089] Sub-step S111, obtaining multiple three-dimensional models from a three-dimensional model library.

[0090] Sub-step S112, obtaining at least one piece of texture description information, and generating a plurality of to-be-used three-dimensional models according to the obtained texture description information and the three-dimensional model.

[0091] In this embodiment, if Fig.10As shown, multiple three-dimensional models can be obtained from a large-scale 3D model library, and the multiple three-dimensional models correspond to different objects, and each three-dimensional model indicates model shape information. A large language model (LLM) can be used to generate text description information based on texture prompt words. The generative model can generate a synthetic model with enhanced texture as a three-dimensional model to be used based on random noise, a three-dimensional model, and a text description information. Optionally, multiple three-dimensional models with textures can be generated as models to be used in the above manner. The synthetic model is a model that includes model shape information and texture information.

[0092] Traditional 6D pose estimation methods usually rely on instance-level training for specific objects or categories, which results in poor performance and limited generalization ability when encountering unseen objects or categories. When faced with new objects, it is usually necessary to train or fine-tune each new object separately, which increases the difficulty of deployment. In an embodiment, a plurality of three-dimensional models to be used are obtained by the above method, and then a pose estimation network is obtained by training the data obtained from the plurality of three-dimensional models to be used. This method can directly perform 6D pose estimation on new objects without the need for separate training or fine-tuning of new objects, thereby improving the generalization ability of the system.

[0093] Step S120: obtaining a plurality of sample data and corresponding labels based on the three-dimensional model to be used.

[0094] Step S130: training the pose estimation network according to the plurality of sample data and corresponding labels.

[0095] In this embodiment, a large amount of object model data can be generated in a simulation environment based on the obtained 3D model to be used. The data may include RGB images and depth images of different objects at multiple angles and postures, covering more than 1,000 categories of common daily necessities and industrial parts of different shapes and appearances. Based on the data, a training data set for the 6D pose estimation network can be obtained, which can effectively improve the generalization performance of the algorithm.

[0096] The training data set includes multiple sample data and corresponding labels. One sample data includes a sample image group corresponding to each of multiple first sample poses obtained according to a sample scene graph, and the label is used to indicate the real pose corresponding to the sample scene graph.

[0097] In the simulation environment, a sample scene graph and the real pose corresponding to the sample scene graph can be obtained at the same time. The sample scene graph includes a sample scene RGB graph and a sample depth graph, and a sample mask graph corresponding to the sample scene RGB graph can also be obtained; multiple first sample poses can be determined based on a to-be-used three-dimensional model, a sample mask graph, and a sample depth graph, and then a sample image group corresponding to each first sample pose can be determined. Afterwards, the pose estimation network is trained using the sample image groups and the real poses corresponding to each sample scene graph.

[0098] In the above training process, the first sample pose is iteratively updated using the pose optimization network, and a small adjustment is made each time based on the current estimate to gradually approach the corresponding real position, so that the RGBD image obtained based on the model to be used in the sample image (i.e., the first sample RGBD image after optimization and adjustment) is more closely matched with the second sample RGBD image obtained by cropping the sample scene image. The pose optimization network can use the following loss function:

[0099]

[0100] Where Δt is the adjustment of position in each iteration, ΔR is the adjustment of rotation in each iteration, The adjustment value from the current optimized pose to the true pose.

[0101] An estimation network can also be trained. In this way, the posture estimation network is trained.

[0102] The above-mentioned pose estimation method provided in this embodiment can achieve high-precision 6D pose estimation when facing objects that have never been seen before through a large-scale synthetic data training model, without the need for separate training or fine-tuning of new objects, which greatly improves the system's generalization ability when dealing with different objects and scenes. In the above method, by adopting a single-view rendering technology, that is, for a first pose, only a first RGBD image corresponding to the first pose is generated, the computational burden of each rendering can be reduced, and the efficiency of pose estimation can be improved. In addition, by using a gradually optimized neural network, the difficulty of adjusting the pose estimation in each iteration is reduced, so that the system can converge to an accurate pose estimate more quickly, further speeding up the entire reasoning process; the strategy of combining the above-mentioned single-view rendering and gradual optimization is adopted to ensure accuracy and improve efficiency. Due to the fast physical speed of the above method, it can be better applied to scenes with high real-time requirements such as disordered grasping of robots.

[0103] In order to perform the corresponding steps in the above embodiments and various possible methods, a method for implementing a 6D pose estimation device 200 is given below. Optionally, the 6D pose estimation device 200 can adopt the above Figure 1The device structure of the electronic device 100 is shown. Fig.11 , Fig.11 One of the block diagrams of the 6D pose estimation device 200 provided in the embodiment of the present application. It should be noted that the basic principle and technical effects of the 6D pose estimation device 200 provided in this embodiment are the same as those in the above-mentioned embodiment. For the sake of brief description, for parts not mentioned in this embodiment, reference can be made to the corresponding contents in the above-mentioned embodiments. In this embodiment, the 6D pose estimation device 200 may include: a scene information acquisition module 220, a determination module 230, a processing module 240 and an estimation module 250.

[0104] The scene information acquisition module 220 is used to obtain a scene graph and a mask graph corresponding to the scene graph, wherein the scene graph includes a target object, and the scene graph includes a scene RGB graph and a scene depth graph.

[0105] The determination module 230 is used to determine a plurality of first poses, wherein the first poses are 6D poses, and the depth-related information in the first poses is determined according to the mask image and the scene depth image.

[0106] The processing module 240 is used to obtain an image group corresponding to each first pose, wherein the image group includes a first RGBD image of a target three-dimensional model corresponding to the target object at a viewing angle corresponding to the first pose, and a second RGBD image obtained by cropping the scene image according to the first pose.

[0107] The estimation module 250 is used to input each image group into a trained pose estimation network to obtain a target 6D pose of the target object, wherein the pose estimation network obtains the target 6D pose based on the matching degree of images in the image group.

[0108] Please refer to Fig.12 , Fig.12 The second block diagram of the 6D pose estimation device 200 provided in the embodiment of the present application. In this embodiment, the 6D pose estimation device 200 may further include a training module 210.

[0109] The training module 210 is used to: obtain a three-dimensional model to be used; based on the three-dimensional model to be used, obtain multiple sample data and corresponding labels, wherein one sample data includes a sample image group corresponding to multiple first sample poses obtained according to a sample scene graph, and the label is used to indicate the real pose corresponding to the sample scene graph; based on the multiple sample data and the corresponding labels, the pose estimation network is trained.

[0110] Optionally, the above modules can be stored in the form of software or firmware. Figure 1 The memory 110 shown in the figure may be fixed in the operating system (OS) of the electronic device 100 and may be Figure 1 Meanwhile, the data and program codes required for executing the above modules may be stored in the memory 110.

[0111] An embodiment of the present application also provides a readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the 6D pose estimation method is implemented.

[0112] In summary, the embodiments of the present application provide a 6D pose estimation method, device, electronic device and readable storage medium, firstly obtain a scene graph including a target object and a mask graph corresponding to the scene graph, then determine multiple first poses in combination with the scene graph, and obtain image groups corresponding to each first pose, and finally input the above image groups into a trained pose estimation network to obtain a target 6D pose of the target object; the above scene graph includes a scene RGB graph and a scene depth graph, the first pose is a 6D pose, the depth-related information in the first pose is determined according to the mask graph and the scene depth graph, the image group includes a first RGBD graph of the target three-dimensional model corresponding to the target object at the perspective corresponding to the first pose, and a second RGBD graph obtained by cropping the scene graph according to the first pose, and the pose estimation network obtains the target 6D pose based on the matching degree of the images in the image group. In this way, the 6D pose of the object can be estimated quickly and accurately.

[0113] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0114] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0115] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0116] The above description is only an optional embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A 6D pose estimation method, characterized in that: The method comprises: Obtain a scene graph and obtain a mask graph corresponding to the scene graph, wherein the scene graph includes a target object, and the scene graph includes a scene RGB graph and a scene depth graph; Determine a plurality of first poses, wherein the first poses are 6D poses, and depth-related information in the first poses is determined based on the mask image and the scene depth map; Obtaining an image group corresponding to each first pose, wherein the image group includes a first RGBD image of a target three-dimensional model corresponding to the target object at a viewing angle corresponding to the first pose, and a second RGBD image obtained by cropping the scene image according to the first pose; Each image group is input into a trained pose estimation network to obtain a target 6D pose of the target object, wherein the pose estimation network obtains the target 6D pose based on the matching degree of images in the image group.

2. The method according to claim 1, characterized in that: The pose estimation network includes a pose optimization network, and the step of inputting each image group into a trained pose estimation network to obtain a target 6D pose of the target object includes: For each image group, the pose optimization network is used to optimize the first pose corresponding to the image group based on the image group to obtain a second pose, wherein the matching degree of the image group corresponding to the optimized pose is greater than or equal to the matching degree of the image group corresponding to the optimized pose before optimization, and the optimized image group includes the second RGBD image and the first RGBD image adjusted based on the optimized pose; The target 6D pose is obtained according to the second pose corresponding to each image group.

3. The method according to claim 2, characterized in that The step of obtaining the target 6D pose according to the second pose corresponding to each image group includes: For each second posture, according to the optimized image group under the second posture, a matching score corresponding to the second posture is calculated, wherein the optimized image group is obtained based on the image group corresponding to the first posture corresponding to the second posture; The target 6D posture is obtained according to each second posture and the matching score corresponding to each second posture.

4. The method according to claim 3, characterized in that The step of obtaining the target 6D posture according to each second posture and the matching score corresponding to each second posture includes: Sorting the determined multiple third postures according to the matching scores to obtain a sorting result, wherein the third posture is a posture selected from the obtained second postures; According to the ranking results, a global score of each third posture is calculated through a multi-head self-attention mechanism; The third pose corresponding to the highest global score among the obtained global scores is used as the target 6D pose.

5. The method according to claim 4, characterized in that The step of sorting the determined multiple third postures according to the matching scores to obtain a sorting result includes: According to the matching scores corresponding to the second postures, a second posture is selected from the obtained second postures as the third posture, wherein the matching score of the selected second posture is greater than or equal to the matching score of the unselected second postures; The determined third posture is sorted in ascending order or descending order according to the corresponding matching score to obtain the sorting result.

6. The method according to claim 1, characterized in that The determining of the plurality of first poses comprises: Obtaining two-dimensional images of the target three-dimensional model at different viewing angles; Obtaining a target depth value corresponding to a target object in the scene image according to the mask image and the scene depth image; For each two-dimensional image, a first pose corresponding to the two-dimensional image is obtained according to the planar pose of the target object indicated by the two-dimensional image and the target depth value, wherein the position of the second RGBD image in an image group in the scene graph is the same as the position of the target object in a two-dimensional image.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Obtaining a three-dimensional model to be used; Based on the three-dimensional model to be used, a plurality of sample data and corresponding labels are obtained, wherein one of the sample data includes a sample image group corresponding to each of the plurality of first sample poses obtained according to the sample scene graph, and the label is used to indicate a real pose corresponding to the sample scene graph; The pose estimation network is trained based on the multiple sample data and corresponding labels.

8. The method according to claim 7, characterized in that The step of obtaining the three-dimensional model to be used comprises: Obtaining a plurality of three-dimensional models from a three-dimensional model library, wherein the plurality of three-dimensional models correspond to different objects; At least one piece of texture description information is obtained, and a plurality of to-be-used three-dimensional models are generated according to the obtained texture description information and the three-dimensional model, wherein at least one of the to-be-used models is a three-dimensional model with a texture.

9. A 6D pose estimation device, characterized in that: The device comprises: A scene information acquisition module, used to obtain a scene graph and a mask graph corresponding to the scene graph, wherein the scene graph includes a target object, and the scene graph includes a scene RGB graph and a scene depth graph; A determination module, configured to determine a plurality of first poses, wherein the first poses are 6D poses, and depth-related information in the first poses is determined according to the mask image and the scene depth image; A processing module, used to obtain an image group corresponding to each first pose, wherein the image group includes a first RGBD image of a target three-dimensional model corresponding to the target object at a viewing angle corresponding to the first pose, and a second RGBD image obtained by cropping the scene image according to the first pose; The estimation module is used to input each image group into a trained pose estimation network to obtain a target 6D pose of the target object, wherein the pose estimation network obtains the target 6D pose of the target object based on the matching degree of the images in the image group.

10. An electronic device, characterized in that: It includes a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor can execute the machine executable instructions to implement the 6D pose estimation method described in any one of claims 1-8.

Images