Image pose processing method and device, readable storage medium and robot

Abstract

The application provides a kind of image pose processing method, device, readable storage medium and robot, wherein the image pose processing method includes: obtaining the spatial feature of target object in sampling image and the point cloud image of target object;Determine the template image matched with the point cloud image, based on the template image, correct the point cloud image, obtain the corrected image;Obtain the first feature and the second feature in the point cloud image, the first feature includes rotation feature, and the second feature includes size feature and displacement feature;According to the combination of template feature and first feature, the combination of second feature and spatial feature and the corrected image, obtain the 6D pose of target object, wherein the template feature is used to indicate the vector feature of template image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to an image pose processing method and device, a readable storage medium and a robot. BACKGROUND

[0002] At present, the category of object pose estimation is only applicable to known objects, and the detection performance for new objects is poor, and it is difficult to be used in scenes with frequent object updates. Due to the above defects of the pose estimation method, the generalization performance of the current pose estimation method is poor. SUMMARY

[0003] The present application aims to at least solve one of the problems in the prior art or related art.

[0004] To this end, the first aspect of the present application provides an image pose processing method.

[0005] The second aspect of the present application provides an image pose processing device.

[0006] The third aspect of the present application provides another image pose processing device.

[0007] The fourth aspect of the present application provides a readable storage medium.

[0008] The fifth aspect of the present application provides a robot.

[0009] Therefore, the first aspect of the present application provides an image pose processing method, comprising: acquiring a spatial feature of a target object in a sample image and a point cloud image of the target object; determining a template image matched with the point cloud image, correcting the point cloud image based on the template image to obtain a corrected image; acquiring a first feature and a second feature in the point cloud image, the first feature comprising a rotation feature, and the second feature comprising a size feature and a displacement feature; and obtaining a 6D pose of the target object according to a combination of a template feature and the first feature, a combination of the second feature and the spatial feature, and the corrected image, wherein the template feature is used to represent a vector feature of the template image.

[0010] The technical solution of the present application provides an image pose processing method, which can better capture the 6D pose of the target object. Specifically, in the case of obtaining a sample image, the target object in the sample image needs to be determined. The target object can be an object different from the background in the sample image, such as a book on the desktop, a fruit plate, etc.

[0011] In the case of acquiring the target object, the point cloud image of the target object is acquired, and a template image can be determined according to the shape of the point cloud image or according to the category of the point cloud image. The template image is a general image of the same kind of object. If the point cloud image is consistent with the template image, the point cloud image can be directly used. If the point cloud image is inconsistent with the template image, the point cloud image needs to be corrected so that the point cloud image can be close to the template image.

[0012] In the case of determining the target object, the point cloud image in the target object needs to be separated out, and then the first feature and the second feature of the point cloud object are determined. Since the first feature includes the rotation feature, the rotation angle of the target object relative to the template image can be determined by combining the rotation feature with the vector feature of the template image.

[0013] In the case of determining the target object, the spatial feature of the target object also needs to be acquired. The second feature includes the size feature and the displacement feature. Therefore, by combining the size feature, the displacement feature and the spatial feature, the size parameter and the coordinate parameter of the target object can be obtained.

[0014] On the basis of correcting the image, the 6D object pose of the target object can be obtained according to the rotation angle, the size parameter and the coordinate parameter of the target object.

[0015] The technical solution considers and utilizes the shape similarity of objects of the same category, designs a variable template field to represent the shape prior and the shape of the observed object, can reconstruct a three-dimensional model of an unobserved object of the same category, and improves the generalization performance of the pose estimation method. The implicit pose field can be obtained based on the training strategy of the template network, thereby significantly improving the accuracy of pose estimation and achieving high effect on unobserved objects of the same category. Since the shape of the unobserved object of the same category can be accurately reconstructed and the pose can be predicted, the performance of the robotic arm grasping is improved.

[0016] In addition, the image pose processing method provided in the application also has the following additional technical features.

[0017] In the above technical solution, based on the template image, the point cloud image is corrected to obtain a corrected image, including: based on the one-to-one correspondence between the points in the point cloud image and the points in the template image, determining the displacement offset between the corresponding points in the point cloud image and the template image; according to the displacement offset, correcting the point cloud image to obtain a corrected image to be confirmed; confirming the difference image of the template image and the image to be confirmed, removing the difference image to obtain the corrected image.

[0018] In the technical solution, in the case of determining the template image, the points in the point cloud image are corresponded with the points in the template image, the displacement offset of the part of points of the point cloud image compared with the template image can be determined according to the corresponding relationship between the points, the point cloud image can be preliminarily corrected according to the displacement offset, and the to-be-confirmed image is obtained. Then, the to-be-confirmed image and the template image are compared again to confirm whether there is a difference between the to-be-confirmed image and the template image. If there is no difference, the to-be-confirmed image can be directly used as the corrected image. If there is a difference between the to-be-confirmed image and the template image, the difference part, that is, the difference image, is confirmed, the difference image in the to-be-confirmed image is removed, and the corrected corrected image is obtained. Through the above manner, the accuracy of the image correction process is improved.

[0019] In any of the above technical solutions, the first spatial feature of the calibration RGB image and the second spatial feature of the calibration grayscale image are obtained, the calibration grayscale image is a grayscale image of the calibration RGB image; the fusion spatial feature is obtained by combining the first spatial feature and the second spatial feature; the vector feature of the key point in the fusion spatial feature is determined; and the first target network is trained according to the calibration RGB image and the vector feature of the key point; wherein the sample image is input into the first target network to obtain the spatial feature.

[0020] In the technical solution, in order to accurately obtain the spatial feature of the target object, the first target network needs to be trained first. Specifically, a calibration RGB image and a calibration grayscale image are obtained first, and the calibration grayscale image is a grayscale image of the calibration RGB image. In the calibration process, the first spatial feature of the calibration RGB image and the second spatial feature of the grayscale image are obtained respectively, and the first spatial feature and the second spatial feature are combined to obtain the fused fusion spatial feature. This step can compensate for the calibration RGB image and the calibration grayscale image.

[0021] In the case of obtaining the fusion spatial feature, the vector feature of the key point can be obtained from the fusion spatial feature, and the key point is used to reflect the key position of the calibration image. Then, the calibration RGB image and the vector feature of the key point are input into the first target network for training, the calibration RGB image is input, and the vector feature of the key point is output, so that the trained first target network can accurately extract the vector feature of the key point in the RGB image in the case of inputting the RGB image. After the training of the first target network is completed, the sample image can be input into the first target network, and the spatial feature of the key point in the sample image can be automatically obtained. Through the training of the first target network in the above manner, the accuracy of obtaining the spatial feature is improved, and the accuracy of obtaining the 6D object pose of the target object is improved.

[0022] In any of the above technical solutions, before obtaining the spatial feature of the target object in the sampling image, further comprising: obtaining a first calibration point cloud of the calibration image; rotating the first calibration point cloud by a calibration angle to obtain a second calibration point cloud; training a second target network according to the first calibration point cloud, the second calibration point cloud and the calibration angle; wherein the sampling image is input into the second target network to obtain the first feature.

[0023] In this technical solution, in order to accurately obtain the first feature of the sampling image, the second target network needs to be trained first. Specifically, the calibration image is first obtained, and then the first calibration point cloud of the calibration image is obtained. Then, the first calibration point cloud is rotated to obtain the second calibration point cloud after rotation. It should be noted that the first calibration point cloud and the second calibration point cloud are the same point cloud, only the point cloud at different angles before and after rotation. When the first calibration point cloud is rotated, the rotation angle needs to be recorded each time, so that the image of the calibration point cloud before rotation, the rotation angle of the calibration point cloud and the image of the calibration point cloud after rotation can be obtained.

[0024] The first calibration point cloud, the second calibration point cloud and the calibration angle are input into the second target network for training. The first calibration point cloud and the second calibration point cloud are input, and the calibration angle is output, so that the trained second target network can accurately obtain the relative rotation angle of the two point clouds when the input point clouds have different rotation angles. After the training of the second target network is completed, the sampling image can be input into the second target network, and the rotation angle of the sampling image relative to the template image, i.e. the first feature, can be automatically obtained. Through the above method for training the second target network, the accuracy of obtaining the first feature is improved, thereby improving the accuracy of obtaining the 6D object pose of the target object.

[0025] In any of the above technical solutions, before obtaining the spatial feature of the target object in the sampling image, further comprising: capturing the calibration image by a camera; obtaining a first size feature and a first displacement feature of a third calibration point cloud of the calibration image; capturing the calibration image after changing the shooting position of the camera to obtain a fourth calibration point cloud; obtaining a second size feature and a second displacement feature of the fourth calibration point cloud according to the intrinsic parameters of the camera; calculating a difference size feature of the second size feature and the first size feature, and a difference displacement feature of the second displacement feature and the first displacement feature; training a third target network according to the third calibration point cloud, the fourth calibration point cloud, the difference size feature and the difference displacement feature; wherein the sampling image is input into the third target network to obtain the second feature.

[0026] In the technical solution, in order to accurately obtain the spatial features of the target object, the third target network needs to be trained first. Specifically, first, a calibration image is captured by a camera, and then a third calibration point cloud of the calibration image is obtained. At this time, the user can actively label the first size feature and the first displacement feature of the third calibration point cloud. Then, the shooting position of the camera is changed. After the position is changed, the fourth calibration point cloud can be captured by the camera. At this time, the moving position of the camera can be obtained according to the intrinsic parameters of the camera, so that the second size feature and the second displacement feature of the fourth calibration point cloud can be indirectly calculated according to the moving position of the camera. Therefore, it is not necessary to label the size feature and the displacement feature of the point cloud every time the shooting position is changed, which is beneficial to improve the training efficiency.

[0027] It should be noted that the third calibration point cloud and the fourth calibration point cloud are the same point cloud, only the shooting position is changed.

[0028] The third calibration point cloud, the fourth calibration point cloud, the difference size feature and the difference displacement feature are input into the third target network for training. The third calibration point cloud and the fourth calibration point cloud are input, and the difference size feature and the difference displacement feature are output, so that the third target network after training can accurately obtain the size difference and the displacement difference of the two point clouds in the case of inputting point clouds with different sizes or positions. After the training of the third target network is completed, the sample image can be input into the third target network, and the size difference and the displacement difference of the sample image compared with the template image, i.e. the second feature, can be automatically obtained. Through the above method for training the third target network, it is beneficial to improve the accuracy of obtaining the third feature, thereby improving the accuracy of obtaining the 6D pose of the target object.

[0029] In any of the above technical solutions, after obtaining the first feature and the second feature in the sample image, the method further comprises: determining a size residual and a displacement residual according to the second feature and the spatial feature; and obtaining the 6D pose of the target object according to the combination of the template feature and the first feature, the combination of the second feature and the spatial feature, and the modified image, comprising: obtaining the 6D pose of the target object according to the combination of the template and the first feature, the combination of the second feature, the spatial feature, the size residual and the displacement residual, and the modified image.

[0030] In the technical solution, when determining the 6D pose of the target object, if the size or position difference of the point cloud of the target object and the template image is large, parameter compensation is needed. By involving the size residual and the displacement residual in the calculation, the 6D pose of the target object can be more accurately obtained.

[0031] The second aspect of the present application provides a processing device of an image pose, comprising: an acquisition module, acquiring a spatial feature of a target object in a sample image and a point cloud image of the target object; a determination module, determining a template image matched with the point cloud image, correcting the point cloud image based on the template image to obtain a corrected image; the acquisition module is further used for: acquiring a first feature and a second feature in the point cloud image, the first feature comprising a rotation feature, and the second feature comprising a size feature and a displacement feature; a processing module, obtaining a 6D pose of the target object according to the combination of a template feature and the first feature, the combination of the second feature and the spatial feature, and the corrected image, wherein the template feature is used to represent a vector feature of the template image.

[0032] The technical scheme of the present application provides a processing device of an image pose, which can better capture the 6D pose of the target object. Specifically, in the case of obtaining a sample image, the target object in the sample image needs to be determined, which can be an object different from the background in the sample image, such as a book on a table or a fruit plate.

[0033] In the case of acquiring the target object, the point cloud image of the target object is acquired, and the template image can be determined according to the shape of the point cloud image or the type of the point cloud image. The template image is a general image of the same type of object. If the point cloud image is consistent with the template image, the point cloud image can be directly used. If the point cloud image is inconsistent with the template image, the point cloud image needs to be corrected so that the point cloud image can be close to the template image.

[0034] In the case of determining the target object, the point cloud image in the target object needs to be separated out, and then the first feature and the second feature of the point cloud object are determined. Since the first feature includes the rotation feature, the rotation angle of the target object relative to the template image can be determined by combining the rotation feature with the vector feature of the template image.

[0035] In the case of determining the target object, the spatial feature of the target object also needs to be acquired. The second feature includes the size feature and the displacement feature, so the size parameter and the coordinate parameter of the target object can be obtained by combining the size feature, the displacement feature and the spatial feature.

[0036] Based on the corrected image, the 6D pose of the target object can be obtained according to the rotation angle, the size parameter and the coordinate parameter of the target object.

[0037] The technical solution considers and utilizes the shape similarity of objects in the same category, designs a variable template field to represent the shape prior and the shape of the observed object, can reconstruct the three-dimensional model of the same object that has never been seen, and improve the generalization performance of the pose estimation method. The implicit pose field can be obtained based on the training strategy of the template network, thereby significantly improving the accuracy of pose estimation, and achieving high effect on the same object that has never been seen. Since the shape of the same object that has never been seen can be accurately reconstructed and the pose can be predicted, the performance of the robotic arm grasping is improved.

[0038] In addition, the image pose processing device provided in the present application also has the following additional technical features.

[0039] In the above technical solution, the correction module is specifically configured to: based on the one-to-one correspondence between the points in the point cloud image and the points in the template image, determine the displacement offset between the corresponding points in the point cloud image and the template image; according to the displacement offset, correct the point cloud image to obtain a corrected to-be-confirmed image; confirm the difference image between the template image and the to-be-confirmed image, remove the difference image, and obtain a corrected image.

[0040] In the technical solution, in the case of determining the template image, the points in the point cloud image are corresponded with the points in the template image, the displacement offset of part of the points in the point cloud image compared with the template image can be determined according to the correspondence between the points, and the point cloud image can be preliminarily corrected according to the displacement offset to obtain a to-be-confirmed image. Then, the to-be-confirmed image is compared with the template image again to confirm whether there is a difference between the to-be-confirmed image and the template image. If there is no difference, the to-be-confirmed image can be directly used as a corrected image. If there is a difference between the to-be-confirmed image and the template image, the difference part, i.e., the difference image, is confirmed, the difference image in the to-be-confirmed image is removed, and a corrected corrected image is obtained. Through the above method, the accuracy of the image correction process is improved.

[0041] In any of the above technical solutions, the image pose processing device further comprises a training module, before acquiring the spatial features of the target object in the sample image, the training module is configured to: acquire first spatial features of a calibration RGB image and second spatial features of a calibration grayscale image, the calibration grayscale image being a grayscale image of the calibration RGB image; obtain fused spatial features by combining the first spatial features and the second spatial features; determine vector features of key points in the fused spatial features; train a first target network according to the calibration RGB image and the vector features of the key points; wherein the sample image is input into the first target network to obtain the spatial features.

[0042] In the technical solution, in order to accurately obtain the spatial feature of the target object, the first target network needs to be trained. Specifically, a calibration RGB image and a calibration grayscale image are first obtained, and the calibration grayscale image is a grayscale image of the calibration RGB image. In the calibration process, the first spatial feature of the calibration RGB image and the second spatial feature of the grayscale image are obtained respectively, and the first spatial feature and the second spatial feature are combined to obtain the fused spatial feature. This step can compensate for the calibration RGB image and the calibration grayscale image.

[0043] In the case of obtaining the fused spatial feature, the vector feature of the key point can be obtained from the fused spatial feature, and the key point is used to represent the key position of the calibration image. Then the calibration RGB image and the vector feature of the key point are input into the first target network for training, the calibration RGB image is input as the input, and the vector feature of the key point is output as the output, so that the trained first target network can accurately extract the vector feature of the key point in the RGB image in the case of inputting the RGB image. After the training of the first target network is completed, the sample image can be input into the first target network, and the spatial feature of the key point in the sample image can be automatically obtained. Training the first target network in the above manner is beneficial to improving the accuracy of obtaining the spatial feature, thereby improving the accuracy of obtaining the 6D object pose of the target object.

[0044] In any of the above technical solutions, the image pose processing apparatus further comprises a training module, before obtaining the spatial feature of the target object in the sample image, the training module is configured to: obtain a first calibration point cloud of a calibration image; rotate the first calibration point cloud by a rotation calibration angle to obtain a second calibration point cloud; train a second target network according to the first calibration point cloud, the second calibration point cloud and the rotation calibration angle; wherein the sample image is input into the second target network to obtain the first feature.

[0045] In the technical solution, in order to accurately obtain the first feature of the sample image, the second target network needs to be trained. Specifically, a calibration image is first obtained, and then a first calibration point cloud of the calibration image is obtained. Then the first calibration point cloud is rotated to obtain a second calibration point cloud after rotation. It should be noted that the first calibration point cloud and the second calibration point cloud are the same point cloud, only different angles of point cloud before and after rotation. When the first calibration point cloud is rotated, the rotation angle needs to be recorded each time, so that the image of the calibration point cloud before rotation, the rotation angle of the calibration point cloud and the image of the calibration point cloud after rotation can be obtained.

[0046] The first calibration point cloud, the second calibration point cloud and the rotation calibration angle are input into the second target network for training, the first calibration point cloud and the second calibration point cloud are taken as input, and the rotation calibration angle is taken as output, so that the second target network after training can accurately obtain the relative rotation angle of the two point clouds in the case that the input point clouds have different rotation angles. After the training of the second target network is completed, the sampling image can be input into the second target network, and the rotation angle of the sampling image compared with the template image, that is, the first feature, can be automatically obtained. Through the above method, the training of the second target network is beneficial to improve the accuracy of obtaining the first feature, thereby improving the accuracy of obtaining the 6D object pose of the target object.

[0047] In any of the above technical solutions, the image pose processing apparatus further comprises a training module, before the spatial feature of the target object in the sampling image is obtained, the training module is configured to: capture a calibration image by using the camera; obtain a first size feature and a first displacement feature of a third calibration point cloud of the calibration image; capture the calibration image after changing the shooting position of the camera to obtain a fourth calibration point cloud; obtain a second size feature and a second displacement feature of the fourth calibration point cloud according to the intrinsic parameters of the camera; calculate a difference size feature of the second size feature and the first size feature, and a difference displacement feature of the second displacement feature and the first displacement feature; train a third target network according to the third calibration point cloud, the fourth calibration point cloud, the difference size feature and the difference displacement feature; and input the sampling image into the third target network to obtain the second feature.

[0048] In this technical solution, in order to accurately obtain the spatial feature of the target object, the third target network needs to be trained first. Specifically, first, a calibration image is captured by using the camera, and then a third calibration point cloud of the calibration image is obtained. At this time, the first size feature and the first displacement feature of the third calibration point cloud can be actively labeled by the user. Then, the shooting position of the camera is changed, and the fourth calibration point cloud can be captured by the camera after the position is changed. At this time, the moving position of the camera can be obtained according to the intrinsic parameters of the camera, so that the second size feature and the second displacement feature of the fourth calibration point cloud can be indirectly calculated according to the moving position of the camera. Therefore, it is not necessary to label the size feature and the displacement feature of the point cloud every time the shooting position is changed, which is beneficial to improve the training efficiency.

[0049] It should be noted that the third calibration point cloud and the fourth calibration point cloud are the same point cloud, only the shooting position of the point cloud is changed.

[0050] The third calibration point cloud, the fourth calibration point cloud, the difference size feature and the difference displacement feature are input into the third target network for training, the third calibration point cloud and the fourth calibration point cloud are taken as input, and the difference size feature and the difference displacement feature are taken as output, so that the third target network after training can accurately obtain the size difference and the displacement difference of two point clouds in the case of inputting point clouds with different sizes or positions. After the training of the third target network is completed, the sampling image can be input into the third target network, and the size difference and the displacement difference of the sampling image compared with the template image, i.e., the second feature, can be automatically obtained. Through the training of the third target network in the above manner, the accuracy of obtaining the third feature is improved, thereby facilitating the improvement of the accuracy of obtaining the 6D object pose of the target object.

[0051] In any of the above technical solutions, after the first feature and the second feature in the sampling image are obtained, the acquisition module is further configured to determine a size residual and a displacement residual according to the second feature and the spatial feature; and the processing module is specifically configured to obtain the 6D pose of the target object according to the combination of the template and the first feature, the combination of the second feature, the spatial feature, the size residual and the displacement residual, and the modified image.

[0052] In this technical solution, when the 6D pose of the target object is determined, if the size or position difference between the point cloud of the target object and the template image is large, parameter compensation is needed, and by participating in the calculation through the size residual and the displacement residual, the 6D pose of the target object can be more accurately obtained.

[0053] The third aspect of the present application provides an image pose processing device, comprising a processor and a memory, the memory stores programs or instructions executable on the processor, and the programs or instructions are executed by the processor to realize the steps of the method of any one of the first aspect.

[0054] The fourth aspect of the present application provides a readable storage medium, the readable storage medium stores programs or instructions, and the programs or instructions are executed by the processor to realize the steps of the method of any one of the first aspect.

[0055] The fifth aspect of the present application provides a robot, comprising: the image pose processing device of any one of the second aspect and the third aspect; and / or the readable storage medium of the fourth aspect.

[0056] Additional aspects and advantages of the present application will be partially given in the following description, partially will become obvious from the following description, or will be understood by the practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0057] The above and / or additional aspects and advantages of the present application will become apparent and more readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings, in which:

[0058] Figure 1 A flowchart illustrating the image pose processing method in an embodiment of the present invention is shown.

[0059] Figure 2 A schematic block diagram of an image pose processing apparatus according to an embodiment of the present invention is shown;

[0060] Figure 3 The second schematic block diagram of the image pose processing device in an embodiment of the present invention is shown. Detailed Implementation

[0061] To better understand the above aspects, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0062] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0063] like Figure 1 As shown, in some embodiments of the present invention, an image pose processing method is provided, including:

[0064] Step 102: Obtain the spatial features of the target object and the point cloud image of the target object in the sampled image;

[0065] Step 104: Determine a template image that matches the point cloud image; based on the template image, correct the point cloud image to obtain the corrected image.

[0066] Step 106: Obtain the first feature and the second feature in the point cloud image. The first feature includes rotation features, and the second feature includes size features and displacement features.

[0067] Step 108: Based on the combination of template features and first features, the combination of second features and spatial features, and the corrected image, the 6D pose of the target object is obtained.

[0068] Template features are used to represent the vector features of the template image.

[0069] This embodiment proposes an image pose processing method. Using this method, the 6D pose of the target object can be captured better. Specifically, when a sampled image is obtained, it is necessary to determine the target object in the sampled image. The target object can be an object in the sampled image that is different from the background, such as a book or fruit plate on a table.

[0070] In the case of acquiring the target object, the point cloud image of the target object is acquired, and a template image can be determined according to the shape of the point cloud image or according to the category of the point cloud image. The template image is a general image of the same kind of object. If the point cloud image is consistent with the template image, the point cloud image can be directly used. If the point cloud image is inconsistent with the template image, the point cloud image needs to be corrected so that the point cloud image can be close to the template image.

[0071] In the case of determining the target object, the point cloud image in the target object needs to be separated out, and then the first feature and the second feature of the point cloud object are determined. Since the first feature includes the rotation feature, the rotation feature and the vector feature of the template image are combined to determine the rotation angle of the target object relative to the template image.

[0072] In the case of determining the target object, the spatial feature of the target object also needs to be acquired. The second feature includes the size feature and the displacement feature. Therefore, the size feature, the displacement feature and the spatial feature are combined to obtain the size parameter and the coordinate parameter of the target object.

[0073] On the basis of correcting the image, the 6D object pose of the target object can be obtained according to the rotation angle, the size parameter and the coordinate parameter of the target object.

[0074] The embodiment considers and utilizes the shape similarity of the same kind of object, designs a variable template field to represent the shape prior and the shape relationship of the observed object, can reconstruct the three-dimensional model of the same kind of object that has never been seen before, and improves the generalization performance of the pose estimation method. The implicit pose field can be obtained based on the training strategy of the template network, thereby significantly improving the accuracy of pose estimation, and having higher effect on the same kind of object that has never been seen before. Since the shape of the same kind of object that has never been seen before can be accurately reconstructed and the pose can be predicted, the performance of the robotic arm grasping is improved.

[0075] In the above embodiment, based on the template image, the point cloud image is corrected to obtain a corrected image, including: based on the one-to-one correspondence between the points in the point cloud image and the points in the template image, determining the displacement offset between the corresponding points in the point cloud image and the template image; according to the displacement offset, correcting the point cloud image to obtain a confirmed image; confirming the difference image of the template image and the confirmed image, removing the difference image to obtain the corrected image.

[0076] In this embodiment, in the case of determining the template image, the points in the point cloud image are corresponded with the points in the template image, the displacement offset of the part points of the point cloud image compared with the template image can be determined according to the correspondence relationship between the points and the points, the point cloud image can be preliminarily corrected according to the displacement offset, and the to-be-confirmed image is obtained. Then, the to-be-confirmed image and the template image are compared again, whether there is a difference between the to-be-confirmed image and the template image is confirmed, if there is no difference, the to-be-confirmed image can be directly used as the corrected image, if there is a difference between the to-be-confirmed image and the template image, the difference part, that is, the difference image, is confirmed, the difference image in the to-be-confirmed image is removed, and the corrected corrected image is obtained. Through the above-mentioned manner, the accuracy of the image correction process is improved.

[0077] In one possible application, we provide a deformable template field (Deformable Template Field), we introduce a deformable template field to establish a point-to-point spatial continuous change between a prior template and an observed object. The deformable template field is a decoder composed of a deformation network and a template network.

[0078] I. Template network;

[0079] (1) We map an implicit encoding z i and a 3D point p to a scalar s.

[0080] T(p,z i )=s:p∈R 3 ,s∈R.

[0081] Where s is the distance between point p and the surface of the object. T(.)=0 implicitly represents the shape surface of the object, R represents a scalar, and R 3 represents a three-dimensional array.

[0082] (2) Given a set of available object models Where represents a three-dimensional point cloud model of instance j in category i, and the implicit encoding of each model can be obtained by back propagation of the above formula. Then we can calculate the implicit template encoding of each category;

[0083]

[0084] (3) Through the constraint of the shared weight of different objects and the category implicit template encoding, the template network can represent the common SDF (Signed Distance Field, representing the distance from the point to the object surface, where the inside of the surface is negative, the outside of the surface is positive, and the surface is 0) of different categories.

[0085] II. Deformation network

[0086] (1) We introduce this module to obtain the point-to-point continuous shape space between the observed object and the prior template.

[0087] D(p, v j ) = (o, As): p e R 3 , e R, D(p, v j ) represents the one-to-one correspondence of the implemented points to the feature space in the general template.

[0088] where o is the displacement offset of a point, and As is the distance correction value of the object in the same category. The deformable template field obtains the change offset of the instance using the deformation network, and then predicts the SDF value of the template through the template network and the category-level template implicit encoding.

[0089] (2) In the training stage, all object models should be converted to the canonical coordinates:

[0090] p c = T (p-t) / .

[0091] where R and t are the true values of rotation and translation, L is the diagonal length, and T is the identity matrix transpose.

[0092] (3) We first train the template network to obtain the implicit template encoding of each category, and then train the entire deformable template field to establish a continuous shape change between the specific object and the template, and the loss function can be expressed as:

[0093]

[0094] where is the true value of the SDF value, and Omega is the three-dimensional space.

[0095] (4) We use v as the true value of the deformation feature to optimize the predicted feature map

[0096] In any of the above embodiments, the first spatial feature of the calibrated RGB image and the second spatial feature of the calibrated grayscale image are obtained, and the calibrated grayscale image is a grayscale image of the calibrated RGB image; the fusion spatial feature is obtained by combining the first spatial feature and the second spatial feature; the vector feature of the key point in the fusion spatial feature is determined; the first target network is trained according to the calibrated RGB image and the vector feature of the key point; wherein the sample image is input into the first target network to obtain the spatial feature.

[0097] In this embodiment, in order to accurately obtain the spatial features of the target object, it is necessary to first train the first target network. Specifically, a calibration RGB image and a calibration grayscale image are first obtained, and the calibration grayscale image is the grayscale image of the calibration RGB image. In the calibration process, the first spatial feature of the calibration RGB image and the second spatial feature of the grayscale image are obtained respectively, and the first spatial feature and the second spatial feature are combined to obtain the fused spatial feature. This step can compensate for the calibration RGB image and the calibration grayscale image.

[0098] In the case of obtaining the fused spatial feature, the vector feature of the key point can be obtained from the fused spatial feature, and the key point is used to reflect the key position of the calibration image. Then the calibration RGB image and the vector feature of the key point are input into the first target network for training, the calibration RGB image is input, and the vector feature of the key point is output, so that the trained first target network can accurately extract the vector feature of the key point in the RGB image under the condition of inputting the RGB image. After the first target network is trained, the sampling image can be input into the first target network, and the spatial feature of the key point in the sampling image can be automatically obtained. Training the first target network in the above manner is conducive to improving the accuracy of obtaining the spatial feature, thereby improving the accuracy of obtaining the 6D object pose of the target object.

[0099] We provide an object information extraction module to obtain the feature key points and information of objects in the whole scene, and use it as the input of the subsequent module for end-to-end learning estimation. It includes a key point detection network and an instance segmentation network.

[0100] We construct the encoder and decoder as the key point detection network.

[0101] (1) Let I ∈ R W×H×3 and D ∈ R W×H×1 represent an RGB image and an aligned depth image, where W and H are the width and length of the image respectively, and a set of RGB data I and depth data D are given as input.

[0102] (2) We use ResNet (Residual Neural Network) to generate low-resolution spatial features where R = 8 and C = 128 are the downsampling ratio and the number of channels. In order to more effectively fuse the RGB data and the depth data from two different distributions, we introduce a W-AdaIN (Adaptive Fusion) structure in the last layer of the encoder to obtain a new set of features

[0103] (3) We use channel connection to fuse and to obtain a new feature Then we get F * into the decoder (up-convolution residual network) to obtain the keypoint heat map and the geometry feature map where N is the number of classes, The heat map is generated by spreading the keypoint of the object through a Gaussian kernel.

[0104] (4) We train the keypoint detection network with the focal loss loss function L k At inference time, we select the 64 peak points of F as the center points of the object We can get the vector feature of the object keypoint from F

[0105] In one possible application, the point cloud image of the target object can be obtained through the instance segmentation network. In the instance segmentation network, we get the feature F * through the FPN (Feature Pyramid Networks) structure, which is a feature pyramid structure with a computational scale of resolution from 1 / 8 to 1 / 2. The network is learned by training the instance-level segmentation loss L seg The object feature extraction module is independently trained by L k and L seg .

[0106] In any of the above embodiments, before obtaining the spatial feature of the target object in the sampling image, further comprising: obtaining a first calibration point cloud of a calibration image; rotating the first calibration point cloud by a calibration angle to obtain a second calibration point cloud; training a second target network according to the first calibration point cloud, the second calibration point cloud and the rotation calibration angle; wherein the sampling image is input into the second target network to obtain the first feature.

[0107] In this embodiment, in order to accurately obtain the first feature of the sampling image, the second target network needs to be trained first. Specifically, first, a calibration image is obtained, then a first calibration point cloud of the calibration image is obtained, then the first calibration point cloud is rotated to obtain a second calibration point cloud after rotation. It should be noted that the first calibration point cloud and the second calibration point cloud are the same point cloud, only different angles of the point cloud before and after rotation. When rotating the first calibration point cloud, the rotation angle needs to be recorded each time, so that the image of the calibration point cloud before rotation, the rotation angle of the calibration point cloud and the image of the calibration point cloud after rotation can be obtained.

[0108] ​The first calibration point cloud, the second calibration point cloud and the rotation calibration angle are input into the second target network for training, the first calibration point cloud and the second calibration point cloud are taken as inputs, and the rotation calibration angle is taken as output, so that the trained second target network can accurately obtain the relative rotation angle of the two point clouds in the case that the input point clouds have different rotation angles. After the training of the second target network is completed, the sampling image can be input into the second target network, and the rotation angle of the sampling image compared with the template image, i.e., the first feature, can be automatically obtained. Through the above method for training the second target network, the accuracy of obtaining the first feature is improved, thereby improving the accuracy of obtaining the 6D object pose of the target object.

[0109] In one possible application, we provide an implicit pose field, in particular:

[0110] (1) The observed point cloud is represented as a spatial rotation embedding f by an encoder r . Then it is concatenated with the object implicit template encoding g obtained from the deformable template field.

[0111] (2) In the training phase, two point clouds of 1024 points are sampled on the template and the rotated template respectively, obtaining P t and P t-r . P t-r is converted to a canonical coordinate view by 6D pose annotation, and finally the loss function between the template point cloud P t and the converted point cloud P t-r is calculated using the chamfer distance:

[0112]

[0113] Where x, y are the coordinates of the points on the template point cloud and the transformed point cloud respectively.

[0114] In any of the above embodiments, before obtaining the spatial feature of the target object in the sampling image, the method further comprises: capturing a calibration image by a camera; obtaining a first size feature and a first displacement feature of a third calibration point cloud of the calibration image; capturing the calibration image after changing the shooting position of the camera to obtain a fourth calibration point cloud; obtaining a second size feature and a second displacement feature of the fourth calibration point cloud according to the intrinsic parameters of the camera; calculating a difference size feature of the second size feature and the first size feature, and a difference displacement feature of the second displacement feature and the first displacement feature; training a third target network according to the third calibration point cloud, the fourth calibration point cloud, the difference size feature and the difference displacement feature; wherein the sampling image is input into the third target network to obtain the second feature.

[0115] In this embodiment, in order to accurately obtain the spatial features of the target object, it is necessary to first train the third target network. Specifically, first, a calibration image is captured by a camera, and then a third calibration point cloud of the calibration image is obtained. At this time, the user can actively label the first size feature and the first displacement feature of the third calibration point cloud. Then, the shooting position of the camera is changed. After the position is changed, the fourth calibration point cloud can be captured by the camera. At this time, the moving position of the camera can be obtained according to the intrinsic parameters of the camera, so that the second size feature and the second displacement feature of the fourth calibration point cloud can be indirectly calculated according to the moving position of the camera. Therefore, it is not necessary to label the size feature and the displacement feature of the point cloud every time the shooting position is changed, which is beneficial to improve the training efficiency.

[0116] It should be noted that the third calibration point cloud and the fourth calibration point cloud are the same point cloud, only the shooting position is changed.

[0117] The third calibration point cloud, the fourth calibration point cloud, the difference size feature and the difference displacement feature are input into the third target network for training. The third calibration point cloud and the fourth calibration point cloud are input, and the difference size feature and the difference displacement feature are output, so that the trained third target network can accurately obtain the size difference and the displacement difference of the two point clouds in the case of inputting point clouds with different sizes or positions. After the training of the third target network is completed, the sampling image can be input into the third target network, and the size difference and the displacement difference of the sampling image compared with the template image, i.e. the second feature, can be automatically obtained. Through the above method for training the third target network, it is beneficial to improve the accuracy of obtaining the third feature, thereby improving the accuracy of obtaining the 6D object pose of the target object.

[0118] In one possible application, (1) we designed a sampling method based on view transformation to simulate the possible situation in the real scene. We placed the object horizontally on the table, and uniformly sampled N camera viewpoints from a hemisphere with a radius of r x U (0, 0.2) Where U is a uniform random sampling.

[0119] (2) PyTorch3D is used to render part of the point cloud of the object. We only need a 6D pose of a view, and other scenes can be calculated by camera point RT transformation.

[0120]

[0121] Where P i is the 6D pose of the object at view i, C i is the camera pose of scene i, C0 is the initial camera pose, and P0 is the initial 6D pose.

[0122] In any of the above embodiments, after obtaining the first feature and the second feature in the sample image, the method further comprises: determining a size residual and a displacement residual according to the second feature and the spatial feature; and obtaining the 6D pose of the target object according to the combination of the template feature and the first feature, the combination of the second feature and the spatial feature, and the corrected image, comprising: obtaining the 6D pose of the target object according to the combination of the template and the first feature, the combination of the second feature, the spatial feature, the size residual and the displacement residual, and the corrected image.

[0123] In this embodiment, when determining the 6D pose of the target object, if the point cloud of the target object has a large difference in size or position from the template image, parameter compensation is needed, and by involving the size residual and the displacement residual in the calculation, the 6D pose of the target object can be obtained more accurately.

[0124] In a possible application, we provide a pose regression module, and the specific application is:

[0125] (1) We propose an implicit template encoding g and a keypoint variation feature to help predict the 6D pose.

[0126] (2) We use a segmentation mask to separate the instance object from the depth map, and then use the camera intrinsic to convert the instance object into a point cloud Here, k is the number of instances in the current scene.

[0127] (3) After the encoder of PointNet++, the rotation feature f is obtained r and the spatial rotation feature f of size and translation ts . We input f r and the template feature g to the pose prediction head and predict a 9-dimensional vector, which represents the 6D pose of the object 3x3 rotation

[0128] (4) We estimate the size residual s res and the displacement residual t res by a size and displacement prediction head with f ts and the deformation feature . The final prediction of size and displacement is s=s res +M{s c} and where M{s c} is a category prior average size. is the average observed point cloud of the object.

[0129] In a possible application, in the embodiment of the present application, the overall loss function L of the end-to-end training is defined as:

[0130]

[0131] wherein λ1, λ2, λ3, λ4 and λ5 are weight coefficients;

[0132] to predict the loss between the 6D object and the ground truth.

[0133] As shown in the embodiments of the present application, a processing device 200 for image pose is provided, comprising: Figure 2 An acquisition module 210 is configured to acquire spatial features of a target object in a sample image and a point cloud image of the target object.

[0134] A determination module 220 is configured to determine a template image matched with the point cloud image, correct the point cloud image based on the template image, and obtain a corrected image.

[0135] The acquisition module 210 is further configured to acquire first features and second features in the point cloud image, wherein the first features include rotation features, and the second features include size features and displacement features.

[0136] A processing module 230 is configured to obtain a 6D pose of the target object based on the combination of template features and the first features, the combination of the second features and the spatial features, and the corrected image, wherein the template features are used to represent vector features of the template image.

[0137] The embodiments of the present application provide a processing device for image pose, which can better capture the 6D pose of a target object. Specifically, in the case of obtaining a sample image, the target object in the sample image needs to be determined. The target object can be an object different from the background in the sample image, such as a book on a table or a fruit plate.

[0138] In the case of obtaining the target object, a point cloud image of the target object is acquired. The template image can be determined according to the shape of the point cloud image or according to the type of the point cloud image. The template image is a general image of the same type of object. If the point cloud image is consistent with the template image, the point cloud image can be directly used. If the point cloud image is inconsistent with the template image, the point cloud image needs to be corrected so that the point cloud image can be close to the template image.

[0139] In the case of determining the target object, the point cloud image in the target object needs to be separated out, and then the first features and the second features of the point cloud object are determined. Since the first features include rotation features, the rotation angle of the target object relative to the template image can be determined by combining the rotation features with the vector features of the template image.

[0140]

[0141] ​In the case of determining the target object, the spatial feature of the target object also needs to be obtained, the second feature includes the size feature and the displacement feature, therefore, the size feature and the displacement feature are combined with the spatial feature, so that the size parameter and the coordinate parameter of the target object can be obtained.

[0142] On the basis of the modified image, the 6D object pose of the target object can be obtained according to the rotation angle, the size parameter and the coordinate parameter of the target object.

[0143] The embodiment considers and utilizes the shape similarity of the same category of objects, designs a variability template field to represent the shape prior and the shape of the observed object, can reconstruct the three-dimensional model of the same category of objects that have never been seen, and improve the generalization performance of the pose estimation method. The implicit pose field can be obtained based on the training strategy of the template network, thereby significantly improving the accuracy of pose estimation, and having higher effect on the same category of objects that have never been seen. Since the shape of the same category of objects that have never been seen can be accurately reconstructed and the pose can be predicted, the performance of the robotic arm grasping is improved.

[0144] In addition, the image pose processing device provided in the present application also has the following additional technical features.

[0145] In the above embodiment, the modification module is specifically configured to: determine a displacement offset between corresponding points in the point cloud image and the template image based on one-to-one correspondence between points in the point cloud image and points in the template image; modify the point cloud image according to the displacement offset to obtain a modified to-be-confirmed image; confirm a difference image between the template image and the to-be-confirmed image, and remove the difference image to obtain the modified image.

[0146] In this embodiment, in the case of determining the template image, the points in the point cloud image are corresponded with the points in the template image, the displacement offset of part of the points in the point cloud image compared with the template image can be determined according to the correspondence between the points and the points, and the point cloud image can be preliminarily modified according to the displacement offset to obtain a to-be-confirmed image. Then the to-be-confirmed image is compared with the template image again to confirm whether there is a difference between the to-be-confirmed image and the template image, if there is no difference, the to-be-confirmed image can be directly used as the modified image, if there is a difference between the to-be-confirmed image and the template image, the difference part, i.e. the difference image, is confirmed, the difference image in the to-be-confirmed image is removed, and the modified modified image is obtained. Through the above method, the accuracy of the image modification process is improved.

[0147] In any of the above embodiments, the image pose processing apparatus further comprises: a training module, before obtaining the spatial feature of the target object in the sample image, the training module is configured to: obtain a first spatial feature of a calibration RGB image and a second spatial feature of a calibration grayscale image, the calibration grayscale image being a grayscale image of the calibration RGB image; combine the first spatial feature and the second spatial feature to obtain a fused spatial feature; determine a vector feature of a key point in the fused spatial feature; train a first target network according to the calibration RGB image and the vector feature of the key point; and wherein the sample image is input into the first target network to obtain the spatial feature.

[0148] In this embodiment, in order to accurately obtain the spatial feature of the target object, the first target network needs to be trained first. Specifically, a calibration RGB image and a calibration grayscale image are obtained first, and the calibration grayscale image is a grayscale image of the calibration RGB image. In the calibration process, the first spatial feature of the calibration RGB image and the second spatial feature of the grayscale image are obtained respectively, and the first spatial feature and the second spatial feature are combined to obtain the fused spatial feature. This step can compensate for the calibration RGB image and the calibration grayscale image.

[0149] In the case of obtaining the fused spatial feature, the vector feature of the key point can be obtained from the fused spatial feature, and the key point is used to represent the key position of the calibration image. Then the calibration RGB image and the vector feature of the key point are input into the first target network for training, the calibration RGB image is input and the vector feature of the key point is output, so that the trained first target network can accurately extract the vector feature of the key point in the RGB image when the RGB image is input. After the training of the first target network is completed, the sample image can be input into the first target network, and the spatial feature of the key point in the sample image can be automatically obtained. Training the first target network in the above manner is beneficial to improve the accuracy of obtaining the spatial feature, thereby improving the accuracy of obtaining the 6D object pose of the target object.

[0150] In any of the above embodiments, the image pose processing apparatus further comprises: a training module, before obtaining the spatial feature of the target object in the sample image, the training module is configured to: obtain a first spatial feature of a calibration RGB image and a second spatial feature of a calibration grayscale image, the calibration grayscale image being a grayscale image of the calibration RGB image; combine the first spatial feature and the second spatial feature to obtain a fused spatial feature; determine a vector feature of a key point in the fused spatial feature; train a first target network according to the calibration RGB image and the vector feature of the key point; and wherein the sample image is input into the first target network to obtain the spatial feature.

[0151] In this embodiment, in order to accurately obtain the first feature of the sampling image, the second target network needs to be trained first. Specifically, a calibration image is first obtained, and then a first calibration point cloud of the calibration image is obtained. Then, the first calibration point cloud is rotated to obtain a second calibration point cloud after rotation. It should be noted that the first calibration point cloud and the second calibration point cloud are the same point cloud, only the point cloud before and after rotation is different. When the first calibration point cloud is rotated, the rotation angle of each time needs to be recorded, so that the image of the calibration point cloud before rotation, the rotation angle of the calibration point cloud, and the image of the calibration point cloud after rotation can be obtained.

[0152] The first calibration point cloud, the second calibration point cloud, and the rotation calibration angle are input into the second target network for training. The first calibration point cloud and the second calibration point cloud are input, and the rotation calibration angle is output. Thus, the trained second target network can accurately obtain the relative rotation angle of the two point clouds when the input point clouds have different rotation angles. After the training of the second target network is completed, the sampling image can be input into the second target network, and the rotation angle of the sampling image relative to the template image, i.e., the first feature, can be automatically obtained. Training the second target network in the above manner is conducive to improving the accuracy of obtaining the first feature, thereby improving the accuracy of obtaining the 6D object pose of the target object.

[0153] In any of the above embodiments, the image pose processing apparatus further includes a training module. Before obtaining the spatial feature of the target object in the sampling image, the training module is configured to: capture a calibration image by a camera; obtain a first size feature and a first displacement feature of a third calibration point cloud of the calibration image; capture the calibration image after changing the shooting position of the camera to obtain a fourth calibration point cloud; obtain a second size feature and a second displacement feature of the fourth calibration point cloud according to the intrinsic parameter of the camera; calculate a difference size feature of the second size feature and the first size feature, and a difference displacement feature of the second displacement feature and the first displacement feature; train a third target network according to the third calibration point cloud, the fourth calibration point cloud, the difference size feature, and the difference displacement feature; and input the sampling image into the third target network to obtain the second feature.

[0154] In this embodiment, in order to accurately obtain the spatial features of the target object, the third target network needs to be trained first. Specifically, first, a calibration image is captured by a camera, and then a third calibration point cloud of the calibration image is obtained. At this time, the first size feature and the first displacement feature of the third calibration point cloud can be actively labeled by a user. Then, the shooting position of the camera is changed. After the position is changed, the fourth calibration point cloud can be captured by the camera. At this time, the moving position of the camera can be obtained according to the intrinsic parameters of the camera, so that the second size feature and the second displacement feature of the fourth calibration point cloud can be indirectly calculated according to the moving position of the camera. Therefore, it is not necessary to label the size feature and the displacement feature of the point cloud every time the shooting position is changed, which is beneficial to improve the training efficiency.

[0155] It should be noted that the third calibration point cloud and the fourth calibration point cloud are the same point cloud, only the shooting position is changed.

[0156] The third calibration point cloud, the fourth calibration point cloud, the difference size feature and the difference displacement feature are input into the third target network for training. The third calibration point cloud and the fourth calibration point cloud are input, and the difference size feature and the difference displacement feature are output, so that the third target network after training can accurately obtain the size difference and the displacement difference of the two point clouds in the case of inputting point clouds with different sizes or positions. After the training of the third target network is completed, the sample image can be input into the third target network, and the size difference and the displacement difference of the sample image compared with the template image, i.e., the second feature, can be automatically obtained. Through the above method for training the third target network, the accuracy of obtaining the third feature is improved, and thus the accuracy of obtaining the 6D pose of the target object is improved.

[0157] In any of the above embodiments, after the first feature and the second feature in the sample image are obtained, the acquisition module is further configured to determine a size residual and a displacement residual according to the second feature and the spatial feature. The processing module is specifically configured to obtain the 6D pose of the target object according to the combination of the template and the first feature, the combination of the second feature, the spatial feature, the size residual and the displacement residual, and the modified image.

[0158] In this embodiment, when the 6D pose of the target object is determined, if the size or position difference of the point cloud of the target object and the template image is poor, parameter compensation needs to be performed. By participating in the calculation through the size residual and the displacement residual, the 6D pose of the target object can be more accurately obtained.

[0159] In one of the embodiments, as Figure 3As shown, there is provided an image pose processing apparatus 1100, comprising: a processor 1102 and a memory 1104, the memory 1104 storing programs or instructions executable on the processor 1102, the programs or instructions being executed by the processor 1102 to implement the steps of the method of any one of the above embodiments.

[0160] In one embodiment, there is provided a readable storage medium, the readable storage medium storing programs or instructions executable by a processor to implement the steps of the method of any one of the above embodiments.

[0161] In one embodiment, there is provided a robot, comprising: an image pose processing apparatus of any one of the above; and / or a readable storage medium of the above.

[0162] In the specification and claims of the present application, the features of the terms "first", "second" can be explicitly or implicitly included one or more features. In the description of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more. In addition, "and / or" in the specification and claims means at least one of the connected objects, and the character " / ", generally means that the front and rear associated objects are in an "or" relationship.

[0163] In the description of the present application, it can be understood that the terms "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the embodiments of the present application and simplifying the description of the embodiments of the present application, and do not indicate or imply that the structures, devices, elements referred to must have a particular orientation, be constructed and operated in a particular orientation, therefore these descriptions cannot be understood as a limitation of the present application.

[0164] In the description of the present application, it can be understood that, unless there is a clear specification and limitation, the terms "mounting", "connecting", "connecting" should be broadly understood, for example, it can be fixedly connected, or it can be detachably connected, or it can be integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0165] In the claims, the specification, and the drawings of the application, terms such as "a plurality" and "a plurality of" encompass two or more instances. The terms "upper" and "lower" are used for convenience with reference to the orientation of the figures and are not limiting. The terms "connected" and "coupled" are not restricted to physical or mechanical connections or couplings. Rather, these connections can be any form of connection or coupling, whether direct or indirect, physical or logical, and can involve the medium of electrical or electronic data signals.

[0166] In the claims, the specification, and the drawings of the application, terms such as "a plurality" and "a plurality of" encompass two or more instances. The terms "upper" and "lower" are used for convenience with reference to the orientation of the figures and are not limiting. The terms "connected" and "coupled" are not restricted to physical or mechanical connections or couplings. Rather, these connections can be any form of connection or coupling, whether direct or indirect, physical or logical, and can involve the medium of electrical or electronic data signals.

[0167] The preferred embodiments of the application are described above in detail. The application may, however, be embodied in various ways without being limited to the embodiments described above, and various changes and modifications can be suggested to one skilled in the art. It is intended to encompass any modification or alteration within the scope of the application.

Claims

1. A method of processing image poses, characterized in that, The method comprises the following steps: obtaining the spatial feature of the target object in the sampling image and the point cloud image of the target object; determining a template image matched with the point cloud image, correcting the point cloud image based on the template image to obtain a corrected image; obtaining a first feature and a second feature in the point cloud image, wherein the first feature comprises a rotation feature, and the second feature comprises a size feature and a displacement feature; obtaining the 6D pose of the target object according to the combination of the template feature and the first feature, the combination of the second feature and the spatial feature, and the corrected image, wherein the template feature is used to represent the vector feature of the template image; after obtaining the first feature and the second feature in the sampling image, the method further comprises the following steps: determining a size residual and a displacement residual according to the second feature and the spatial feature; obtaining the 6D pose of the target object according to the combination of the template feature and the first feature, the combination of the second feature and the spatial feature, and the corrected image, comprises: obtaining the 6D pose of the target object according to the combination of the template and the first feature, the combination of the second feature, the spatial feature, the size residual and the displacement residual, and the corrected image.

2. The method of claim 1, wherein, the method of correcting the point cloud image based on the template image to obtain a corrected image comprises: determining the displacement offset between the corresponding points in the point cloud image and the template image based on the one-to-one correspondence between the points in the point cloud image and the points in the template image; correcting the point cloud image according to the displacement offset to obtain a corrected image to be confirmed; confirming the difference image of the template image and the image to be confirmed, removing the difference image to obtain a corrected image.

3. The method according to claim 1 or 2, characterized in that, before obtaining the spatial feature of the target object in the sampling image, the method further comprises the following steps: obtaining a first spatial feature of a calibration RGB image and a second spatial feature of a calibration grayscale image, wherein the calibration grayscale image is a grayscale image of the calibration RGB image; combining the first spatial feature and the second spatial feature to obtain a fused spatial feature; determining the vector feature of the key point in the fused spatial feature; training a first target network according to the calibration RGB image and the vector feature of the key point; wherein the sampling image is input into the first target network to obtain the spatial feature.

4. The method according to claim 1 or 2, characterized in that, before obtaining the spatial feature of the target object in the sampling image, the method further comprises the following steps: obtaining a first calibration point cloud of a calibration image; rotating the first calibration point cloud by a calibration angle to obtain a second calibration point cloud; training a second target network according to the first calibration point cloud, the second calibration point cloud and the rotation calibration angle; wherein the sampling image is input into the second target network to obtain the first feature.

5. The method according to claim 1 or 2, characterized in that, before obtaining the spatial feature of the target object in the sampling image, the method further comprises the following steps: shooting a calibration image through a camera; obtaining a first size feature and a first displacement feature of a third calibration point cloud of the calibration image; shooting the calibration image after changing the shooting position of the camera to obtain a fourth calibration point cloud; According to the camera intrinsic parameters, a second size feature and a second displacement feature of the fourth calibration point cloud are obtained, a difference size feature of the second size feature and the first size feature, and a difference displacement feature of the second displacement feature and the first displacement feature are calculated; According to the third calibration point cloud, the fourth calibration point cloud, the difference size feature and the difference displacement feature, a third target network is trained; Wherein, the sampling image is input into the third target network, and the second feature is obtained.

6. An image pose processing apparatus, characterized by comprising: Comprising: An acquisition module acquires the spatial feature of the target object in the sampling image and the point cloud image of the target object; A determination module determines a template image matched with the point cloud image, and corrects the point cloud image based on the template image to obtain a corrected image; The acquisition module is further used to acquire a first feature and a second feature in the point cloud image, the first feature comprising a rotation feature, and the second feature comprising a size feature and a displacement feature; A processing module obtains the 6D pose of the target object according to the combination of the template feature and the first feature, the combination of the second feature and the spatial feature, and the corrected image, wherein the template feature is used to represent the vector feature of the template image; After acquiring the first feature and the second feature in the sampling image, the acquisition module is further used to determine a size residual and a displacement residual according to the second feature and the spatial feature; and the processing module is specifically used to obtain the 6D pose of the target object according to the combination of the template and the first feature, the combination of the second feature, the spatial feature, the size residual and the displacement residual, and the corrected image.

7. An image pose processing apparatus, characterized by comprising: Comprising: A processor and a memory, the memory stores programs or instructions executable on the processor, and the programs or instructions are executed by the processor to realize the steps of the method in any one of claims 1 to 5.

8. A readable storage medium, characterized by, The readable storage medium stores programs or instructions, and the programs or instructions are executed by the processor to realize the steps of the method in any one of claims 1 to 5.

9. A robot, characterized in that Comprising: The device of claim 6 or 7; And / or The readable storage medium of claim 8.

Images