Six-dimensional pose estimation method, network training method, apparatus, device, and medium

Abstract

Embodiments of the present application provide a six-dimensional pose estimation method, a network training method, an apparatus, a device and a medium, wherein: image data including a to-be-estimated object is obtained; coordinate association information of the to-be-estimated object on a depth image in the image data is determined; based on the image data and the coordinate association information, rotation information and translation information of the to-be-estimated object from a world coordinate system to a camera coordinate system are determined; and based on the rotation information and the translation information, a six-dimensional pose of the to-be-estimated object in the camera coordinate system is determined.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of image processing, and relate to but are not limited to a six-dimensional pose estimation method, a network training method, an apparatus, a device, and a medium. BACKGROUND

[0002] A red green blue-depth (RGB-D) sensor provides features of surface texture and geometry of an object, and with the price of the RGB-D sensor decreasing, an RGB-D image becomes a mainstream input form of six-dimensional pose estimation. In the related art, two neural networks are used to extract features of a color image and a depth image respectively; the features of the color image and the depth image are fused, and then complex post-processing is added to obtain displacement and rotation of the object. In this way, the scheme of six-dimensional pose estimation is too complex, is not concise, and cannot realize six-dimensional pose estimation with a unified neural network. SUMMARY

[0003] Embodiments of the present application provide a six-dimensional pose estimation technical scheme.

[0004] The technical scheme of the embodiments of the present application is implemented as follows:

[0005] The embodiments of the present application provide a six-dimensional pose estimation method, which comprises:

[0006] Obtaining image data comprising an object to be estimated;

[0007] Determining coordinate association information of the object to be estimated on a depth image in the image data;

[0008] Based on the image data and the coordinate association information, determining rotation information and translation information of the object to be estimated from a world coordinate system to a camera coordinate system;

[0009] Based on the rotation information and the translation information, determining a six-dimensional pose of the object to be estimated in the camera coordinate system.

[0010] In some embodiments, the determining, based on the image data and the coordinate association information, of the rotation information and the translation information of the object to be estimated from the world coordinate system to the camera coordinate system comprises: fusing the image data and the coordinate association information to obtain fused data; and determining, based on the fused data, the rotation information and the translation information of the object to be estimated from the world coordinate system to the camera coordinate system. In this way, the spatial transformation in the neural network can not change the mapping relationship between the world coordinates of the object to be estimated and the two-dimensional coordinates in the depth image.

[0011] In some embodiments, the determining the rotation information and the translation information of the object to be estimated from the world coordinate system to the camera coordinate system based on the fusion data comprises: performing feature extraction on the fusion data to obtain a fusion feature map; detecting the object to be estimated based on the fusion feature map to determine at least one detection box of the object to be estimated; and processing each detection box to obtain the rotation information and the translation information of the object to be estimated in each detection box. In this way, by using the basic subnetwork and the rotation and translation subnetwork in the six-dimensional pose estimation network, the rotation information and the translation information of the object to be estimated in each detection box from the world coordinate system to the camera coordinate system can be more accurately predicted, and thus the estimated six-dimensional pose of the object to be estimated is more accurate.

[0012] In some embodiments, the processing each detection box to obtain the rotation information and the translation information of the object to be estimated in each detection box comprises: performing at least one feature space transformation on each detection box to obtain first transformed features; wherein the output of the last feature space transformation is the input of the next feature space transformation; and performing feature space transformations of different spatial dimensions on the first transformed features respectively to obtain the rotation information and the translation information. In this way, by using two shared fully connected layers and two independent fully connected layers to obtain the rotation information and the three-dimensional translation information, the six-dimensional pose of the object to be estimated can be accurately estimated based thereon.

[0013] In some embodiments, the determining the coordinate association information of the object to be estimated on the depth image in the image data comprises: determining a first coordinate of a world coordinate of the object to be estimated projected onto the depth image in a rectangular coordinate system on a plane on which the depth image is located; performing coordinate system conversion on the first coordinate to determine a second coordinate of the object to be estimated in the camera coordinate system; encoding the first coordinate using a preset trigonometric function to obtain encoding information; and determining at least one of the first coordinate, the second coordinate, and the encoding information as the coordinate association information. In this way, the coordinate association information is determined in multiple ways, thereby facilitating the introduction of the coordinate association information in the image data to solve the projection decomposition problem when the same neural network is used to process the color image and the depth image.

[0014] In some embodiments, the fusing the image data and the coordinate-related information to obtain fused data comprises: in a case where the coordinate-related information comprises at least one of the first coordinate and the second coordinate, splicing at least one of the first coordinate and the second coordinate and the image data to obtain the fused data. In this way, the coordinate-related information and the image data are spliced, which can alleviate the destruction of the mapping relationship between the coordinates in the depth image and the world coordinates due to convolution.

[0015] In some embodiments, the fusing the image data and the coordinate-related information to obtain fused data comprises: in a case where the coordinate-related information comprises the encoding information, element-wise adding the image data and the encoding information to obtain the fused data. In this way, the encoding information and the image data are added, which can increase the channels of the fused data and alleviate the destruction of the mapping relationship between the coordinates in the depth image and the world coordinates due to convolution.

[0016] In some embodiments, the obtaining the image data of the object to be estimated comprises: obtaining a color image and a depth image of the object to be estimated; and splicing the color image and the depth image to obtain the image data. In this way, by inputting the spliced image data into the six-dimensional pose estimation network, the same network can be used to perform feature extraction and other processing on the depth image and the color image.

[0017] The embodiment of the application provides a kind of six-dimensional attitude estimation network training method, it is characterized in that, the six-dimensional attitude estimation network is obtained by training the six-dimensional attitude estimation network to be trained, the six-dimensional attitude estimation network to be trained at least includes: the rotation translation sub-network to be trained and the basic sub-network to be trained, the method comprises: obtaining the sample fusion data of sample object with labeled six-dimensional attitude;Wherein, the sample fusion data includes the training image data of the sample object and training coordinate association information;Using the basic sub-network to be trained, the feature extraction of the fusion data is carried out, and sample fusion feature map is obtained;Using the rotation translation sub-network to be trained, the predicted rotation information and the predicted translation information of the sample object from world coordinate system to camera coordinate system are determined based on the sample fusion feature map;Based on the predicted rotation information and the predicted translation information, and the real rotation information and the real translation information of the sample object, determine first loss;At least based on the first loss and the basic loss of the predicted region of interest, the network parameters of the six-dimensional attitude estimation network to be trained are adjusted, to make the loss of the adjusted six-dimensional attitude estimation network output satisfy convergence condition.Such, by introducing rotation translation sub-network in network to predict the rotation information and translation information of sample object, simplify network structure, and by introducing training coordinate association information in training image data, end-to-end training can be realized in a convolutional neural network.

[0018] In some embodiments, the sample fusion data of sample object with labeled six-dimensional attitude is obtained by fusing the training image data and the training coordinate association information to obtain the sample fusion data.

[0019] In some embodiments, the predicted rotation information and the predicted translation information of the sample object from world coordinate system to camera coordinate system are determined based on the sample fusion feature map using the rotation translation sub-network to be trained, comprising: using the rotation translation sub-network to be trained based on the sample fusion feature map, detecting the sample object to determine at least one detection frame of the sample object;Using the rotation translation sub-network to be trained to process each detection frame, to obtain the predicted rotation information and the predicted translation information of the sample object in each detection frame.Such, by introducing basic sub-network and rotation translation sub-network in the process of training, the rotation information and translation information of sample object from world coordinate system to camera coordinate system in each detection frame can be more accurately predicted, and then the estimated 6D attitude of sample object is more accurate.

[0020] In some embodiments, the to-be-trained rotation and translation sub-network comprises two connected first full connection layers, a second full connection layer and a third full connection layer, and the processing of each bounding box by the to-be-trained rotation and translation sub-network to obtain the predicted rotation information and the predicted translation information of the sample object in the each bounding box comprises: performing feature space transformation on the each bounding box by the two connected first full connection layers to obtain first transformed sample features; performing feature space transformation on the first transformed sample features by the second full connection layer and the third full connection layer respectively to obtain the predicted rotation information and the predicted translation information. In this way, the rotation information and the three-dimensional translation information are obtained through the two shared full connection layers and the two independent full connection layers in the to-be-trained rotation and translation sub-network, and the 6D pose of the sample object can be accurately estimated based on this.

[0021] In some embodiments, the to-be-trained six-dimensional pose estimation network further comprises a world coordinate sub-network, and the method further comprises: determining, by the to-be-trained world coordinate sub-network, predicted world coordinates of the sample object in the world coordinate system based on the predicted region of interest; determining a second loss based on the predicted world coordinates and the real world coordinates of the sample object; and adjusting the network parameters of the to-be-trained six-dimensional pose estimation network based on at least the first loss and a base loss of the to-be-trained base sub-network, which comprises: adjusting the network parameters of the to-be-trained six-dimensional pose estimation network based on the first loss, the second loss and the base loss. In this way, the sub-network of the three-dimensional coordinates is introduced into the trained six-dimensional pose network to assist the network in predicting the three-dimensional coordinates of the object, which can improve the training efficiency of the trained six-dimensional pose network and make the trained six-dimensional pose network more robust.

[0022] In some embodiments, the determination of the first loss based on the predicted rotation information and the predicted translation information, and the real rotation information and the real translation information of the sample object comprises: fusing the predicted rotation information and the predicted translation information of any point on the sample object to obtain a predicted six-dimensional pose; fusing the real rotation information and the real translation information of the any point to obtain a real six-dimensional pose; and determining the first loss based on the predicted six-dimensional pose and the real six-dimensional pose. In this way, the difference between the predicted six-dimensional pose and the real six-dimensional pose of each sampling point in the object to be estimated can be effectively determined to determine the loss of the predicted rotation information and the predicted translation information of the rotation and translation sub-network.

[0023] In some embodiments, the determining the second loss based on the predicted world coordinate and the real world coordinate of the sample object comprises: determining a difference between the predicted world coordinate and the real world coordinate in each coordinate dimension; and fusing the differences in multiple coordinate dimensions to obtain the second loss. In this way, a more accurate second loss can be obtained by determining the difference between the predicted world coordinate and the real world coordinate in each coordinate dimension, and the second loss can play an auxiliary role in the process of training the to-be-trained six-dimensional pose estimation network.

[0024] In some embodiments, after the predicting the region of interest in the sample fusion data by using the to-be-trained base subnetwork, the method further comprises: determining a third loss of detecting the sample object in the predicted region of interest by using a detection subnetwork in the to-be-trained base subnetwork; determining a fourth loss of instance segmentation of the sample object based on the predicted region of interest by using a mask subnetwork in the to-be-trained base subnetwork; and obtaining the base loss based on the third loss and the fourth loss. In this way, the object detection and instance segmentation can be fully considered in the process of training the to-be-trained six-dimensional pose estimation network, so that the performance of the base subnetwork in the trained six-dimensional pose estimation network is higher.

[0025] Embodiments of the present application provide a six-dimensional pose estimation device, the device comprising:

[0026] The first acquisition module is configured to acquire image data comprising a to-be-estimated object;

[0027] The first determination module is configured to determine coordinate association information of the to-be-estimated object in a depth image of the image data;

[0028] The second determination module is configured to determine rotation information and translation information of the to-be-estimated object from a world coordinate system to a camera coordinate system based on the image data and the coordinate association information;

[0029] The third determination module is configured to determine a six-dimensional pose of the to-be-estimated object in the camera coordinate system based on the rotation information and the translation information.

[0030] Embodiments of the present application provide a training device of a six-dimensional pose network, the six-dimensional pose estimation network being obtained by training a to-be-trained six-dimensional pose estimation network, the to-be-trained six-dimensional pose estimation network comprising at least a to-be-trained rotation and translation subnetwork and a to-be-trained base subnetwork, the device comprising:

[0031] The second acquisition module is configured to acquire sample fusion data of a sample object with a labeled six-dimensional pose; wherein the sample fusion data comprises training image data and training coordinate association information of the sample object.

[0032] The first feature extraction module is configured to perform feature extraction on the fusion data by using the to-be-trained basic subnetwork to obtain sample fusion feature maps.

[0033] The first prediction module is configured to determine predicted rotation information and predicted translation information of the sample object from a world coordinate system to a camera coordinate system based on the sample fusion feature maps by using the to-be-trained rotation and translation subnetwork.

[0034] The fourth determination module is configured to determine a first loss based on the predicted rotation information and the predicted translation information, and real rotation information and real translation information of the sample object.

[0035] The first adjustment module is configured to adjust network parameters of the to-be-trained six-dimensional pose estimation network based on at least the first loss and the basic loss of the predicted region of interest, so that a loss output by the adjusted six-dimensional pose estimation network satisfies a convergence condition.

[0036] Embodiments of the present application provide a computer storage medium, which stores computer executable instructions. The computer executable instructions, when executed, can implement the steps of the above method.

[0037] Embodiments of the present application provide a computer device, which comprises a memory and a processor. The memory stores computer executable instructions. When the processor executes the computer executable instructions on the memory, the steps of the above method can be implemented.

[0038] Embodiments of the present application provide a six-dimensional pose estimation method, a network training method, an apparatus, a device and a medium. For the acquired image data of a to-be-estimated object, first, the coordinate association information of the to-be-estimated object on the depth image in the image data is determined. Then, the coordinate association information and the image data are combined to determine the selection information and the translation information of the to-be-estimated object from the world coordinate system to the camera coordinate system. In this way, the projection relationship between the world coordinate of the to-be-estimated object and the coordinate on the depth image can be maintained, and the same neural network can be used to process the image data including the depth image and the color image, so that the network for six-dimensional pose estimation is more concise. Finally, the six-dimensional pose of the to-be-estimated object in the camera coordinate system can be determined by the determined rotation information and translation information. In this way, the neural network for six-dimensional pose estimation can be simplified, and the efficiency of six-dimensional pose estimation can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 An implementation flowchart of the six-dimensional pose estimation method provided by the embodiment of the present application is shown in FIG. 1.

[0040] Figure 2 Another implementation flowchart of the six-dimensional pose estimation method provided by the embodiment of the present application is shown in FIG. 2.

[0041] Figure 3 An implementation flowchart of the training method of the six-dimensional pose estimation network provided by the embodiment of the present application is shown in FIG. 3.

[0042] Figure 4 A projection diagram of the color image and the depth image provided by the embodiment of the present application is shown in FIG. 4.

[0043] Figure 5 A network architecture diagram of the six-dimensional pose estimation method provided by the embodiment of the present application is shown in FIG. 5.

[0044] Figure 6 An architecture diagram of the subnetwork provided by the embodiment of the present application is shown in FIG. 6.

[0045] Figure 7 An application scenario diagram of the six-dimensional pose estimation method provided by the embodiment of the present application is shown in FIG. 7.

[0046] Figure 8 A structural composition diagram of the six-dimensional pose estimation device provided by the embodiment of the present application is shown in FIG. 8.

[0047] Figure 9 A structural composition diagram of the training device of the six-dimensional pose estimation network provided by the embodiment of the present application is shown in FIG. 9.

[0048] Figure 10 A composition structure diagram of the computer device of the embodiment of the present application is shown in FIG. 10. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the specific technical scheme of the present application will be further described in detail below with reference to the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but not to limit the scope of the present application.

[0050] In the following description, "some embodiments" are described, which describe a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subset of all possible embodiments, and can be combined with each other without conflict.

[0051] In the following description, the terms "first", "second", "third", etc. are merely used to distinguish similar instances, and do not represent a specific order or sequence of the instances. Understandably, the "first", "second", "third", etc. can be interchanged in a specific order or sequence as allowed, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this application is for the purpose of describing embodiments of this application only, and is not intended to be limiting of this application.

[0053] Before further detailing the embodiments of the present application, the terms and phrases involved in the embodiments of the present application are explained, and the terms and phrases involved in the embodiments of the present application are applicable to the following explanations.

[0054] 1) Six-dimensional pose estimation is an accurate estimation of 3D position and 3D orientation information of an object. Six-dimensional pose estimation refers to estimating the six-dimensional pose of an object in the camera coordinate system, i.e., 3D position and 3D orientation. At this time, the coordinate system of the original object itself can be regarded as the world coordinate system, i.e., the displacement and rotation transformation of the object from the world system to the camera system are obtained.

[0055] 2) The RGB-D image is two images: one is a normal three-channel color image, and the other is a depth image. The image is similar to a grayscale image, except that each pixel value is the actual distance from the sensor to the object. Usually, the color image and the depth image are registered, so there is a one-to-one correspondence between the pixels.

[0056] The following describes an exemplary application of the device for pose estimation provided by the embodiments of the present application. The device provided by the embodiments of the present application can be implemented as various types of user terminals having an image acquisition function, such as a notebook computer, a tablet computer, a desktop computer, a camera, a mobile device (e.g., a personal digital assistant, a dedicated messaging device, a portable game device), etc., or as a server. In the following, exemplary applications when the device is implemented as a terminal or a server will be described.

[0057] The method can be applied to a computer device, and the functions implemented by the method can be implemented by calling program codes by a processor in the computer device. Of course, the program codes can be saved in a computer storage medium, and therefore the computer device at least includes a processor and a storage medium.

[0058] With RGB-D sensors more readily available, it is a better choice to obtain high-precision six-dimensional pose estimation results using RGB-D images. In the depth image plane, the projected 3D structure of the physical world is preserved by the 1D depth values and their built-in plane pixel coordinates (u, v). Any spatial transformation will change (u, v); for example, resizing, flipping, cropping, or pooling operations in a Convolutional Neural Networks (CNN) framework will destroy the binding between pixel values and (u, v) coordinates. Therefore, the modified depth image or features no longer preserve the 3D structure.

[0059] Based on this, the embodiment of the present application provides a six-dimensional pose estimation method, which takes the coordinate association information together with the RGB-D image as input, and uses a single CNN backbone network to perform six-dimensional pose estimation. The six-dimensional pose estimation method can be implemented through a trained six-dimensional pose estimation network, as shown in Figure 1 , which is described in combination with the steps as shown in Figure 1 .

[0060] In step S101, image data including a to-be-estimated object is obtained.

[0061] In some embodiments, the to-be-estimated object can be an object, a pedestrian, a vehicle, a tree, or a building, etc. in an arbitrary scene. The image data of the to-be-estimated object is a color-depth image collected for the to-be-estimated object, including a color image and a depth image collected for the to-be-estimated object. The color image can be a Red Green Blue (RGB) image, and the color image and the depth image have a one-to-one correspondence between pixel points. In the image data, the to-be-estimated object and other objects can be mutually occluded, or can not be mutually occluded; for example, the image data is obtained by collecting a dining table on which multiple tea sets are placed, and if the to-be-estimated object is a teapot, the teapot and the teacup can be mutually occluded, or can not be mutually occluded.

[0062] In some embodiments, the number of to-be-estimated objects can be one, two, or more. Meanwhile, when the number of included to-be-estimated objects is two or more, the areas where different to-be-estimated objects are located in the image data can be adjacent, distant, or partially overlapped, etc., and the areas occupied by different to-be-estimated objects in the image data can be the same or different.

[0063] In some possible implementation manners, the image acquisition device can be used to acquire the image data of the object to be estimated, to obtain the color image and the depth image in the image data; or the color image and the depth image of the object to be estimated can be received from other devices. The RGB image of the object to be estimated can be acquired, and the depth image of the object to be estimated can be obtained by performing depth projection on the RGB image. In this way, the color image and the depth image in the image data are corresponding.

[0064] In step S102, the coordinate association information of the object to be estimated on the depth image in the image data is determined.

[0065] In some embodiments, the coordinate association information is used to represent the coordinate information of the three-dimensional world coordinate of the object to be estimated projected on the depth image, and can also represent the corresponding relationship between the three-dimensional world coordinate of the object to be estimated and the two-dimensional coordinate on the depth image. The coordinate association information of each point of the object to be estimated is determined; or the key points of the object to be estimated are determined first, and then the coordinate association information of each key point on the depth image is determined. The coordinate association information of the object to be estimated at least includes one of the following: the first coordinate of the three-dimensional world coordinate of the object to be estimated projected on the depth image, the horizontal and vertical coordinates (x, y) of the first coordinate in the camera coordinate determined by back projection on the two-dimensional coordinate (i.e., the second coordinate), and the encoding information obtained by encoding the first coordinate by using the trigonometric function.

[0066] Since the image data is acquired for the object to be estimated, the three-dimensional world coordinate of the object to be estimated is known, and the two-dimensional rectangular coordinate system is established by taking the depth image as a plane, so that the first coordinate of each point of the object to be estimated on the depth image can be obtained. In this way, the second coordinate in the camera coordinate can be obtained by back projection on the first coordinate, and the encoding information can be obtained by encoding the first coordinate by using the trigonometric function.

[0067] In step S103, the rotation information and the translation information of the object to be estimated from the world coordinate system to the camera coordinate system are determined based on the image data and the coordinate association information.

[0068] In some embodiments, the image data and the coordinate association information are combined, and global analysis is performed on the combined information to determine the rotation information and the translation information of the object to be estimated. If one of the first coordinate, the second coordinate and the encoding information is included in the coordinate association information, the one coordinate association information is combined with the color image and the depth image of the object to be estimated; if two of the first coordinate, the second coordinate and the encoding information are included in the coordinate association information, the two information is combined with the color image and the depth image of the object to be estimated.

[0069] In some possible implementation manners, after the image data is combined with the coordinate association information, feature extraction is performed on the combined data to determine a plurality of regions of interest. The plurality of regions of interest are fused by using a plurality of fully connected layers to determine rotation information and translation information of the to-be-estimated object from a world coordinate system to a camera coordinate system. The rotation information represents a spatial rotation condition of each sampling point in the to-be-estimated object, and is used to determine a three-dimensional pose of the sampling point, which can be represented by using a 1*4 quaternion rotation matrix. The translation information represents a three-dimensional position of each sampling point in the to-be-estimated object, and can be represented by using a 1*3 translation matrix.

[0070] In step S104, six-dimensional poses of the to-be-estimated object are determined based on the rotation information and the translation information.

[0071] In some embodiments, the six-dimensional (6D) poses of the to-be-estimated object include three-dimensional positions of each sampling point in the to-be-estimated object in the camera coordinate system and three-dimensional direction information of the sampling point in the three-dimensional camera space. A coordinate system of the to-be-estimated object itself can be regarded as a world coordinate system, and thus an RT transformation from the world coordinate system of the to-be-estimated object to the camera coordinate system is obtained.

[0072] In some possible implementation manners, the three-dimensional vector of the sampling point after rotation is obtained by multiplying the quaternion rotation matrix in the rotation information and the homogeneous coordinates of the corresponding sampling point. The three-dimensional vector represents the pose of the sampling point in the three-dimensional camera space. The three-dimensional position of the sampling point in the camera coordinate system is obtained by multiplying the translation matrix in the translation information and the world coordinates of the corresponding sampling point. The 6D pose of the sampling point is obtained by combining the pose of the sampling point in the three-dimensional camera space and the three-dimensional position.

[0073] In the embodiments of the present application, for the obtained image data of the to-be-estimated object, first, the coordinate association information of the to-be-estimated object on the depth image in the image data is determined. Then, the coordinate association information is combined with the image data to determine the rotation information and the translation information of the to-be-estimated object from the world coordinate system to the camera coordinate system. In this way, the image data including the depth image and the color image can be processed by using the same neural network without destroying the projection relationship between the world coordinates of the to-be-estimated object and the coordinates on the depth image, so that the network for 6D pose estimation is more concise. Finally, the 6D pose of the to-be-estimated object in the camera coordinate system is determined according to the determined rotation information and translation information. In this way, the neural network for 6D pose estimation is simplified, and the efficiency of 6D pose estimation is improved.

[0074] In some embodiments, the image data and the coordinate association information are fused, so that the RGB image and the depth image in the image data can be processed by using the same neural network, that is, the above step S103 can be implemented by Figure 2 The step achieves:

[0075] In step S201, the image data and the coordinate association information are fused to obtain fused data.

[0076] In some embodiments, if the coordinate association information only includes the two-dimensional coordinates of any sampling point on the object to be estimated in the depth image, the two-dimensional coordinates are spliced with the image data to obtain the fused data.

[0077] The coordinate association information can also include the horizontal and vertical coordinates of the sampling point in the camera coordinate system obtained by back-projection of the two-dimensional coordinates, and the two-dimensional coordinates, the horizontal and vertical coordinates, and the image data are spliced to obtain the fused data.

[0078] The coordinate association information can also include encoded information obtained by encoding the two-dimensional coordinates, and the encoded information is fused on the basis of the spliced two-dimensional coordinates, horizontal and vertical coordinates, and image data to obtain the fused data.

[0079] In some possible implementations, by introducing coordinate association information in image data including color images and depth images, the problem of projection decomposition when using the same neural network to process color images and depth images is solved. The coordinate association information can include the following forms:

[0080] Method one: in the rectangular coordinate system of the plane where the depth image is located, the first coordinate of the world coordinate of the object to be estimated projected onto the depth image is determined.

[0081] Here, a rectangular coordinate system is established with the plane where the depth image in the image data is located, for example, a rectangular coordinate system is established with a certain vertex of the depth image as the origin. For any sampling point on the object to be estimated, the two-dimensional coordinates of the world coordinate of the sampling point in the world coordinate system projected into the rectangular coordinate system of the depth image are determined, that is, the first coordinate (u, v) is obtained. The first coordinate on the depth image has the same height and width as the image data. The first coordinate has two channels, one channel U stores the u coordinate, and the other channel V stores the v coordinate, that is, the value of u at the pixel (u, v) is stored in the U channel, and v at the pixel (u, v) is stored in the V channel.

[0082] Method two: the first coordinate is converted in the coordinate system to determine the second coordinate of the object to be estimated in the camera coordinate system.

[0083] Here, the two-dimensional coordinates in the coordinate system of the depth image are back projected to the camera coordinate system by back projecting the first coordinates of all sampling points on the object to be estimated, to obtain the horizontal and vertical coordinates in the camera coordinate system, i.e., the second coordinates. The second coordinates are the coordinates corresponding to each sampling point. The second coordinates also have two channels, and the second coordinates have the same height and width as the image data.

[0084] In some possible implementation manners, if at least one of the first coordinates and the second coordinates is taken as the coordinate association information, the above step S201 can be implemented by the following process:

[0085] In a case where the coordinate association information includes at least one of the first coordinates and the second coordinates, at least one of the first coordinates and the second coordinates is spliced with the image data to obtain the fusion data.

[0086] Here, if the coordinate association information is the first coordinates or the second coordinates, the first coordinates or the second coordinates are spliced with the image data according to the channel dimension of the image data to obtain the fusion data. In this way, the fusion data includes the image data and the first coordinates or the second coordinates in the coordinate association information.

[0087] If the coordinate association information is the first coordinates and the second coordinates, the first coordinates, the second coordinates corresponding to each sampling point in the object to be estimated, and the image data are spliced according to the channel dimension to obtain the fusion data. In this way, the fusion data includes the image data, the first coordinates, and the second coordinates. In this way, by splicing the coordinate association information with the image data in a case where the coordinate association information includes at least one of the first coordinates and the second coordinates, the mapping relationship between the coordinates in the depth image and the world coordinates can be relieved due to convolution.

[0088] Method three: encoding the first coordinates by using a preset trigonometric function to obtain the encoding information.

[0089] Here, the first coordinates are encoded by using a trigonometric function by the encoding method of the Transfomer and then added to the input. Since the encoding information is multi-channel, the encoding information can be added and fused with other channels.

[0090] In some possible implementation manners, if the coordinate association information is the encoding information, the above step S201 can be implemented by the following process:

[0091] In a case where the coordinate association information includes the encoding information, the image data and the encoding information are element-wise added to obtain the fusion data.

[0092] Here, if the encoding information is included in the coordinate association information, the encoding information is added to the image data element by element to obtain the fusion data.

[0093] If the encoding information and at least one of the first coordinate and the second coordinate are included in the coordinate association information, at least one of the first coordinate and the second coordinate is spliced with the image data in the channel dimension, and then the spliced data is added to the encoding information to obtain the fusion data. In this way, the encoding information is added to the image data, which can increase the channels of the fusion data.

[0094] Through the above-mentioned ways one to three, the coordinate association information with rich information can be obtained, so that the fusion data obtained after the coordinate association information and the image data are fused is more rich.

[0095] In step S202, the rotation information and the translation information of the object to be estimated from the world coordinate system to the camera coordinate system are determined based on the fusion data.

[0096] In some embodiments, the fusion data obtained after the coordinate association information is fused with the image data is input into a six-dimensional pose estimation network, and the color image and the depth image in the fusion data are processed by the same network. This can not only affect the mapping relationship between the world coordinates of the object to be estimated and the coordinates on the depth image, but also can more accurately predict the rotation information and the translation information of the object to be estimated from the world coordinate system to the camera coordinate system.

[0097] Through the above-mentioned steps S201 and S202, the color image-depth image of the object to be estimated and the coordinate association information are fused and then input into the same neural network, so that the spatial transformation in the neural network will not change the mapping relationship between the world coordinates of the object to be estimated and the two-dimensional coordinates in the depth image,

[0098] In some embodiments, the fusion data is input into the same six-dimensional pose estimation network to estimate the six-dimensional pose, and the six-dimensional pose estimation network includes a basic sub-network and a rotation and translation sub-network. The above-mentioned step S202 can be implemented through the following steps S221 to S223 (not shown in the figure):

[0099] In step S221, feature extraction is performed on the fusion data to obtain a fusion feature map.

[0100] In some embodiments, the base subnetwork is employed to extract features from the fusion data to obtain a fusion feature map. In the process of estimating the 6D pose of the object to be estimated using the six-dimensional pose estimation network, the base subnetwork includes a backbone network, a region proposal network (RPN), and a detection subnetwork; wherein the backbone network can be implemented by a residual network (ResNet50) + feature pyramid network (FPN), and is used to extract features from image data and obtain a feature pyramid of the image data. The region proposal network is used to select candidate boxes from each size of feature map in the feature pyramid to determine a region of interest; and the detection subnetwork is used to detect the object in the region of interest to output a detection box. In this way, the backbone network in the base subnetwork is employed to extract features to obtain a fusion feature map, and the fusion feature map is a pyramid feature map, i.e., the fusion feature map includes multiple feature maps of different sizes.

[0101] In step S222, the object to be estimated is detected based on the fusion feature map to determine at least one detection box of the object to be estimated.

[0102] In some embodiments, the detection subnetwork (such as a bbox network) in the base subnetwork can be employed to determine a region of interest in the image data based on the fusion feature map; and by detecting the object in each region of interest, a detection box in each region of interest is obtained to obtain at least one detection box.

[0103] In some embodiments, the region proposal network in the base subnetwork is employed to select candidate boxes from the object to be estimated in the fusion feature map including multiple feature maps of different sizes, and the region of interest is obtained by aligning the extracted candidate boxes. In this way, the region of interest in each size of fusion feature map, i.e., multiple regions of interest, can be obtained.

[0104] In some embodiments, the object to be estimated can also be detected based on the fusion feature map by a transformer model to determine at least one detection box of the object to be estimated; in this way, the fusion feature map is taken as the input of the transformer model, and the detection of the object to be estimated can be realized to output the detection box of the object to be estimated.

[0105] The step S221 and the step S222 first perform feature extraction and feature pyramid network processing on the fusion data to obtain a fusion feature map; then input the obtained fusion feature maps of multiple sizes into a region selection network to select a candidate frame, and perform region of interest alignment on the selected candidate frame to obtain a region of interest; and finally perform object detection on the region of interest through a detection subnetwork.

[0106] The step S223 processes each detection frame to obtain the rotation information and the translation information of the to-be-estimated object in the each detection frame.

[0107] Here, the rotation and translation subnetwork is used to process each detection frame to obtain the rotation information and the translation information of the to-be-estimated object in the each detection frame. The rotation and translation subnetwork is used to determine the spatial rotation and three-dimensional translation of the to-be-estimated object. Each detection frame detected in the step S222 is input into the rotation and translation subnetwork; the rotation and translation subnetwork is used to process the region of interest corresponding to the each detection frame to obtain the rotation information and the translation information of the to-be-estimated object in the each detection frame, that is, to determine the spatial rotation and three-dimensional position of the to-be-estimated object. In this way, by using the basic subnetwork and the rotation and translation subnetwork in the six-dimensional pose estimation network, the rotation information and the translation information of the to-be-estimated object in each detection frame from the world coordinate system to the camera coordinate system can be more accurately predicted, and the estimated 6D pose of the to-be-estimated object is more accurate.

[0108] In some possible implementation manners, the rotation information and the translation information of the to-be-estimated object are obtained by processing each detection frame in the image data, that is, the step S223 can be implemented by the following steps:

[0109] Firstly, at least one feature space transformation is performed on the each detection frame to obtain first transformed features.

[0110] In some possible implementation manners, the output of the last feature space transformation is the input of the next feature space transformation. The two connected first fully connected layers can be used to perform feature space transformation on the each detection frame to obtain the first transformed features. In this way, the previous first fully connected layer in the connected first fully connected layers implements the last feature space transformation, and the next first fully connected layer implements the next feature space transformation.

[0111] Secondly, different spatial dimension feature space transformations are respectively performed on the first transformed features to obtain the rotation information and the translation information.

[0112] In some possible implementation manners, the second full connection layer and the third full connection layer are respectively adopted to perform feature space transformation on the first transformed features in different spatial dimensions, to obtain the rotation information and the translation information. The input dimensions of the second full connection layer and the third full connection layer are the same, and are N; the output dimension of the second full connection layer is 4, and the output dimension of the third full connection is 3. The first transformed features are respectively input into the second full connection layer and the third full connection layer; the second full connection layer is adopted to perform weighted summation on the input first transformed features, to implement feature space transformation, to obtain a rotation matrix with a dimension of 1×4, that is, the rotation information. In this way, the rotation of the object to be estimated is represented by using a quaternion, and the rotation of the object to be estimated in space can be effectively described. The third full connection layer is adopted to perform weighted summation on the input first transformed features, to implement feature space transformation, to obtain a translation matrix with a dimension of 1×3, that is, the translation information, and the spatial position of the object to be estimated can be effectively represented. In this way, the 6D pose of the object to be estimated can be accurately estimated based on the above first step and second step.

[0113] In some embodiments, the image data of the object to be estimated is obtained by splicing a depth image and a color image of the object to be estimated, that is, the above step S101 can be implemented through the following steps S111 and S112 (not shown in the figure):

[0114] In step S111, a color image and a depth image of the object to be estimated are obtained.

[0115] In some embodiments, for any object to be estimated, a color image and a depth image of the object to be estimated are acquired by an image acquisition device; or the color image and the depth image of the object to be estimated are received from other devices.

[0116] In step S112, the color image and the depth image are spliced to obtain the image data.

[0117] In some embodiments, the color image and the depth image are spliced according to the image channel dimension to obtain the image data. In some possible implementation manners, the color image can be an RGB image, and the RGB image and the depth image are spliced together according to the image channel dimension to obtain RGB-D image data.

[0118] By splicing the depth image and the color image according to the channel dimension through the above steps S111 and S112, the obtained image data includes both two-dimensional color information and depth information of the object to be estimated, and then the spliced image data is input into the six-dimensional pose estimation network, so as to facilitate feature extraction and other processing of the depth image and the color image by the same network.

[0119] This application provides a training method for a six-dimensional pose estimation network. The six-dimensional pose estimation network is obtained by training an existing six-dimensional pose estimation network to be trained. The existing six-dimensional pose estimation network to be trained includes: a rotation and translation sub-network to be trained, a world coordinate sub-network to be trained, and a base sub-network to be trained for predicting regions of interest. The training process of the six-dimensional pose estimation network to be trained is as follows: Figure 3 As shown, combined with Figure 3 The steps shown are explained below:

[0120] Step S301: Obtain sample fusion data of the sample objects with labeled six-dimensional poses.

[0121] In some embodiments, the sample fusion data includes training image data and training coordinate association information of the sample objects. The sample fusion data is obtained by fusing the training image data and the training coordinate association information.

[0122] First, acquire training image data including sample objects with labeled six-dimensional poses.

[0123] Here, the training image data includes color training images and depth training images of the sample object. The sample object can be any type of object, such as a physical object, pedestrian, vehicle, tree, or building; the sample object in the training image data has been labeled with its six-dimensional pose. The training image data is obtained by stitching the color training image and depth training image of the sample object together according to the channel dimension. The training image data can be RGB-D image data of the sample object.

[0124] Then, the training coordinate association information of the sample object on the depth training image in the training image data is determined.

[0125] Here, the coordinate association information is used to represent the coordinate information of the sample object's three-dimensional world coordinates projected onto the depth image, and can also represent the correspondence between the sample object's three-dimensional world coordinates and the two-dimensional coordinates on the depth image. Determining the training coordinate association information of the sample object on the depth training image is the same process as determining the coordinate association information of the object to be estimated on the depth image in step S102. That is, the training coordinate association information includes at least one of the following: determining the two-dimensional coordinates of the sample object projected onto the depth training image from the world coordinates of the plane where the depth training image is located; performing coordinate system transformation on the two-dimensional coordinates to determine the horizontal and vertical coordinates of the sample object in the camera coordinate system; and encoding the two-dimensional coordinates on the depth training image using a preset trigonometric function to obtain encoded information.

[0126] In step S302, the fusion data is subjected to feature extraction by using the to-be-trained basic subnetwork to obtain a sample fusion feature map.

[0127] In some embodiments, after the training coordinate association information is determined, the training coordinate association information is combined with the training image data, and the combined data is input into the to-be-trained six-dimensional pose estimation network. The to-be-trained six-dimensional pose estimation network at least includes a to-be-trained rotation and translation subnetwork and a to-be-trained basic subnetwork for predicting a region of interest.

[0128] After the training image data and the training coordinate association information are combined, they are input into the to-be-trained basic subnetwork. The to-be-trained basic subnetwork includes a backbone network, a detection subnetwork and a mask subnetwork. By using the basic subnetwork, object detection and instance segmentation of the training image data and the training coordinate association information are realized. Thus, the basic loss includes a detection loss and an instance segmentation loss. The training image data and the training coordinate association information are subjected to feature extraction by the backbone network to obtain a sample fusion feature map.

[0129] In step S303, the to-be-trained rotation and translation subnetwork is used to determine, based on the sample fusion feature map, predicted rotation information and predicted translation information of the sample object from a world coordinate system to a camera coordinate system.

[0130] In some embodiments, the to-be-trained basic subnetwork is used to determine a predicted region of interest in the sample fusion data, and the to-be-trained rotation and translation subnetwork is used to determine, based on the predicted region of interest, predicted rotation information and predicted translation information of the sample object from a world coordinate system to a camera coordinate system. After the region of interest is determined, the detection subnetwork is used to perform object detection on the region of interest to obtain a plurality of detection boxes and determine a loss of each detection box. The instance segmentation subnetwork is used to perform instance segmentation based on the detection boxes to obtain an instance segmentation loss.

[0131] The to-be-trained rotation and translation subnetwork is used to predict, based on the predicted region of interest of each size, predicted rotation information and predicted translation information of the sample object from a world coordinate system to a camera coordinate system. The predicted rotation information represents the predicted rotation of the sample object in the camera coordinate system, and the predicted translation information represents the spatial position of the predicted sample object in the camera coordinate system.

[0132] In step S304, a first loss is determined based on the predicted rotation information and the predicted translation information, and real rotation information and real translation information of the sample object.

[0133] In some embodiments, the loss of predicting the predicted rotation information and the predicted translation information of the sample object by the rotation translation sub-network to be trained is determined by the difference between the predicted rotation information and the predicted translation information and the real rotation information and the real translation information of the sample object, i.e., the first loss.

[0134] In some possible implementations, the first loss is determined by analyzing the difference between the predicted six-dimensional pose and the real six-dimensional pose, i.e., the above-mentioned step S304 can be implemented by the following steps S341-S343 (not shown in the figure):

[0135] In step S341, the predicted rotation information and the predicted translation information of any point on the sample object are fused to obtain a predicted six-dimensional pose.

[0136] Here, the any point on the sample object can be any pixel point on the sample object. The feature information of the point is first multiplied element by element with the predicted rotation information, and then the multiplication result is summed element by element with the predicted translation information to obtain the predicted six-dimensional pose.

[0137] In step S342, the real rotation information and the real translation information of the any point are fused to obtain a real six-dimensional pose.

[0138] Here, the feature information of the point is multiplied element by element with the real rotation information of the point, and then the multiplication result is summed element by element with the real translation information to obtain the real six-dimensional pose.

[0139] In step S343, the first loss is determined based on the predicted six-dimensional pose and the real six-dimensional pose.

[0140] Here, the difference between the predicted six-dimensional pose and the real six-dimensional pose of each point in the sample object is determined, and the average value of the differences of a plurality of points is determined as the first loss.

[0141] The above-mentioned steps S341-S343 can effectively determine the loss of predicting the rotation information and the translation information by the rotation translation sub-network by analyzing the difference between the predicted six-dimensional pose and the real six-dimensional pose of each sampling point in the object to be estimated.

[0142] In step S305, the network parameters of the six-dimensional pose estimation network to be trained are adjusted based on at least the first loss and the base loss of the predicted region of interest, so that the loss output by the adjusted six-dimensional pose estimation network satisfies the convergence condition.

[0143] Here, the total loss is determined based on the first loss and a base loss of the predicted region of interest determined by the to-be-trained base subnetwork; and the network parameters of the to-be-trained six-dimensional pose estimation network are adjusted through the total loss, so that the loss output by the adjusted six-dimensional pose estimation network satisfies a convergence condition. The convergence condition can be that the value of the loss output by the adjusted six-dimensional pose estimation network converges.

[0144] In the embodiments of the present application, in the training process of the to-be-trained six-dimensional pose network, the coordinate association information is added to the training image data including the color training image and the depth training image, so that the to-be-trained six-dimensional pose network can simultaneously extract features from the color training image and the depth training image; and the rotation information and the translation information of the sample object are predicted by adding the rotation translation subnetwork to the backbone network, which simplifies the network structure. Thus, an end-to-end training is realized by using one convolutional neural network, and a good balance between accuracy and speed is achieved.

[0145] In some embodiments, the six-dimensional pose is estimated by inputting the sample fusion data into the same to-be-trained six-dimensional pose estimation network, which includes the to-be-trained base subnetwork and the to-be-trained rotation translation subnetwork. The step S303 can be implemented by the following steps S331 and S332 (not shown in the figure):

[0146] In step S331, the to-be-trained rotation translation subnetwork is used to detect the sample object based on the sample fusion feature map, and at least one detection box of the sample object is determined.

[0147] In step S332, the to-be-trained rotation translation subnetwork is used to process each detection box to obtain the predicted rotation information and the predicted translation information of the sample object in each detection box.

[0148] The implementation process of the above steps S331 and S332 is similar to that of the above steps S222 and S223, that is, the above steps S331 and S332 can be implemented by the implementation process of the above steps S222 and S223. Thus, in the training process, the base subnetwork and the rotation translation subnetwork are introduced, which can more accurately predict the rotation information and the translation information of the sample object in each detection box from the world coordinate system to the camera coordinate system, and further make the estimated six-dimensional pose of the sample object more accurate.

[0149] In some possible implementation manners, the predicted rotation information and the predicted translation information of the sample object are obtained by processing each detection box in the sample image data, that is, the above step S332 can be implemented by the following steps:

[0150] In a first step, the two connected first fully connected layers are used to perform feature space transformation on each detection box to obtain first transformed sample features.

[0151] In some embodiments, the rotation and translation sub-network to be trained includes two connected first fully connected layers, a second fully connected layer, and a third fully connected layer. The input and output dimensions of the two connected first fully connected layers are both N, and the output of a first fully connected layer is the input of a second first fully connected layer. The detection boxes in all regions of interest are input to a first fully connected layer of the two connected first fully connected layers, and the first fully connected layer is used to perform weighted summation on all detection boxes to convert each detection box from a low-dimensional feature space to a high-dimensional feature space, perform feature space transformation, and obtain a weighted summation result. The weighted summation result is input to a second first fully connected layer to perform weighted summation again to obtain first transformed sample features.

[0152] In a second step, the second fully connected layer and the third fully connected layer are used to perform feature space transformation on the first transformed sample features to obtain predicted rotation information and predicted translation information.

[0153] In this way, the two shared fully connected layers and the two independent fully connected layers in the rotation and translation sub-network to be trained are used to obtain rotation information and three-dimensional translation information, and the 6D pose of a sample object can be accurately estimated based on the rotation information and the three-dimensional translation information.

[0154] In some embodiments, to further improve the accuracy of the six-dimensional pose network to be trained, a sub-network for predicting the three-dimensional coordinates of an object, i.e., a world coordinate sub-network, is added to the six-dimensional pose network to be trained. The training process of the six-dimensional pose network to be trained further includes the following steps S307 to S309 (not shown in the figure):

[0155] In step S307, the world coordinate sub-network to be trained is used to determine predicted world coordinates of the sample object in the world coordinate system based on the predicted region of interest.

[0156] Here, the training image data and the training coordinate association information are input to the basic sub-network to be trained as input data, the input data is processed by the basic sub-network to obtain a region of interest output by the basic sub-network through feature pyramid processing, candidate box selection, and region of interest alignment. The region of interest is input to the world coordinate sub-network to be trained to predict the coordinates of the sample object in the world coordinate system to obtain predicted world coordinates.

[0157] In step S308, a second loss is determined based on the predicted world coordinates and the true world coordinates of the sample object.

[0158] Here, the difference between the predicted world coordinates and the real world coordinates of the sample object in each coordinate dimension is taken as the second loss.

[0159] In some possible implementations, the second loss is determined by comparing the predicted coordinates and the real coordinates in each dimension, i.e., the above step S308 can be implemented through the following steps S381 and S382 (not shown in the figure):

[0160] In step S381, the difference between the predicted world coordinates and the real world coordinates in each coordinate dimension is determined.

[0161] For example, the predicted world coordinates are three-dimensional (x, y, z), and then the difference in the horizontal axis, the vertical axis, and the Z axis is determined.

[0162] In step S382, the differences in multiple coordinate dimensions are fused to obtain the second loss.

[0163] Here, the differences in multiple coordinate dimensions are multiplied by weights and then element-wise summed to obtain the second loss. In this way, the second loss can be more accurate by determining the difference between the predicted world coordinates and the real world coordinates in each coordinate dimension, and can play an auxiliary role in training the six-dimensional pose estimation network to be trained.

[0164] After the second loss is determined, the above step S305 can be implemented through the following step S309:

[0165] In step S309, the network parameters of the six-dimensional pose estimation network to be trained are adjusted based on the first loss, the second loss, and the base loss.

[0166] Here, the first loss, the second loss, and the base loss are multiplied by corresponding weights and then element-wise summed to obtain a total loss; for example, first, different adjustment parameters are used to adjust the first loss, the second loss, and the base loss respectively to obtain an adjusted first loss, an adjusted second loss, and an adjusted base loss; then, the adjusted first loss, the adjusted second loss, and the adjusted base network loss are fused to obtain a total loss; finally, the network parameters of the six-dimensional pose estimation network to be trained are adjusted based on the total loss.

[0167] By introducing the sub-network of three-dimensional coordinates into the trained six-dimensional pose network through the above steps S307 to S309 to assist the network in predicting the three-dimensional coordinates of the object, the efficiency of training the six-dimensional pose network can be improved, and the robustness of the trained six-dimensional pose network is stronger.

[0168] In some possible implementation manners, the loss of the detection sub-network and the mask sub-network, i.e., the base loss, can be determined through the following process:

[0169] In a first step, a third loss of object detection on the sample object in the predicted region of interest is determined by using the detection sub-network in the base sub-network to be trained.

[0170] Here, the predicted region of interest of the fusion data determined by the backbone network in the base sub-network to be trained is input into the detection sub-network. The detection sub-network performs object detection on each region of interest, outputs a detection box, and determines the difference between each detection box and the real detection box of the sample object, to obtain the third loss.

[0171] In a second step, a fourth loss of instance segmentation on the sample object is determined by using the mask sub-network in the base sub-network to be trained based on the predicted region of interest.

[0172] Here, the predicted region of interest of the fusion data determined by the backbone network in the base sub-network to be trained is input into the mask sub-network. The mask sub-network performs instance segmentation on the sample object in the fusion data based on the input region of interest, and determines the fourth loss based on the predicted result of the instance segmentation and the real segmentation result of the sample object.

[0173] In a third step, the base loss is obtained based on the third loss and the fourth loss.

[0174] Here, the base loss is obtained by assigning different weights to the third loss and the fourth loss, respectively.

[0175] By the first step to the third step, the third loss of object detection and the fourth loss of instance segmentation are taken as the base loss of the base sub-network to be trained, so that the object detection and the instance segmentation can be fully considered in the process of training the six-dimensional pose estimation network to be trained, thereby improving the performance of the base sub-network in the trained six-dimensional pose estimation network.

[0176] In the following, an exemplary application of the embodiment of the present application in an actual application scenario will be described by taking the six-dimensional pose estimation using the same neural network to process color images and depth images as an example.

[0177] The embodiment of the application provides a six-dimensional pose estimation method, first, coordinate information of an image is added to neural network input. Then, an object displacement and rotation subnetwork (such as an RT subnetwork) and an object world coordinate subnetwork (such as an abc subnetwork) are added on the basis of Mask RCNN, and multi-task joint training is performed. Finally, on the basis of original loss, an abc loss and an RT loss are added, and a convolutional neural network is trained. The trained network is used for six-dimensional pose estimation of an input image, and the six-dimensional pose of an object in the image is obtained.

[0178] In some embodiments, six-dimensional pose estimation plays an important role in emerging applications, such as autonomous driving, intelligent robot grasping and augmented reality. An RGB-D sensor can provide surface texture and geometry of a physical three-dimensional world. In the related art, the heterogeneity of the two types of data is explained by using different backbone networks. The embodiment of the application finds that the root cause of being unable to use a single backbone to extract features is hidden in the 3D visual projection equation, as shown in formula (1):

[0179]

[0180] In formula (1), a point (a, b, c) on a three-dimensional object is described, first, rotated by a rotation matrix R ∈ SO3 and translated by T ∈ R 3 to a position (x, y, d) in the camera coordinate system; then, the point is projected to a pixel at (u, v) in the image plane with a camera intrinsic matrix K. The 3D structure of the visible point in the 3D model from the camera perspective is preserved by the depth value (d) of each pixel and its coordinates (u, v) in the depth image. This means that each pixel value in the single-channel depth image is linked to the built-in coordinates of the pixel. Generally, (u, v) can be used to preserve the 3D structure, and changing (u, v) will destroy the 3D projection equation. However, the pooling, cropping, division of the region of interest and the spatial transformation such as resizing, cropping and flipping in data augmentation in the convolutional operator change (u, v) in the CNN framework. As shown in Figure 4 , Figure 4 The projection schematic diagram of the color image and the depth image provided by the embodiment of the application is from Figure 4It can be seen that the projection failure caused by ROI transformation, including cropping, resizing and region of interest alignment (ROI-Align); wherein the dashed line 41 connects the 3D object 42 (such as the object to be estimated) and its projection ROI 43 in the depth image 44 in the two-dimensional image plane 45 (in which the horizontal axis is represented as u and the vertical axis is represented as v). Any pixel (d, u, v) in the ROI 43 and its corresponding point (a, b, c) on the 3D object 42 follow the projection equation. If ROI-Align modifies the intrinsic coordinates (u, v), the above formula (1) no longer holds, that is, the ROI 43 is moved to the upper left corner of the depth image 44.

[0181] After these spatial transformation processes, the depth image and the 3D object cannot satisfy the projection equation, that is, the "projection decomposition". Therefore, the CNN framework in the related art is difficult to process the RGB image and the depth image at the same time.

[0182] In the embodiments of the present application, a simple and effective method is proposed to solve the "projection decomposition problem": providing the UV data (corresponding to the coordinate association information in the above embodiments) together with the image data to the two-dimensional CNN. The image data and the UV data are combined as 3D data, and the depth value d is decoupled from its intrinsic coordinates in the image plane. In this way, for the pixel value (d, u, v) of each pixel, the projection equation still holds after spatial transformation. Therefore, the CNN framework can be used to extract features from the RGB-D image.

[0183] Formula (1) also shows that the network should map the visible points in the RGB-D image to the original coordinates of these visible points in the 3D object. In order to realize an accurate, real-time and practical network architecture, the embodiments of the present application propose an end-to-end six-dimensional pose estimation network, which uses a unified backbone network to extract features from the Mask Region-CNN (Mask R-CNN) based RGB-D image. The Mask R-CNN uses parallel multi-subnetworks to perform object detection and instance segmentation. On this basis, an RT subnetwork (corresponding to the rotation and translation subnetwork in the above embodiments) is added to directly predict the rotation matrix R and the translation vector T, and an abc subnetwork is used for mapping any point on the color image.

[0184] In some embodiments, object detection and instance segmentation are required in the process of implementing six-dimensional pose estimation. In the embodiments of the present application, Mask R-CNN is used as the basic network, and the six-dimensional pose estimation network adopts the basic network structure of Mask R-CNN. After solving the projection decomposition problem, the data features of two heterogeneous data can be extracted using one backbone network. At the same time, on the basis of Mask R-CNN, parallel regression tasks of RT subnetwork and abc subnetwork are added. The RT subnetwork is used to predict the six-dimensional pose, and the abc subnetwork is used to guide the backbone network to map the visible point to the 3D model corresponding to the visible point. As shown in Figure 5 Figure 5 As can be seen from the network architecture diagram of the six-dimensional pose estimation method provided by the embodiments of the present application, Figure 5 the input data 501 includes: RGB image, depth image, normal UV (corresponding to the first coordinate in the above embodiments), XY (i.e. the coordinate value calculated by inversely projecting each pixel's corresponding normal UV coordinate according to formula (1), corresponding to the second coordinate in the above embodiments), depth normal vector and encoding information (PE); first, these data are input into the residual network ResNet backbone network + FPN 502 for feature extraction, and a feature pyramid is determined based on the extracted features to obtain feature maps of multiple sizes; second, the RPN 503 is used to extract multiple candidate boxes in the feature map, and multiple regions of interest 504 are obtained by aligning the regions of interest of the extracted candidate boxes; finally, the bbox subnetwork 505 is used to perform object detection and classification based on the regions of interest 504 to obtain the output result 51; the mask subnetwork 506 is used to perform object segmentation based on the regions of interest 504 to obtain the output result 52; the RT subnetwork 507 is used to perform six-dimensional pose estimation based on the regions of interest 504 to obtain the output result 53, which includes the rotation matrix and translation matrix of the object; and the abc subnetwork 508 is used to predict the three-dimensional coordinates of the object based on the regions of interest 504 to obtain the output result 54.

[0185] To solve the projection decomposition problem, coordinate association information is added to the input RGB-D data, and the coordinate association information and the RGB-D data are jointly input into the unified backbone network. For the RGB-D data, the RGB image and the corresponding depth image are combined along the channel dimension.

[0186] For the second coordinate in the coordinate association information, the first coordinate can be encoded into the inversely projected second coordinate, i.e. XY (x, y), according to the given camera intrinsic matrix and the depth image through formula (1) for each pixel's corresponding first coordinate (u, v). Wherein, XY is also two channels, and the introduction method is also to splice the corresponding input in the channel, i.e. to connect XY with the RGB-D data along the channel dimension. ​

[0187] In general, the three forms of coordinate-related information have their own advantages. The first coordinate is more direct, the second coordinate XY hides internal reference information, and the encoding information can be added to other input channels to be added and fused with other channels, so that multiple coordinate-related information work together to achieve the best network performance. Using the three forms of coordinate-related information together can greatly alleviate the mapping relationship between the convolution damage (d, u, v) and the coordinates (a, b, c).

[0188] In the embodiments of the present application, RT subnetwork and abc subnetwork are used to predict the world coordinate points of R matrix, T matrix and 3D model, where R is a rotation matrix in quaternion form, T ∈ R 3 is a translation matrix. These subnetworks are added to Mask R-CNN as two new parallel branch networks and take the features of all candidate boxes extracted by the region of interest alignment as input. As shown in Figure 6 Figure 6 The architecture diagram of the subnetwork provided by the embodiments of the present application is shown in FIG. 6. As can be seen from FIG. 6, the ROI 60 of 14x14x256 is input into the RT subnetwork 601; the RT subnetwork 601 has a similar structure to the bounding box (bbox) subnetwork in Mask R-CNN, including two shared layers and two independent fully connected layers (for example, FC 61-FC 64, where the input and output dimensions of FC 61 and FC 62 are both N, the input dimension of FC 63 is N, and the output dimension is 4, the input dimension of FC 64 is N, and the output dimension is 3) to obtain the R matrix 602 of 1x4 and the T matrix 603 of 1x3. The ROI 60 of 14x14x256 is input into the abc subnetwork 604; the abc subnetwork 604 and the mask subnetwork are similar, which is a Fully Convolutional Network (FCN) structure, including 4 convolutional layers (for example, convolutional layers 65-68) of 3x3 and 1 convolutional layer 67 of 1x1 to output the three-dimensional coordinate points 69 (abc) of 14x14x3.

[0189] In addition to the classification, detection and segmentation loss functions in the original Mask R-CNN, there are two new loss functions of the RT subnetwork and the abc subnetwork. The formula of the RT head loss L rt is shown in formula (2):

[0190]

[0191] ​Where O is the set of keypoints sampled from the 3D model of the object, m represents the number of vertices in the keypoint set, x represents any point in the keypoint set, R is the predicted rotation matrix, and T is the predicted translation vector. R* is the true rotation matrix, and T* is the true translation vector. The loss of the abc subnetwork is shown in Equation (3):

[0192] L abc =|aa*|+|bb*|+|cc*| (3);

[0193] Where (a, b, c) represent the coordinates of a pixel. Therefore, the overall loss function is:

[0194] L=λ0+λ1L abc +λ2L mask +λ3(L bbox +L cls )+λ4L rpn (4);

[0195] Where λ0, λ1, λ2, λ3, λ4 represent the weights of each loss.

[0196] In this embodiment, no additional time-consuming post-processing is used; instead, the estimation results are output directly from the RT network. Thus, by directly outputting the estimation results from the RT network to achieve pose estimation, the inference efficiency and practicality of the network architecture are significantly improved.

[0197] In this embodiment, a lightweight ABC subnetwork is added to the 3D coordinate regression task to introduce auxiliary loss for the auxiliary model training. During inference, the ABC subnetwork in the 6D pose estimation network is removed, and the 6D pose estimation result is obtained directly from the RT subnetwork. This significantly improves inference efficiency and reduces the difficulty of engineering implementation. Figure 7 As shown, Figure 7 This diagram illustrates an application scenario of the six-dimensional pose estimation method provided in this embodiment. Images 701 to 705 above show ground truth images where objects to be estimated are occluded. Images 711 to 715 below show the six-dimensional poses of the objects to be estimated obtained using the six-dimensional pose estimation network provided in this embodiment. Specifically, six-dimensional pose 711 is obtained by estimating the pose of image 701, six-dimensional pose 712 is obtained by estimating the pose of image 702, six-dimensional pose 713 is obtained by estimating the pose of image 703, six-dimensional pose 714 is obtained by estimating the pose of image 704, and six-dimensional pose 715 is obtained by estimating the pose of image 705. As can be seen from six-dimensional poses 711 to 715, even if objects in the input image are occluded to varying degrees, the six-dimensional poses of each object in the image can still be effectively estimated.

[0198] In the embodiments of the present application, the "projection decomposition" problem hidden in the CNN-based deep image processing is disclosed, and the projection failure is repaired by explicitly providing UV data together with depth data to the backbone network. Therefore, only one general two-dimensional CNN backbone network is needed to extract features from the RGB-D image. Based on this, an end-to-end six-dimensional pose estimation framework based on the mask R-CNN framework is proposed.

[0199] In this network framework, ResNet and FPN are used as the backbone network to extract features from the RGB-D image, and feature fusion is no longer needed. The six-dimensional pose estimation result is obtained from the RT subnetwork, which has a simple inference process and does not require time-consuming post-processing. Moreover, the abc subnetwork is added to the network framework as an auxiliary task of the training network. In this way, the time efficiency is improved while better performance and better robustness are achieved.

[0200] The embodiments of the present application provide a six-dimensional pose estimation device, Figure 8 The structural composition diagram of the six-dimensional pose estimation device provided by the embodiments of the present application is shown in Figure 8 As shown in the figure, the six-dimensional pose estimation device 800 comprises:

[0201] The first acquisition module 801 is configured to acquire image data comprising a to-be-estimated object;

[0202] The first determination module 802 is configured to determine coordinate association information of the to-be-estimated object on a depth image in the image data;

[0203] The second determination module 803 is configured to determine rotation information and translation information of the to-be-estimated object from a world coordinate system to a camera coordinate system based on the image data and the coordinate association information;

[0204] The third determination module 804 is configured to determine a six-dimensional pose of the to-be-estimated object in the camera coordinate system based on the rotation information and the translation information.

[0205] In some embodiments, the second determination module 803 comprises:

[0206] The first fusion sub-module is configured to fuse the image data and the coordinate association information to obtain fusion data;

[0207] The first determination sub-module is configured to determine the rotation information and the translation information of the to-be-estimated object from the world coordinate system to the camera coordinate system based on the fusion data.

[0208] In some embodiments, the first determination sub-module comprises:

[0209] a first extraction unit, configured to perform feature extraction on the fusion data to obtain a fusion feature map;

[0210] a first determination unit, configured to determine a region of interest in the image data based on the fusion feature map;

[0211] a second determination unit, configured to detect the to-be-estimated object based on the region of interest to determine at least one detection frame of the to-be-estimated object;

[0212] a first processing unit, configured to process each detection frame to obtain the rotation information and the translation information of the to-be-estimated object in each detection frame.

[0213] In some embodiments, the first processing unit comprises:

[0214] a first transformation subunit, configured to perform at least one feature space transformation on each detection frame to obtain first transformed features; wherein the output of the previous feature space transformation is the input of the next feature space transformation;

[0215] a second transformation subunit, configured to perform feature space transformation of different spatial dimensions on the first transformed features respectively to obtain the rotation information and the translation information.

[0216] In some embodiments, the first determination module 802 comprises:

[0217] a second determination sub-module, configured to determine a first coordinate of a world coordinate projection of the to-be-estimated object on the depth image in a rectangular coordinate system of a plane on which the depth image is located;

[0218] a third determination sub-module, configured to perform coordinate system conversion on the first coordinate to determine a second coordinate of the to-be-estimated object in the camera coordinate system;

[0219] a first encoding sub-module, configured to encode the first coordinate by using a preset trigonometric function to obtain encoding information;

[0220] a fourth determination sub-module, configured to determine at least one of the first coordinate, the second coordinate and the encoding information as the coordinate association information.

[0221] In some embodiments, the first fusion sub-module is further configured to:

[0222] in a case where the coordinate association information comprises at least one of the first coordinate and the second coordinate, splicing at least one of the first coordinate and the second coordinate with the image data to obtain the fusion data.

[0223] In some embodiments, the first fusion submodule is further configured to:

[0224] When the coordinate association information includes the encoding information, the image data and the encoding information are added element by element to obtain the fused data.

[0225] In some embodiments, the first acquisition module 801 includes:

[0226] The first acquisition submodule is used to acquire the color image and depth image of the object to be estimated;

[0227] The first stitching submodule is used to stitch the color image and the depth image together to obtain the image data.

[0228] This application provides a training device for the aforementioned six-dimensional pose estimation network. Figure 9 This is a schematic diagram of the structural composition of the training device for the six-dimensional pose estimation network provided in the embodiments of this application, as shown below. Figure 9 As shown, the training device 900 for the six-dimensional pose estimation network includes:

[0229] The second acquisition module 901 is used to acquire sample fusion data of a sample object with labeled six-dimensional pose; wherein, the sample fusion data includes training image data and training coordinate association information of the sample object;

[0230] The first feature extraction module 902 is used to extract features from the fused data using the base sub-network to be trained, and obtain a sample fusion feature map.

[0231] The first prediction module 903 is used to determine the predicted rotation information and predicted translation information of the sample object from the world coordinate system to the camera coordinate system based on the sample fusion feature map using the rotation and translation sub-network to be trained.

[0232] The fourth determining module 904 is used to determine the first loss based on the predicted rotation information and the predicted translation information, as well as the actual rotation information and actual translation information of the sample object;

[0233] The first adjustment module 905 is used to adjust the network parameters of the six-dimensional pose estimation network to be trained, based at least on the first loss and the basic loss of the predicted region of interest, so that the loss output by the adjusted six-dimensional pose estimation network satisfies the convergence condition.

[0234] In some embodiments, the second acquisition module 901 is further configured to:

[0235] The training image data and the training coordinate association information are fused to obtain the sample fusion data.

[0236] In some embodiments, the first prediction module 903 comprises:

[0237] A first detection submodule is configured to detect the sample object based on the sample fusion feature map by using the to-be-trained rotation and translation subnetwork, and determine at least one detection box of the sample object.

[0238] A first processing submodule is configured to process each detection box by using the to-be-trained rotation and translation subnetwork, and obtain the predicted rotation information and the predicted translation information of the sample object in each detection box.

[0239] In some embodiments, the to-be-trained rotation and translation subnetwork comprises two connected first full connection layers, a second full connection layer and a third full connection layer, and the first processing submodule comprises:

[0240] A first transformation unit is configured to perform feature space transformation on each detection box by using the two connected first full connection layers, and obtain first transformed sample features.

[0241] A second transformation unit is configured to perform feature space transformation on the first transformed sample features by using the second full connection layer and the third full connection layer respectively, and obtain the predicted rotation information and the predicted translation information.

[0242] In some embodiments, the to-be-trained six-dimensional pose estimation network further comprises a world coordinate subnetwork, and the apparatus further comprises:

[0243] A fifth determination module is configured to determine the predicted world coordinates of the sample object in the world coordinate system based on the predicted region of interest by using the to-be-trained world coordinate subnetwork.

[0244] A sixth determination module is configured to determine a second loss based on the predicted world coordinates and the real world coordinates of the sample object.

[0245] The first adjustment module 905 is further configured to:

[0246] Adjust the network parameters of the to-be-trained six-dimensional pose estimation network based on the first loss, the second loss and the basic loss.

[0247] In some embodiments, the fourth determination module 904 comprises:

[0248] a second fusion submodule configured to fuse the predicted rotation information and the predicted translation information of any point on the sample object to obtain a predicted six-dimensional pose;

[0249] a third fusion submodule configured to fuse the real rotation information and the real translation information of the any point to obtain a real six-dimensional pose;

[0250] a seventh determination submodule configured to determine the first loss based on the predicted six-dimensional pose and the real six-dimensional pose.

[0251] In some embodiments, the sixth determination module comprises:

[0252] an eighth determination submodule configured to determine a difference between the predicted world coordinate and the real world coordinate in each coordinate dimension;

[0253] a fourth fusion submodule configured to fuse the differences in multiple coordinate dimensions to obtain the second loss.

[0254] In some embodiments, the apparatus further comprises:

[0255] a seventh determination module configured to determine, by using a detection subnetwork in the to-be-trained basic subnetwork, a third loss of detecting the sample object in the predicted region of interest;

[0256] an eighth determination module configured to determine, by using a mask subnetwork in the to-be-trained basic subnetwork, a fourth loss of instance segmentation of the sample object based on the predicted region of interest;

[0257] a ninth determination module configured to obtain the basic loss based on the third loss and the fourth loss.

[0258] It should be noted that the description of the above apparatus embodiments is similar to the description of the above method embodiments, and has similar beneficial effects to the method embodiments. For technical details not disclosed in the apparatus embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0259] It should be noted that, in the embodiments of the present application, if the training method of the posture estimation network is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a terminal, a server, etc.) to execute all or part of the methods described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, and various media that can store program codes. Thus, the embodiments of the present application are not limited to any specific hardware and software combination.

[0260] Correspondingly, the embodiments of the present application further provide a computer program product, which includes computer executable instructions, and the computer executable instructions can implement the steps in the six-dimensional posture estimation method provided by the embodiments of the present application after being executed.

[0261] Correspondingly, the embodiments of the present application further provide a computer storage medium, which stores computer executable instructions, and the computer executable instructions implement the steps in the six-dimensional posture estimation method provided by the embodiments of the present application when executed by a processor.

[0262] Correspondingly, the embodiments of the present application provide a computer device, Figure 10 The schematic diagram of the composition structure of the computer device in the embodiments of the present application is shown in FIG. 10. Figure 10 As shown in FIG. 10, the computer device 1000 includes a processor 1001, at least one communication bus, a communication interface 1002, at least one external communication interface and a memory 1003. The communication interface 1002 is configured to realize the connection and communication between the components. The communication interface 1002 can include a display screen, and the external communication interface can include a standard wired interface and a wireless interface. The processor 1001 is configured to execute the image processing program in the memory to implement the steps in the six-dimensional posture estimation method provided by the embodiments of the present application.

[0263] The above six-dimensional posture estimation device, computer device and storage medium embodiments are similar to the description of the above method embodiments, have similar technical descriptions and beneficial effects to the corresponding method embodiments, and are limited in length. The technical details not disclosed in the six-dimensional posture estimation device, computer device and storage medium embodiments of the present application can be understood by referring to the description of the method embodiments of the present application.

[0264] It should be understood that "an embodiment" or "one embodiment" or "an implementation" or "one implementation" as described throughout the specification means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one implementation of the application. Therefore, appearances of the phrases "in one embodiment" or "in an embodiment" or "in an implementation" or "in one implementation" in various places throughout the specification are not necessarily referring to the same embodiment. Further, the described features, structures, or characteristics can be combined in any suitable manner in one or more implementations. It should be understood that the sequence of steps in the above-described implementations can not necessarily be the sequence in which the steps are performed, unless specifically stated otherwise in the steps. The sequence of steps should be determined based on the function and logic of the steps, and the sequence of steps should not limit the implementations of the application. The sequence of the above-described implementations is only for description, and does not represent the advantages or disadvantages of the implementations. It should be noted that, in this document, the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusion, so that processes, methods, articles, or devices including a series of features are not limited to those features, but also include other features not explicitly listed, or inherent features of such processes, methods, articles, or devices. Without more limitations, the feature defined by the statement "comprising a" does not exclude the presence of other identical features in the process, method, article, or device including the feature.

[0265] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. The above-described device embodiments are only illustrative, for example, the division of the units is only a logical functional division, and actual implementation can have another division manner, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each component part shown or discussed can be through some interface, indirect coupling or communication connection between devices or units, which can be electrical, mechanical or other forms.

[0266] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; they can be located in one place or distributed on multiple network units; and some or all of the units can be selected to achieve the purposes of the embodiments of the present application according to actual needs. In addition, the functional units in each embodiment of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be realized in the form of hardware, or in the form of hardware plus software function unit. Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction-related hardware, and the above-mentioned program can be stored in a computer readable storage medium, and the program executes the steps including the above-mentioned method embodiments when executed; and the above-mentioned storage medium includes: mobile storage device, read only memory (Read Only Memory, ROM), magnetic disc or optical disc and various storage program codes.

[0267] Alternatively, the integrated units of the present application, if implemented in the form of software function modules and sold or used as independent products, can also be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application can be embodied in the form of software products, which are stored in a storage medium and include a number of instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the methods described in the embodiments of the present application. The above-mentioned storage medium includes: mobile storage device, ROM, magnetic disc or optical disc and various storage program codes. The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any skilled person in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method of six-dimensional pose estimation, the method comprising: The method comprises: acquiring a color image and a depth image of an object to be estimated; stitching the color image and the depth image to obtain a color-depth image; determining coordinate association information of the object to be estimated on the depth image in the color-depth image; the coordinate association information comprises at least two of the following: a first coordinate of a world coordinate of the object to be estimated projected onto the depth image, a second coordinate of the first coordinate in a camera coordinate determined by back-projection on a two-dimensional coordinate, and encoding information obtained by encoding the first coordinate by a trigonometric function; fusing the color-depth image and the coordinate association information to obtain fusion data; based on the fusion data, determining rotation information and translation information of the object to be estimated from a world coordinate system to a camera coordinate system; based on the rotation information and the translation information, determining a six-dimensional pose of the object to be estimated in the camera coordinate system.

2. The method of claim 1, wherein, The method comprises: feature extraction on the fusion data to obtain a fusion feature map; detecting the object to be estimated based on the fusion feature map to determine at least one detection box of the object to be estimated; processing each detection box to obtain the rotation information and the translation information of the object to be estimated in each detection box.

3. The method of claim 2, wherein, The method comprises: at least one feature space transformation on each detection box to obtain first transformed features; wherein the output of the last feature space transformation is the input of the next feature space transformation; respectively performing feature space transformations of different spatial dimensions on the first transformed features to obtain the rotation information and the translation information.

4. The method of claim 1, wherein, The method comprises: in the case that the coordinate association information comprises the first coordinate and the second coordinate, stitching the first coordinate and the second coordinate with the color-depth image to obtain the fusion data.

5. The method of claim 1, wherein, The method comprises: in the case that the coordinate association information comprises at least one of the first coordinate and the second coordinate and the encoding information, stitching at least one of the first coordinate and the second coordinate with the color-depth image to obtain a stitched color-depth image, and performing element-wise addition of the stitched color-depth image and the encoding information to obtain the fusion data.

6. A method for training a six-dimensional pose estimation network, the method comprising: The six-dimensional pose estimation network is obtained by training a to-be-trained six-dimensional pose estimation network, and the to-be-trained six-dimensional pose estimation network comprises at least a to-be-trained rotation and translation sub-network and a to-be-trained basic sub-network. The method comprises: obtain sample fusion data of a sample object with labeled six-dimensional pose; the sample fusion data is obtained by fusing a training color-depth image and training coordinate association information of the sample object; the training color-depth image is obtained by splicing a color image and a depth image of the sample object; the training coordinate association information includes at least two of the following: first coordinates of world coordinates of the sample object projected onto a depth image, second coordinates of the first coordinates in a camera coordinate system determined by back-projection of two-dimensional coordinates, and encoding information obtained by encoding the first coordinates by a trigonometric function; extract features of the fusion data by using the to-be-trained basic subnetwork to obtain sample fusion feature maps; determine predicted rotation information and predicted translation information of the sample object from a world coordinate system to a camera coordinate system based on the sample fusion feature maps by using the to-be-trained rotation and translation subnetwork; determine a first loss based on the predicted rotation information and the predicted translation information, and real rotation information and real translation information of the sample object; adjust network parameters of the to-be-trained six-dimensional pose estimation network based on at least the first loss and a basic loss of a predicted region of interest, so that a loss output by the adjusted six-dimensional pose estimation network meets a convergence condition.

7. The method of claim 6, wherein, The method further comprises: detect the sample object based on the sample fusion feature maps by using the to-be-trained rotation and translation subnetwork to determine at least one detection box of the sample object; process each detection box by using the to-be-trained rotation and translation subnetwork to obtain the predicted rotation information and the predicted translation information of the sample object in the each detection box.

8. The method of claim 7, wherein, The to-be-trained rotation and translation subnetwork comprises two connected first, second, and third fully connected layers, and the processing of each detection box by using the to-be-trained rotation and translation subnetwork to obtain the predicted rotation information and the predicted translation information of the sample object in the each detection box comprises: perform feature space transformation on the each detection box by using the two connected first fully connected layers to obtain first transformed sample features; perform feature space transformation on the first transformed sample features by using the second and third fully connected layers to obtain the predicted rotation information and the predicted translation information.

9. The method according to any one of claims 6 to 8, characterized in that, The to-be-trained six-dimensional pose estimation network further comprises a world coordinate subnetwork, and the method further comprises: determine predicted world coordinates of the sample object in the world coordinate system based on the predicted region of interest by using the to-be-trained world coordinate subnetwork; determine a second loss based on the predicted world coordinates and real world coordinates of the sample object; and determine a third loss based on the second loss and the first loss. The network parameters of the to-be-trained six-dimensional pose estimation network are adjusted based on the first loss and a base loss of the to-be-trained base subnetwork. The network parameters of the to-be-trained six-dimensional pose estimation network are adjusted based on the first loss, the second loss and the base loss.

10. The method according to any one of claims 6 to 8, characterized in that, The first loss is determined based on the predicted rotation information and the predicted translation information, and real rotation information and real translation information of the sample object. The predicted rotation information and the predicted translation information of any point on the sample object are fused to obtain a predicted six-dimensional pose. The real rotation information and the real translation information of the any point are fused to obtain a real six-dimensional pose. The first loss is determined based on the predicted six-dimensional pose and the real six-dimensional pose.

11. The method of claim 9, wherein, The second loss is determined based on the predicted world coordinates and real world coordinates of the sample object. The difference between the predicted world coordinates and the real world coordinates in each coordinate dimension is determined. The differences in multiple coordinate dimensions are fused to obtain the second loss.

12. The method according to any one of claims 6 to 8, characterized in that, After the predicted region of interest in the sample fusion data is determined by using the to-be-trained base subnetwork, the method further comprises: A third loss of detecting the sample object in the predicted region of interest is determined by using a detection subnetwork in the to-be-trained base subnetwork. A fourth loss of instance segmentation of the sample object is determined based on the predicted region of interest by using a mask subnetwork in the to-be-trained base subnetwork. The base loss is obtained based on the third loss and the fourth loss.

13. A six-dimensional pose estimation apparatus, characterized by The device comprises: A first acquisition module is configured to acquire a color image and a depth image of a to-be-estimated object, and to splice the color image and the depth image to obtain a color-depth image. A first determination module is configured to determine coordinate association information of the to-be-estimated object on a depth image in the color-depth image, wherein the coordinate association information comprises at least two of the following: a first coordinate of a world coordinate of the to-be-estimated object projected onto the depth image, a second coordinate of the first coordinate in a camera coordinate system determined by inverse projection on a two-dimensional coordinate, and encoding information of the first coordinate obtained by trigonometric function encoding. A second determination module is configured to fuse the color-depth image and the coordinate association information to obtain fusion data, and to determine rotation information and translation information of the to-be-estimated object from a world coordinate system to a camera coordinate system based on the fusion data. A third determination module is configured to determine a six-dimensional pose of the to-be-estimated object in the camera coordinate system based on the rotation information and the translation information.

14. An apparatus for training a six-dimensional pose estimation network, comprising: The six-dimensional pose estimation network is obtained by training a to-be-trained six-dimensional pose estimation network, and the to-be-trained six-dimensional pose estimation network comprises at least a to-be-trained rotation and translation subnetwork and a to-be-trained base subnetwork, and the device comprises: The second acquisition module is configured to acquire sample fusion data of a sample object with a labeled six-dimensional pose; the sample fusion data is obtained by fusing a training color-depth image and training coordinate-related information of the sample object; the training color-depth image is obtained by splicing a color image and a depth image of the sample object; the training coordinate-related information includes at least two of the following: first coordinates of world coordinates of the sample object projected onto a depth image, second coordinates of the first coordinates in a camera coordinate system determined by back-projection of two-dimensional coordinates, and encoding information of the first coordinates obtained by encoding the first coordinates through a trigonometric function; The first feature extraction module is configured to perform feature extraction on the fusion data by using the to-be-trained basic subnetwork to obtain sample fusion feature maps; The first prediction module is configured to determine predicted rotation information and predicted translation information of the sample object from a world coordinate system to a camera coordinate system based on the sample fusion feature maps by using the to-be-trained rotation and translation subnetwork; The fourth determination module is configured to determine a first loss based on the predicted rotation information and the predicted translation information, and real rotation information and real translation information of the sample object; The first adjustment module is configured to adjust network parameters of the to-be-trained six-dimensional pose estimation network based on at least the first loss and a basic loss of a predicted region of interest, so that a loss output by the adjusted six-dimensional pose estimation network satisfies a convergence condition.

15. A computer storage medium, comprising, The computer storage medium stores computer executable instructions, which, when executed, implement the steps of the six-dimensional pose estimation method of any one of claims 1 to 5, or which, when executed, implement the steps of the training method of the six-dimensional pose estimation network of any one of claims 6 to 12.

16. An electronic device, comprising: The electronic device includes a memory and a processor, and the memory stores computer executable instructions. When the processor executes the computer executable instructions on the memory, the steps of the six-dimensional pose estimation method of any one of claims 1 to 5 are implemented, or when the processor executes the computer executable instructions on the memory, the steps of the training method of the six-dimensional pose estimation network of any one of claims 6 to 12 are implemented.

Images