Pose optimization method, robot, storage medium, equipment and program product

Abstract

The invention provides a pose optimization method, a robot, a storage medium, equipment and a program product. The method comprises the following steps: receiving a target image containing a target object; performing edge detection on the target image to obtain actual edge information of the target object; performing minimization solution on the target pose optimization energy function to calculate the pose optimization offset of the target object; optimizing the initial pose of the target object by using the pose optimization offset to obtain an optimized pose of the target object; wherein the re-projection calculation item corresponding to the target pose optimization energy function is calculated by using a target pose optimization model, and the target pose optimization model is a model based on deep learning. According to the method, the calculation efficiency and accuracy of the target pose optimization energy function are improved by using the target pose optimization model, the operation is fast, and the precision is higher.

Description

Technical Field

[0001] The present application relates to the technical field of object posture optimization, and in particular to a posture optimization method, a robot, a storage medium, a device and a program product. Background Art

[0002] The pose estimation of an object refers to determining the position (translation) and orientation (rotation) of an object in three-dimensional space. Related object pose estimation methods usually rely on depth sensors or structured light scanners, which are expensive, have complex usage scenarios, and have strict requirements on environmental conditions. The pose estimation method based on deep learning does not need to rely on complex three-dimensional sensing equipment. However, the related methods have large amount of calculation and are inaccurate in the related calculation process of the pose optimization equation.

[0003] Based on this, the present application provides a posture optimization method, a robot, a storage medium, a device and a program product to improve the relevant technology. Summary of the invention

[0004] The purpose of this application is to provide a posture optimization method, a robot, a storage medium, a device and a program product, which are fast in calculation and more accurate.

[0005] The purpose of this application is achieved by the following technical solutions:

[0006] In a first aspect, the present application provides a posture optimization method, the method comprising:

[0007] receiving a target image including a target object;

[0008] Performing edge detection on the target image to obtain actual edge information of the target object;

[0009] Minimizing the target posture optimization energy function to calculate the optimized posture offset of the target object;

[0010] Optimizing the initial pose of the target object using the pose optimization offset to obtain an optimized pose of the target object;

[0011] Among them, the reprojection calculation items corresponding to the target pose optimization energy function are calculated using a target pose optimization model, and the target pose optimization model is a model based on deep learning. The input of the target pose optimization model includes the actual edge information and the reprojected edge information based on the initial pose.

[0012] In some embodiments, performing edge detection on the target image to obtain actual edge information of the target object includes:

[0013] The target image is subjected to edge detection using a deep learning-based target edge detection model to obtain actual edge information of the target object.

[0014] In some embodiments, the target edge detection model performs edge detection on the target image to obtain actual edge information of the target object, including:

[0015] Extracting image feature information corresponding to the target image;

[0016] The image feature information is decoded to obtain actual edge information of the target object.

[0017] In some embodiments, the object edge detection model includes an encoder and a decoder;

[0018] The encoder is used to extract image feature information corresponding to the target image;

[0019] The decoder is used to decode the image feature information to obtain actual edge information of the target object.

[0020] In some embodiments, a skip connection is used between the encoder and the decoder.

[0021] In some embodiments, the process of calculating the reprojection calculation item by the target pose optimization model includes:

[0022] Performing cross-attention encoding based on the actual edge information and the reprojected edge information to obtain cross-attention feature information;

[0023] The cross-attention feature information is fully connected to calculate the reprojection calculation item.

[0024] In some embodiments, performing cross-attention encoding based on the actual edge information and the reprojected edge information to obtain cross-attention feature information includes:

[0025] Performing position coding processing on the actual edge information to obtain actual position coding information; and performing position coding processing on the reprojected edge information to obtain reprojected position coding information;

[0026] The actual position encoding information and the reprojected position encoding information are used to perform cross-attention encoding to obtain cross-attention feature information.

[0027] In some embodiments, the performing cross-attention encoding using the actual position encoding information and the reprojected position encoding information to obtain cross-attention feature information includes:

[0028] Performing full connection processing on the actual position encoding information to obtain actual fully connected feature information; and performing full connection processing on the reprojected position encoding information to obtain reprojected fully connected feature information;

[0029] Cross-attention encoding is performed on the actual fully-connected feature information and the reprojected fully-connected feature information to obtain cross-attention feature information.

[0030] In some embodiments, the target image is an RGB image.

[0031] In some embodiments, the method is applicable to one or more application scenarios in robotics, virtual reality, mixed reality, and intelligent manufacturing.

[0032] In a second aspect, the present application provides a robot comprising a control module and an image sensor, wherein the control module is used to execute any one of the above methods to obtain an optimized posture of a target object; and the image sensor is used to acquire a target image containing the target object.

[0033] In a third aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements any of the above methods when executed by a processor.

[0034] In a fourth aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements any one of the above methods when executing the computer program.

[0035] In a fifth aspect, the present application provides a computer program product, wherein the computer program product comprises a computer program, and the computer program implements any one of the above methods when executed by a processor.

[0036] The present application provides a posture optimization method, a robot, a storage medium, a device and a program product, which receive a target image containing a target object, perform edge detection on the target image, and extract the actual edge information of the target object. Next, the posture optimization offset is calculated by minimizing the target posture optimization energy function. In the calculation process of the target posture optimization energy function, its corresponding reprojection calculation item is calculated using the target posture optimization model. The reprojection edge information of the target object is calculated based on the initial posture, and combined with the actual edge information, it is input into the target posture optimization model based on deep learning to calculate the reprojection calculation item. Finally, the initial posture of the target object is adjusted using the posture optimization offset to obtain the optimized posture, that is, the optimized posture. The present application uses a target posture optimization model based on deep learning to replace the part of the relevant posture optimization equation that has a large amount of calculation and is inaccurate, thereby improving the calculation efficiency and accuracy of the target posture optimization energy function, and the calculation is fast and more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The present application is further described below in conjunction with the accompanying drawings and specific implementation methods.

[0038] Figure 1 It is a flow chart of a posture optimization method provided in an embodiment of the present application.

[0039] Figure 2 It is a schematic diagram of the principle of a target edge detection model provided in an embodiment of the present application.

[0040] Figure 3 It is a schematic diagram of the principle of a target posture optimization model provided in an embodiment of the present application.

[0041] Figure 4 It is a structural block diagram of a robot provided in an embodiment of the present application.

[0042] Figure 5 It is a structural block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in the implementation mode of this application. Obviously, the described implementation mode is only a part of the implementation mode of this application, not all the implementation modes. Based on the implementation mode in this application, all other implementation modes obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0044] In the description of the embodiments of the present application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0045] With the rapid development of computer vision technology, object pose estimation has become an important research field. Object pose estimation refers to determining the rotation and translation of an object in three-dimensional space, which is applicable to many fields such as robot navigation, augmented reality, and autonomous vehicles. Related object pose estimation methods rely on depth sensors or structured light scanners to obtain three-dimensional information of objects, and then use methods such as template matching or point cloud registration to calculate the object pose. However, these methods have limitations such as high cost, complex equipment, and strict environmental requirements. In addition, the performance of depth sensors is often affected when dealing with fast-moving objects or in poor lighting conditions.

[0046] To improve these problems, related object pose estimation methods based on monocular RGB images use the color and texture information of the image to infer the spatial position and direction of the object. Although some progress has been made, related technologies still face challenges in accuracy and robustness when dealing with complex backgrounds, lighting changes, and object occlusion.

[0047] With the continuous development of deep learning, more and more methods have begun to use deep learning models. For example, the pose estimation algorithm based on deep learning takes a monocular RGB image as input information, and the output is a direct or indirect quantity of the pose. Some algorithms, such as GDR-Net, directly output the pose of the object, and some algorithms, such as CDPN, output the dense key point coordinates of the object, and then solve the pose of the object through the PnP algorithm. However, the accuracy and stability of the object pose calculated by these two algorithms (which can be used as an example of the initial pose) still need to be improved. Therefore, some pose optimization methods have emerged to perform secondary optimization on the initial pose estimated by the deep learning model. Pose optimization methods based on regional information, such as PWP3D and RBOT, use the color histogram information of the RGB image to segment the object, extract the foreground and background information, and combine the object model to perform secondary iterative optimization of the object pose according to the reprojected contour of the initial pose. However, this type of optimization method mainly relies on the color histogram of the object, which is only effective for few textures that are significantly different from the background, and the generalization is still insufficient.

[0048] In view of the shortcomings of related pose optimization methods, the applicant proposed an object pose optimization method based on RGB images, which uses a deep learning-based algorithm to extract the edge contour of the object and combines the deep learning model to calculate the optimization function of the object pose. While ensuring the speed and accuracy of object pose optimization, the generalization of pose optimization is ensured.

[0049] The implementation methods of the present application will be described in detail below.

[0050] See also Figure 1 , Figure 1 It is a flow chart of a posture optimization method provided in an embodiment of the present application.

[0051] In order to improve the relevant technology, an embodiment of the present application provides a posture optimization method, which includes steps S101 to S104.

[0052] Step S101: receiving a target image including a target object.

[0053] Step S102: performing edge detection on the target image to obtain actual edge information of the target object.

[0054] Step S103: Minimizing the target posture optimization energy function to calculate the optimized posture offset of the target object.

[0055] Step S104: optimizing the initial pose of the target object using the pose optimization offset to obtain an optimized pose of the target object.

[0056] Among them, the reprojection calculation items corresponding to the target pose optimization energy function are calculated using a target pose optimization model, and the target pose optimization model is a model based on deep learning. The input of the target pose optimization model includes the actual edge information and the reprojected edge information based on the initial pose.

[0057] In some embodiments, the reprojection calculation term corresponding to the target pose optimization energy function is, for example, obtained by derivation of the target pose optimization energy function with respect to the pose optimization offset.

[0058] In some embodiments, the method can be applied to one or more application scenarios in robots, virtual reality (VR), mixed reality (MR) and intelligent manufacturing. The robot can be, for example, a bipedal robot (also known as a humanoid robot, a humanoid robot), a quadrupedal robot, a wheeled robot or other bionic robot, which is not limited in the above embodiments.

[0059] The above embodiment does not limit the target object, and its shape can be, for example, a cylinder, a cone, a rotating body, a cross-section body, etc. Among them, the cylinder includes a cylinder and a prism, the cone includes a cone and a pyramid, the rotating body includes a cylinder, a frustum, a cone, a sphere, an ellipsoid, a spherical cap, a bow ring, a circular ring, a dike ring, a fan ring, a jujube-shaped body, etc., and the cross-section body includes a prism, a frustum, a truncated cylinder, a truncated prism, a truncated cone, a spherical cap, a spherical segment, etc. The above embodiment does not limit the size of the target object, and its size can be, for example, the order of millimeters, the order of centimeters, the order of decimeters, the order of meters, etc.

[0060] In some embodiments, the target image may be an RGB image. The above embodiments do not limit the target image, which may be, for example, an RGB image, an HSV image, etc. Among them, an RGB image is an image containing information of three channels: red, green, and blue. An HSV image is an image containing information of three types: hue, saturation, and brightness. The target image is image data containing the object to be analyzed (i.e., the target object), such as an RGB image captured by a camera or other imaging device.

[0061] The target pose optimization energy function is a mathematical function that can optimize the initial pose according to the error between the actual edge and the reprojected edge. By minimizing this function, the pose of the object can be optimized to make it closer to the real position. The pose optimization offset refers to the amount of adjustment to the initial pose during the optimization process. This offset corresponds to the difference between the initial pose and the optimization target. The reprojected edge information based on the initial pose refers to the edge information of the object projected on the image plane based on the three-dimensional model and the initial pose of the target object. The deep learning-based model refers to an algorithm model that uses a deep neural network for data processing. In other words, the input of the target pose optimization model includes the actual edge information and the reprojected edge information, and the output of the target pose optimization model includes the reprojection calculation item.

[0062] The posture optimization method of the above embodiment receives a target image containing a target object, first performs edge detection on the image to extract the actual edge information of the target object. Subsequently, the target posture optimization energy function is minimized and solved using a target posture optimization model based on deep learning, thereby calculating the optimized posture offset of the target object. The input of the target posture optimization model includes actual edge information and reprojected edge information based on the initial posture, and the output includes a reprojection calculation item. Finally, the initial posture of the target object is adjusted using the posture optimization offset to obtain the optimized posture of the target object. Compared with the related methods, the above embodiment uses a target posture optimization model based on deep learning to calculate the optimization function of the object posture (i.e., the target posture optimization energy function), while ensuring the speed and accuracy of the optimized posture of the target object, ensuring the generalization of posture optimization. By introducing a deep learning model for edge detection and / or posture optimization, it is possible to automatically learn and extract more discriminative image features, reduce the need for manual parameter adjustment, and is suitable for a variety of different application scenarios. In addition, this method can effectively deal with partial occlusion of objects and background interference problems, thereby improving the generalization ability and reliability of pose optimization.

[0063] Related object edge detection algorithms, for example, use the Sobel algorithm or the Canny algorithm, but these algorithms have low generalization and accuracy, and it is difficult to meet the accuracy requirements of the position optimization algorithm for edge detection. Therefore, the applicant adopts an edge detection algorithm based on deep learning. The input of the algorithm is, for example, an RGB image containing a target object, and the edge information of the target object is output. In order to distinguish it from the edge information obtained by reprojection, the edge information obtained by edge detection can be called actual edge information, and the edge information obtained by reprojection can be called reprojection edge information. As an example, the size of the input RGB image is (3×H×W), 3 represents the three RGB color channels of the RGB image, and H and W represent the height and width of the image. The actual edge information can be represented by an object edge image, and the size of the output object edge image is (1×H×W), 1 represents that the object edge image has a black and white color channel, and H and W represent the height and width of the image. In the object edge image, the pixel value of the image representing the object edge is 0 or 1, 0 means that there is no object edge at the pixel position, and 1 means that there is an object edge at the pixel position.

[0064] See also Figure 2 , Figure 2This is a schematic diagram of the principle of an object edge detection model provided by an embodiment of the present application. The object edge detection algorithm model (i.e., the object edge detection model) can, for example, adopt an encoder-decoder structure. The encoder compresses the input RGB image and extracts the image feature information. The decoder decodes the image feature information and outputs the detected object edge (i.e., the actual edge information). A jump connection can be used between the encoder and the decoder to retain image information of different scale features. Figure 2 As shown, the white cube in the input RGB image is the target object, and the output image is the edge of the target object.

[0065] When performing edge detection on the target image, a target edge detection model based on deep learning may be used, or other object edge detection algorithms may be used, which is not limited in the above embodiment. In some embodiments, performing edge detection on the target image to obtain actual edge information of the target object may include: performing edge detection on the target image using a target edge detection model based on deep learning to obtain actual edge information of the target object.

[0066] The above-mentioned embodiment adopts a target edge detection model based on deep learning, which can maintain a high detection accuracy in complex backgrounds, illumination changes, and partial occlusion of objects, compared with traditional edge detection methods. The actual edge information extracted by the target edge detection model is combined with the reprojected edge information for pose optimization, which further improves the accuracy and stability of pose calculation. This method not only enhances the robustness of pose estimation, but also effectively handles the pose optimization problem of objects in different application scenarios, and has strong generalization ability and applicability.

[0067] In some embodiments, the target edge detection model performs edge detection on the target image to obtain actual edge information of the target object, which may include: extracting image feature information corresponding to the target image; and decoding the image feature information to obtain actual edge information of the target object.

[0068] The above embodiments do not limit the structure of the target edge detection model. In some embodiments, the target edge detection model may include an encoder and a decoder; the encoder is used to extract image feature information corresponding to the target image; the decoder is used to decode the image feature information to obtain actual edge information of the target object.

[0069] The encoder is responsible for extracting image feature information from the input target image. As an example, the encoder can compress the image information layer by layer through a series of convolution operations to extract multi-scale and multi-level feature representations to better understand the structure and edge information of the image. The decoder decodes the image feature information extracted by the encoder and decodes it into the actual edge information of the target object.

[0070] The above embodiments do not limit the connection mode between the encoder and the decoder. In some embodiments, a jump connection can be used between the encoder and the decoder. The jump connection can introduce feature information of different levels in the decoding process. For example, the jump connection directly passes the high-resolution features extracted early in the encoder to the decoder, thereby avoiding the loss of this part of the feature information in the decoding process, and achieving higher detection accuracy and detail recovery capability.

[0071] In the above embodiment, the target edge detection model adopts an encoder-decoder structure, and retains the high-resolution features in the encoder through jump connections, further improving the edge detection accuracy. By introducing the encoder-decoder structure, more accurate edge features can be extracted from the target image. In particular, during the decoding process, by using jump connections, the intermediate layer feature information of the encoder is directly passed to the decoder, so that the decoder retains more high-resolution details when restoring the edge. This improvement can still accurately extract the edge information of the target object when processing complex scenes or backgrounds, improves the accuracy and robustness of edge detection, and thus further improves the accuracy of posture optimization. Therefore, this method introduces a target edge detection model based on deep learning to extract the edge of the target object, replacing the related posture optimization method that uses regional color histograms for edge extraction, which is more stable and more accurate.

[0072] In the pose optimization algorithm, the foreground and background of the object can be extracted to obtain the actual edge of the object, and then the reprojected edge of the object model on the image is calculated based on the initial pose, and the initial pose is optimized based on the error between the actual edge and the reprojected edge.

[0073] Based on this idea, the corresponding target pose optimization energy function can be shown as formula (1).

[0074]

[0075] In formula (1), Φ(x c) represents the distance between the actual edge of each edge point and the reprojected edge of the 3D model of the target object, I represents the target image, and the P(Φ|I) function represents the conditional probability of the distance between the actual edge and the reprojected edge according to the initial pose under the target image I. If the initial pose is accurate, the distance between the two should be 0, that is, the reprojected edge completely fits the actual edge. If the initial pose is inaccurate, the distance is larger. Ω represents the pixel area of ​​the target image I, x c Represents the surface point X in the 3D model of the target object M According to the projection of the initial pose P of the target object on the target image I (or the image coordinate system), it can be regarded as a reprojected edge point, which can be expressed by the following formula (2). c It can be expressed as (u, v), which represents the object contour point (or edge point) projected onto the image coordinate system. As an example, u =10, v =20, indicating that the pixel position corresponding to the 10th pixel point in the horizontal direction and the 20th pixel point in the vertical direction on the target image I is the projection position of an edge point of the target object.

[0076] In formula (2), λ(X c ) function is about X C function, λ represents the projection, X C is the three-dimensional position of the target object in the camera coordinate system, X C =(X, Y, Z). x , f y , p x , p y is the camera internal parameter, R and T represent the rotation and displacement of the pose, and X, Y, and Z are the surface point X in the three-dimensional model of the target object. M The three-dimensional position of c The pixel color at the pixel location, H e is a unit step function, which can be expressed by formula (3).

[0077] In formula (3), π represents the ratio of circumference to circumference, and b is a hyperparameter, for example, 0.5. f (S) represents the foreground probability, P b (S) represents the background probability, and both can be calculated using the color histogram method based on posterior probability.

[0078] According to the above, Φ(x c ) represents the distance between the actual edge of each edge point and the reprojected edge, which can be expressed by formula (4), where C (contour) represents the actual edge contour of the object, Ω f represents the foreground area (i.e., the object projection contour), Ω b represents the background area, and both can be represented by Pf (S) and P b (S) is calculated. c∈C, c is a point in the actual edge contour C, C is composed of a series of points, and formula (4) d is calculated as x c The shortest distance to contour C. As an example, traverse the points in contour C and find the minimum distance.

[0079]

[0080]

[0081] The pose optimization equation can be expressed by equation (5), where P r represents the optimized pose (i.e. the optimized pose result), P r For example, a 4×4 matrix can be expressed as [R r , T r ], R r represents the optimized rotation matrix, T r represents the optimized translation matrix, [Δθ r ] × Denotes Δθ r The antisymmetric matrix of is, for example, a 3×3 matrix. Therefore, the pose optimization offset can be expressed as θ=[θ r ,θ t ], where θ r represents the amount of rotation, θ t Indicates the amount of translation.

[0082]

[0083] In order to calculate the pose optimization offset θ, the target pose optimization energy function P(Φ|I) can be differentiated with respect to θ to obtain equation (6).

[0084] Formula (7) and Formula (8) are specific expressions of the two parts in Formula (6).

[0085]

[0086] The second part of formula (6) It can be quickly calculated by combining the existing parameters with a fixed formula (the fixed formula can be given in advance, for example), and the first part The calculation of is more complicated, and the results calculated by the relevant algorithm are not accurate enough, so The reprojection calculation item corresponding to the target pose optimization energy function can be calculated using a deep learning model (e.g., a target pose optimization model). The deep learning model has a powerful function fitting capability, and has obvious advantages in generalization and accuracy over related algorithms. Using a target pose optimization model based on deep learning to replace the parts of the related pose optimization equations that are computationally intensive and inaccurate is faster and more accurate. In some embodiments, The input consists of the reprojected edge x based on the initial pose c , and the foreground P f (S) and background P b (S). The foreground and background P f (S) and P b (S) The actual edge that can be extracted by the target edge detection model Find. Therefore The input can be replaced by x c and Targeted becomes calculate The input of the deep learning model (i.e., the target pose optimization model) is x c and

[0087] The target pose optimization model can be based on the Transformer architecture, with input x c and x c and By image pixel position Indicates. Edge position x c and It is represented by n pixel positions, where n is a positive integer. As an example, suppose x c The image consists of 300 pixels with an image resolution of (256, 256). Two adjacent points are located at (100, 100) and (103, 104). After normalization preprocessing, the positions of the two points become (0.391, 0.391) and (0.402, 0.406). At this time, the difference between the digital representations of the two points is very small, which is not conducive to the deep learning model (i.e., the target pose optimization model) to distinguish their positional relationship. Based on this, in x c and 2D position encoding can be performed before inputting the target pose optimization model. The encoding formula is shown in formula (9), where L is an integer. Assuming L = 10, the position dimension of a pixel changes from 2 to 40, and the position representation of adjacent boundary points is quite different, which is conducive to the deep learning model to distinguish the positions of different boundary points.

[0088]

[0089] Assume x c and It consists of 300 pixels with a dimension of (300, 2). After position encoding, its dimension becomes (300, 40). Then, the position-encoded x c and Input a multi-layer fully connected network (MLP) and get and The dimension is (300, h), where h is an integer. As an example, h can be set to 128. Then and Cross-attention encoding is performed to obtain F, as shown in equations (10) and (11). Finally, feature F is obtained through a multi-layer fully connected network (MLP)

[0090]

[0091] See also Figure 3 , Figure 3 is a schematic diagram of the principle of a target posture optimization model provided in an embodiment of the present application, which is used to calculate The target pose optimization model calculates After that, the optimized object pose (i.e., optimized pose) can be calculated according to equations (5) and (6).

[0092] In some embodiments, the target pose optimization model can be trained by reprojection error. As an example, the reprojection error is expressed as formula (12), where is the optimized pose, represents the rotation matrix of the optimized pose, Represents the translation matrix of the optimized pose. X i represents the position of each surface point of the 3D model of the target object, i is the serial number of the surface point, assuming that the surface of the 3D model of the target object is composed of n surface points, then X i represents the i-th surface point. gt , T gt ] is the true value of the pose. Among them, R gt is the rotation of the true value of the pose, for example, a 3×3 matrix, T gt is the translation of the true value of the pose, for example, a three-dimensional vector. During training, x c The object pose truth value [R gt , T gt ]After adding a small amount of random perturbation error, the three-dimensional model is projected according to the posture with added noise.

[0093]

[0094] The above embodiment does not limit the way in which the target pose optimization model calculates the reprojection calculation item. In some embodiments, the process of the target pose optimization model calculating the reprojection calculation item may include: performing cross-attention encoding based on the actual edge information and the reprojection edge information to obtain cross-attention feature information; performing full-connection processing on the cross-attention feature information to calculate the reprojection calculation item. As an example, the target pose optimization model may include an attention module and a fully connected layer (e.g., an output fully connected layer), the attention module is used to perform a cross-attention encoding operation, and the fully connected layer is used to perform a fully connected operation. The above embodiment uses the attention module to optimize the features of the edge position of the input model to improve the accuracy of pose optimization.

[0095] In order to facilitate the target pose optimization model to distinguish the positions of adjacent edge points, in some embodiments, the cross-attention encoding based on the actual edge information and the reprojected edge information to obtain cross-attention feature information may include: performing position encoding processing on the actual edge information to obtain actual position encoding information; and, performing position encoding processing on the reprojected edge information to obtain reprojection position encoding information; performing cross-attention encoding using the actual position encoding information and the reprojection position encoding information to obtain cross-attention feature information. As an example, the target pose optimization model may include a position encoding unit, which is used to perform position encoding operations. The above embodiment uses position encoding to perform position dimensionality upscaling on the edge position points of the input model, which is beneficial for the model to distinguish the positions of adjacent edge points.

[0096] Before position encoding, the edge information may be normalized. In some embodiments, the position encoding of the actual edge information to obtain actual position encoding information may include: normalizing the actual edge information to obtain actual normalized information; position encoding the actual normalized information to obtain actual position encoding information. In some embodiments, the position encoding of the reprojected edge information to obtain reprojected position encoding information may include: normalizing the reprojected edge information to obtain reprojected normalized information; position encoding the reprojected normalized information to obtain reprojected position encoding information. As an example, the target pose optimization model may include a normalization unit, which is used to perform a normalization operation.

[0097] In some embodiments, the cross-attention encoding using the actual position encoding information and the reprojection position encoding information to obtain cross-attention feature information may include: performing full-connection processing on the actual position encoding information to obtain actual fully-connected feature information; and performing full-connection processing on the reprojection position encoding information to obtain reprojection fully-connected feature information; performing cross-attention encoding on the actual fully-connected feature information and the reprojection fully-connected feature information to obtain cross-attention feature information.

[0098] The present application also provides a posture optimization device, which includes an image receiving module, an edge detection module, a solution module and an optimization module. The image receiving module is used to receive a target image containing a target object. The edge detection module is used to perform edge detection on the target image to obtain the actual edge information of the target object. The solution module is used to minimize the target posture optimization energy function to calculate the posture optimization offset of the target object. The optimization module is used to optimize the initial posture of the target object using the posture optimization offset to obtain the optimized posture of the target object. Among them, the reprojection calculation item corresponding to the target posture optimization energy function is calculated using a target posture optimization model, and the target posture optimization model is a model based on deep learning. The input of the target posture optimization model includes the actual edge information and the reprojected edge information based on the initial posture.

[0099] In some embodiments, the edge detection module can be used to: perform edge detection on the target image using a deep learning-based target edge detection model to obtain actual edge information of the target object.

[0100] In some embodiments, the target edge detection model performs edge detection on the target image to obtain actual edge information of the target object, which may include: extracting image feature information corresponding to the target image; and decoding the image feature information to obtain actual edge information of the target object.

[0101] In some embodiments, the target edge detection model may include an encoder and a decoder; the encoder is used to extract image feature information corresponding to the target image; and the decoder is used to decode the image feature information to obtain actual edge information of the target object.

[0102] In some embodiments, a skip connection may be used between the encoder and the decoder.

[0103] In some embodiments, the process of calculating the reprojection calculation item by the target pose optimization model may include: performing cross-attention encoding based on the actual edge information and the reprojection edge information to obtain cross-attention feature information; performing full connection processing on the cross-attention feature information to calculate the reprojection calculation item.

[0104] In some embodiments, the target pose optimization model can perform cross-attention encoding based on the actual edge information and the reprojected edge information in the following manner to obtain cross-attention feature information: performing position encoding processing on the actual edge information to obtain actual position encoding information; and, performing position encoding processing on the reprojected edge information to obtain reprojection position encoding information; performing cross-attention encoding using the actual position encoding information and the reprojection position encoding information to obtain cross-attention feature information.

[0105] In some embodiments, the target pose optimization model can use the actual position encoding information and the reprojection position encoding information to perform cross-attention encoding to obtain cross-attention feature information in the following manner: fully connect the actual position encoding information to obtain actual fully connected feature information; and fully connect the reprojection position encoding information to obtain reprojection fully connected feature information; cross-attention encode the actual fully connected feature information and the reprojection fully connected feature information to obtain cross-attention feature information.

[0106] In some embodiments, the target image may be an RGB image.

[0107] In some embodiments, the device may be applicable to one or more application scenarios in robotics, virtual reality, mixed reality, and intelligent manufacturing.

[0108] See also Figure 4 , Figure 4 It is a structural block diagram of a robot provided in an embodiment of the present application.

[0109] An embodiment of the present application also provides a robot, comprising a control module and an image sensor, wherein the control module is used to execute any one of the above methods to obtain an optimized posture of a target object; and the image sensor is used to obtain a target image containing the target object.

[0110] In some embodiments, the robot may further include a laser sensor. The laser sensor may include a 2D laser sensor and / or a 3D laser sensor.

[0111] In some embodiments, the robot may also include an odometer and / or an IMU.

[0112] In some embodiments, the robot may further include one or more of an angle encoder, a torque sensor, and a PIR sensor.

[0113] In some embodiments, the robot may be a multi-jointed robot.

[0114] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above methods is implemented.

[0115] An embodiment of the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, any of the above methods is implemented.

[0116] The computer program product may be a portable compact disk read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the computer program product of the present application is not limited thereto, and the computer program product may be any combination of one or more computer readable media.

[0117] An embodiment of the present application further provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements any of the above methods when executing the computer program.

[0118] See also Figure 5 , Figure 5 It is a structural block diagram of a computer device provided in an embodiment of the present application.

[0119] The embodiments of the present application do not limit the computer device, which may be, for example, a local computer device, a cloud computer device, a distributed computer device, etc.

[0120] The computer device may include: a memory 110, a processor 120, and a communication interface 130. The memory 110, the processor 120, and the communication interface 130 are connected via an internal connection path.

[0121] The memory 110 is used to store computer programs. In some implementations, the computer programs may include codes for implementing the methods of the embodiments of the present application.

[0122] The processor 120 is used to execute the computer program stored in the memory 110 to control the communication interface 130 to receive input data and information and output data such as operation results. In some implementations, when the solution of the embodiment of the present application is implemented by software or firmware, the computer program for implementing the solution of the embodiment of the present application can be stored in the processor 120 and executed by the processor 120.

[0123] The memory 110 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (Programmable ROM, PROM), an erasable programmable read-only memory (Erasable PROM, EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory may be a random access memory (RAM). It should be noted that the memory 110 described herein is intended to include, but is not limited to, any memory of these and other suitable types. As an example, the memory 110 includes a random access memory (RAM), a cache memory and a read-only memory (ROM). Among them, the memory 110 stores a computer program, and the computer program can be executed by the processor 120 so that the processor 120 implements the steps of any of the above methods.

[0124] The processor 120 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor 120 may be any conventional processor, etc.

[0125] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 120 or an instruction in the form of software. The method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware processor for execution, or a combination of hardware and software modules in the processor 120 for execution. The software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 110, and the processor 120 reads the information in the memory 110 and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it is not described in detail here.

[0126] In some implementations, in addition to the hardware units described above, the computer device may also include software modules, where the software modules may be, for example, an operating system, a basic input output system (BIOS), application software, etc.

[0127] The operating system is used to manage the hardware and / or software resources of computer devices and is the core and cornerstone of computer devices. The operating system needs to handle basic tasks such as managing and configuring memory, determining the priority of system resource supply and demand, controlling input and output devices, operating networks, and managing file systems. In order to facilitate user operations, most operating systems will provide an operating interface for users to interact with the system.

[0128] BIOS is used to run hardware initialization during the power-on boot phase and provide runtime services for the operating system and application programs. In some implementations, BIOS can also monitor and display processor temperature and execute functions such as adjusting temperature protection strategies.

[0129] Application software is also called application program, which can be understood as software written for a specific application purpose of the user. It is one of the main categories of computer software. For example, application software can be a program used to achieve power control, temperature management, etc.

[0130] It should be noted that although some embodiments of the present application take mobile robots as examples, the present application can be applied to other bionic robots, such as AGVs, drones, etc., and the present application is not limited to this.

[0131] It should be understood that the specific examples in this specification are only intended to help those skilled in the art to better understand the implementation methods of the present application, rather than to limit the scope of protection of the present application.

[0132] It is understood that in the various implementations of this specification, the size of the sequence number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application.

[0133] It can be understood that the various implementation modes described in this specification can be implemented individually or in combination, and this application is not limited to this.

[0134] Unless otherwise specified, all technical and scientific terms used in this specification have the same meaning as those generally understood by those skilled in the art of the technical field of this specification. The terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit the scope of this specification. The term "and / or" used in this specification includes any and all combinations of one or more related listed items. The singular forms of "a", "above", and "the" used in this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0135] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this specification.

[0136] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of each embodiment described above can refer to the corresponding process in other embodiments and will not be repeated here.

[0137] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0138] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the technical solution of the present application.

[0139] In addition, each functional unit in each embodiment of the present specification may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0140] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of this specification. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0141] The above are only specific implementations of this specification, but the protection scope of this application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in this specification, which should be included in the protection scope of this specification. Therefore, the protection scope of this application should be based on the protection scope of the claims.

Claims

1. A posture optimization method, characterized in that: The method comprises: receiving a target image including a target object; Performing edge detection on the target image to obtain actual edge information of the target object; Minimizing the target posture optimization energy function to calculate the optimized posture offset of the target object; Optimizing the initial pose of the target object using the pose optimization offset to obtain an optimized pose of the target object; Among them, the reprojection calculation items corresponding to the target pose optimization energy function are calculated using a target pose optimization model, and the target pose optimization model is a model based on deep learning. The input of the target pose optimization model includes the actual edge information and the reprojected edge information based on the initial pose.

2. The posture optimization method according to claim 1, characterized in that: The performing edge detection on the target image to obtain actual edge information of the target object includes: The target image is subjected to edge detection using a deep learning-based target edge detection model to obtain actual edge information of the target object.

3. The posture optimization method according to claim 2, characterized in that: The target edge detection model performs edge detection on the target image to obtain actual edge information of the target object, including: Extracting image feature information corresponding to the target image; The image feature information is decoded to obtain actual edge information of the target object.

4. The posture optimization method according to claim 3, characterized in that: The target edge detection model includes an encoder and a decoder; The encoder is used to extract image feature information corresponding to the target image; The decoder is used to decode the image feature information to obtain actual edge information of the target object.

5. The posture optimization method according to claim 1, characterized in that: The process of calculating the reprojection calculation item by the target pose optimization model includes: Performing cross-attention encoding based on the actual edge information and the reprojected edge information to obtain cross-attention feature information; The cross-attention feature information is fully connected to calculate the reprojection calculation item.

6. The posture optimization method according to claim 5, characterized in that: The cross-attention encoding is performed based on the actual edge information and the reprojected edge information to obtain cross-attention feature information, including: Performing position coding processing on the actual edge information to obtain actual position coding information; and performing position coding processing on the reprojected edge information to obtain reprojected position coding information; The actual position encoding information and the reprojected position encoding information are used to perform cross-attention encoding to obtain cross-attention feature information.

7. The posture optimization method according to claim 6, characterized in that: The cross-attention encoding is performed using the actual position encoding information and the re-projected position encoding information to obtain cross-attention feature information, including: Performing full connection processing on the actual position encoding information to obtain actual fully connected feature information; and performing full connection processing on the reprojected position encoding information to obtain reprojected fully connected feature information; Cross-attention encoding is performed on the actual fully-connected feature information and the reprojected fully-connected feature information to obtain cross-attention feature information.

8. The posture optimization method according to claim 1, characterized in that: The target image is an RGB image.

9. The posture optimization method according to claim 1, characterized in that: The method is applicable to one or more application scenarios in robotics, virtual reality, mixed reality and intelligent manufacturing.

10. A robot, characterized in that: The robot comprises a control module and an image sensor, wherein the control module is used to execute the method according to any one of claims 1 to 8 to obtain an optimized posture of a target object; and the image sensor is used to obtain a target image containing the target object.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

12. A computer device, characterized in that: The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 9 when executing the computer program.

13. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images