Posture optimization method, robot, storage medium, device and program product

Abstract

The present application provides a posture optimization method, robot, storage medium, device, and program product, the method comprising: receiving a target image containing a target object; performing edge detection on the target image to obtain actual edge information of the target object; minimizing a target posture optimization energy function to calculate a posture optimization offset for the target object; optimizing the initial posture of the target object using the posture optimization offset to obtain an optimized posture of the target object; wherein the corresponding reprojection calculation term of the target posture optimization energy function is calculated using a target posture optimization model, and the target posture optimization model is a deep learning-based model. The present application uses the target posture optimization model to improve the computational efficiency and accuracy of the target posture optimization energy function, resulting in faster calculations and higher accuracy.

Description

Technical Field

[0001] The present application relates to the technical field of object posture optimization, and in particular to a posture optimization method, a robot, a storage medium, a device, and a program product. Background Art

[0002] Estimating an object's pose refers to determining its position (translation) and orientation (rotation) in three-dimensional space. Related object pose estimation methods typically rely on depth sensors or structured light scanners, which are expensive, complex to use, and have strict environmental requirements. Deep learning-based pose estimation methods do not rely on complex three-dimensional sensing equipment. However, these methods are computationally intensive and inaccurate during the calculation of the pose optimization equations.

[0003] Based on this, the present application provides a posture optimization method, a robot, a storage medium, a device and a program product to improve the relevant technology. Summary of the Invention

[0004] The purpose of this application is to provide a posture optimization method, robot, storage medium, device and program product with fast calculation and higher accuracy.

[0005] The purpose of this application is achieved by the following technical solutions:

[0006] In a first aspect, the present application provides a posture optimization method, the method comprising:

[0007] receiving a target image containing a target object;

[0008] Performing edge detection on the target image to obtain actual edge information of the target object;

[0009] Minimizing the target pose optimization energy function to calculate the pose optimization offset of the target object;

[0010] Optimizing the initial pose of the target object using the pose optimization offset to obtain an optimized pose of the target object;

[0011] Among them, the reprojection calculation item corresponding to the target pose optimization energy function is calculated using a target pose optimization model, and the target pose optimization model is a model based on deep learning. The input of the target pose optimization model includes the actual edge information and the reprojected edge information based on the initial pose.

[0012] In some embodiments, performing edge detection on the target image to obtain actual edge information of the target object includes:

[0013] The target image is subjected to edge detection using a deep learning-based target edge detection model to obtain actual edge information of the target object.

[0014] In some embodiments, the target edge detection model performs edge detection on the target image to obtain actual edge information of the target object, including:

[0015] Extracting image feature information corresponding to the target image;

[0016] The image feature information is decoded to obtain actual edge information of the target object.

[0017] In some embodiments, the object edge detection model includes an encoder and a decoder;

[0018] The encoder is used to extract image feature information corresponding to the target image;

[0019] The decoder is used to decode the image feature information to obtain actual edge information of the target object.

[0020] In some embodiments, a skip connection is used between the encoder and the decoder.

[0021] In some embodiments, the process of calculating the reprojection calculation term by the target pose optimization model includes:

[0022] Performing cross-attention encoding based on the actual edge information and the reprojected edge information to obtain cross-attention feature information;

[0023] The cross-attention feature information is fully connected to calculate the reprojection calculation item.

[0024] In some embodiments, performing cross-attention encoding based on the actual edge information and the reprojected edge information to obtain cross-attention feature information includes:

[0025] Performing position coding processing on the actual edge information to obtain actual position coding information; and performing position coding processing on the reprojected edge information to obtain reprojected position coding information;

[0026] Cross-attention encoding is performed using the actual position encoding information and the re-projected position encoding information to obtain cross-attention feature information.

[0027] In some embodiments, performing cross-attention encoding using the actual position encoding information and the reprojected position encoding information to obtain cross-attention feature information includes:

[0028] Performing full connection processing on the actual position encoding information to obtain actual fully connected feature information; and performing full connection processing on the reprojected position encoding information to obtain reprojected fully connected feature information;

[0029] Cross-attention encoding is performed on the actual fully-connected feature information and the reprojected fully-connected feature information to obtain cross-attention feature information.

[0030] In some embodiments, the target image is an RGB image.

[0031] In some embodiments, the method is applicable to one or more application scenarios in robotics, virtual reality, mixed reality, and intelligent manufacturing.

[0032] In a second aspect, the present application provides a robot comprising a control module and an image sensor, wherein the control module is used to execute any one of the above methods to obtain an optimized posture of a target object; and the image sensor is used to obtain a target image containing the target object.

[0033] In a third aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above methods is implemented.

[0034] In a fourth aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements any one of the above methods when executing the computer program.

[0035] In a fifth aspect, the present application provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, implements any of the above methods.

[0036] The present application provides a posture optimization method, a robot, a storage medium, a device and a program product, which receive a target image containing a target object, perform edge detection on the target image, and extract the actual edge information of the target object. Next, the posture optimization offset is calculated by minimizing the target posture optimization energy function. During the calculation of the target posture optimization energy function, its corresponding reprojection calculation item is calculated using the target posture optimization model. The reprojection edge information of the target object is calculated based on the initial posture, and combined with the actual edge information, it is input into the target posture optimization model based on deep learning to calculate the reprojection calculation item. Finally, the initial posture of the target object is adjusted using the posture optimization offset to obtain the optimized posture, that is, the optimized posture. The present application uses a target posture optimization model based on deep learning to replace the computationally intensive and inaccurate part of the relevant posture optimization equation, thereby improving the computational efficiency and accuracy of the target posture optimization energy function, and the operation is fast and more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The present application is further described below with reference to the accompanying drawings and specific implementation methods.

[0038] Figure 1 This is a flow chart of a posture optimization method provided in an embodiment of the present application.

[0039] Figure 2 This is a schematic diagram of the principle of a target edge detection model provided in an embodiment of the present application.

[0040] Figure 3 It is a schematic diagram of the principle of a target posture optimization model provided in an embodiment of the present application.

[0041] Figure 4 This is a structural block diagram of a robot provided in an embodiment of the present application.

[0042] Figure 5 This is a structural block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making any creative work are within the scope of protection of this application.

[0044] In the description of the embodiments of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly indicate the number of technical features indicated. Therefore, features defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0045] With the rapid development of computer vision technology, object pose estimation has become an important research field. Object pose estimation refers to determining the rotation and translation of an object in three-dimensional space, and is applicable to many fields such as robot navigation, augmented reality, and autonomous vehicles. Related object pose estimation methods rely on depth sensors or structured light scanners to obtain three-dimensional information of the object, and then use methods such as template matching or point cloud registration to calculate the object pose. However, these methods have limitations such as high cost, complex equipment, and strict environmental requirements. In addition, the performance of depth sensors is often affected when processing fast-moving objects or in poor lighting conditions.

[0046] To address these issues, methods for object pose estimation based on monocular RGB images have been developed, leveraging the image's color and texture information to infer the object's spatial position and orientation. Despite progress, these techniques still face challenges in accuracy and robustness when dealing with complex backgrounds, varying lighting conditions, and object occlusion.

[0047] With the continuous development of deep learning, an increasing number of methods are utilizing deep learning models. For example, deep learning-based pose estimation algorithms take a monocular RGB image as input and output direct or indirect measurements of the pose. Some algorithms, such as GDR-Net, directly output the object pose, while others, such as CDPN, output dense keypoint coordinates of the object and then use the PnP algorithm to solve the pose. However, the accuracy and stability of the object poses calculated by these two algorithms (which can serve as examples of the initial pose) still need to be improved. Therefore, some pose optimization methods have emerged, which perform secondary optimization on the initial pose estimated by deep learning models. Pose optimization methods based on region information, such as PWP3D and RBOT, use the color histogram information of the RGB image to segment the object, extract foreground and background information, and combine it with the object model to perform secondary iterative optimization of the object pose based on the reprojected outline of the initial pose. However, these optimization methods rely primarily on the object's color histogram and are only effective for small textures that are significantly different from the background, and their generalization is still insufficient.

[0048] To address the shortcomings of related pose optimization methods, the applicant proposed an object pose optimization method based on RGB images. This method uses a deep learning-based algorithm to extract object edge contours and combines it with a deep learning model to calculate the optimization function for the object pose. This method ensures the speed and accuracy of pose optimization while also ensuring the generalizability of pose optimization.

[0049] The implementation methods of this application will be described in detail below.

[0050] See also Figure 1 , Figure 1 This is a flow chart of a posture optimization method provided in an embodiment of the present application.

[0051] In order to improve the relevant technology, an embodiment of the present application provides a posture optimization method, which includes steps S101 to S104.

[0052] Step S101: receiving a target image containing a target object.

[0053] Step S102: performing edge detection on the target image to obtain actual edge information of the target object.

[0054] Step S103: Minimizing the target pose optimization energy function to calculate the pose optimization offset of the target object.

[0055] Step S104: optimizing the initial pose of the target object using the pose optimization offset to obtain the optimized pose of the target object.

[0056] Among them, the reprojection calculation item corresponding to the target pose optimization energy function is calculated using a target pose optimization model, and the target pose optimization model is a model based on deep learning. The input of the target pose optimization model includes the actual edge information and the reprojected edge information based on the initial pose.

[0057] In some embodiments, the reprojection calculation term corresponding to the target pose optimization energy function is, for example, obtained by derivatizing the target pose optimization energy function with respect to the pose optimization offset.

[0058] In some embodiments, the method can be applied to one or more application scenarios in robotics, virtual reality (VR), mixed reality (MR), and intelligent manufacturing. The robot can be, for example, a bipedal robot (also known as a humanoid robot, a humanoid robot), a quadrupedal robot, a wheeled robot, or other biomimetic robot, although the above embodiments do not limit this.

[0059] The above embodiments do not limit the target object; its shape may be, for example, a cylinder, a cone, a solid of revolution, a cross-section, etc. Cylinders include cylinders and prisms; cones include cones and pyramids; solids of revolution include cylinders, frustums, cones, spheres, ellipsoids, spherical caps, bow rings, rings, dike rings, fan rings, and jujube-shaped shapes; and cross-sections include prisms, frustums, truncated cylinders, truncated prisms, truncated cones, spherical caps, and spherical segments. The above embodiments do not limit the size of the target object; its size may be, for example, on the order of millimeters, centimeters, decimeters, meters, etc.

[0060] In some embodiments, the target image may be an RGB image. The above embodiments do not limit the target image, and it may be, for example, an RGB image, an HSV image, or the like. An RGB image is an image containing information about three channels: red, green, and blue. An HSV image is an image containing information about hue, saturation, and brightness. The target image is image data containing the object to be analyzed (i.e., the target object), such as an RGB image captured by a camera or other imaging device.

[0061] The target pose optimization energy function is a mathematical function that optimizes the initial pose based on the error between the actual edge and the reprojected edge. By minimizing this function, the object's pose can be optimized to be closer to its true position. The pose optimization offset refers to the amount by which the initial pose is adjusted during the optimization process. This offset corresponds to the difference between the initial pose and the optimization target. The reprojected edge information based on the initial pose refers to the object's edge information projected onto the image plane based on the target object's three-dimensional model and initial pose. A deep learning-based model refers to an algorithmic model that uses deep neural networks for data processing. In other words, the input of the target pose optimization model includes actual edge information and reprojected edge information, and the output of the target pose optimization model includes reprojection calculation items.

[0062] The posture optimization method of the above embodiment receives a target image containing a target object, first performs edge detection on the image to extract the actual edge information of the target object. Subsequently, the target posture optimization energy function is minimized using a target posture optimization model based on deep learning, thereby calculating the posture optimization offset of the target object. The input of the target posture optimization model includes actual edge information and reprojected edge information based on the initial posture, and the output includes a reprojection calculation term. Finally, the initial posture of the target object is adjusted using the posture optimization offset to obtain the optimized posture of the target object. Compared with the related methods, the above embodiment uses a target posture optimization model based on deep learning to calculate the optimization function of the object posture (i.e., the target posture optimization energy function), while ensuring the speed and accuracy of the target object optimization posture, ensuring the generalization of posture optimization. By introducing a deep learning model for edge detection and / or posture optimization, it is possible to automatically learn and extract more discriminative image features, reduce the need for manual parameter adjustment, and be applicable to a variety of different application scenarios. In addition, this method can effectively deal with partial occlusion of objects and background interference problems, thereby improving the generalization ability and reliability of pose optimization.

[0063] Related object edge detection algorithms, for example, use the Sobel algorithm or the Canny algorithm, but these algorithms have low generalization and accuracy, and it is difficult to meet the accuracy requirements of the pose optimization algorithm for edge detection. Therefore, the applicant adopts an edge detection algorithm based on deep learning. The input of the algorithm is, for example, an RGB image containing the target object, and the output is the edge information of the target object. In order to distinguish it from the edge information obtained by reprojection, the edge information obtained by edge detection can be called actual edge information, and the edge information obtained by reprojection can be called reprojected edge information. As an example, the size of the input RGB image is (3×H×W), 3 represents the three RGB color channels of the RGB image, and H and W represent the height and width of the image. The actual edge information can be represented by an object edge image, and the size of the output object edge image is (1×H×W), 1 represents that the object edge image has a black and white color channel, and H and W represent the height and width of the image. In the object edge image, the pixel value of the image representing the object edge is 0 or 1, 0 indicates that there is no object edge at the pixel position, and 1 indicates that there is an object edge at the pixel position.

[0064] See also Figure 2 , Figure 2This is a schematic diagram of the principle of a target edge detection model provided by an embodiment of the present application. The object edge detection algorithm model (i.e., the target edge detection model) can adopt an encoder-decoder structure, for example. The encoder compresses the input RGB image and extracts the image feature information. The decoder decodes the image feature information and outputs the detected object edge (i.e., the actual edge information). A jump connection can be used between the encoder and the decoder to retain image information of different scale features. Figure 2 As shown, the white cube in the input RGB image is the target object, and the output image is the edge of the target object.

[0065] When performing edge detection on a target image, a deep learning-based target edge detection model or other object edge detection algorithms may be used, and the above embodiment does not limit this. In some embodiments, performing edge detection on the target image to obtain actual edge information of the target object may include: performing edge detection on the target image using a deep learning-based target edge detection model to obtain actual edge information of the target object.

[0066] By employing a deep learning-based object edge detection model, the above-described embodiment maintains high detection accuracy compared to traditional edge detection methods, even in complex backgrounds, with varying lighting conditions, and with partial object occlusion. By utilizing the actual edge information extracted by this object edge detection model and combining it with reprojected edge information for pose optimization, the accuracy and stability of pose calculations are further improved. This method not only enhances the robustness of pose estimation but also effectively addresses object pose optimization problems in diverse application scenarios, demonstrating strong generalization and applicability.

[0067] In some embodiments, the target edge detection model performs edge detection on the target image to obtain actual edge information of the target object, which may include: extracting image feature information corresponding to the target image; and decoding the image feature information to obtain actual edge information of the target object.

[0068] The above embodiments do not limit the structure of the target edge detection model. In some embodiments, the target edge detection model may include an encoder and a decoder; the encoder is used to extract image feature information corresponding to the target image; and the decoder is used to decode the image feature information to obtain the actual edge information of the target object.

[0069] The encoder is responsible for extracting image features from the input target image. For example, the encoder can compress image information layer by layer through a series of convolution operations, extracting multi-scale, multi-level feature representations to better understand the image's structure and edge information. The decoder decodes the image features extracted by the encoder into the actual edge information of the target object.

[0070] The above embodiments do not limit the connection method between the encoder and decoder. In some embodiments, a skip connection can be used between the encoder and decoder. Skip connections can introduce feature information at different levels during the decoding process. For example, skip connections can directly pass high-resolution features extracted early in the encoder to the decoder, thereby preventing this feature information from being lost during the decoding process, achieving higher detection accuracy and detail recovery capabilities.

[0071] In the above embodiment, the target edge detection model adopts an encoder-decoder structure, and retains the high-resolution features in the encoder through jump connections, further improving the edge detection accuracy. By introducing the encoder-decoder structure, more accurate edge features can be extracted from the target image. In particular, during the decoding process, by using jump connections, the intermediate layer feature information of the encoder is directly passed to the decoder, so that the decoder retains more high-resolution details when restoring the edges. This improvement can still accurately extract the edge information of the target object when processing complex scenes or backgrounds, improves the accuracy and robustness of edge detection, and thus further improves the accuracy of pose optimization. Therefore, this method introduces a target edge detection model based on deep learning to extract the edge of the target object, replacing the related pose optimization method that uses regional color histograms for edge extraction, which is more stable and more accurate.

[0072] In the pose optimization algorithm, the foreground and background of the object can be extracted to obtain the actual edge of the object. Then, the reprojected edge of the object model on the image is calculated based on the initial pose, and the initial pose is optimized based on the error between the actual edge and the reprojected edge.

[0073] Based on this idea, the corresponding target pose optimization energy function can be shown as formula (1).

[0074]

[0075] In formula (1), Φ(x c) represents the distance between the actual edge of each edge point and the reprojected edge of the 3D model of the target object, I represents the target image, and the P(Φ|I) function represents the conditional probability of the distance between the actual edge and the reprojected edge according to the initial pose under the target image I. If the initial pose is accurate, the distance between the two should be 0, that is, the reprojected edge completely matches the actual edge. If the initial pose is inaccurate, the distance is larger. Ω represents the pixel area of ​​the target image I, x c Represents the surface point X in the 3D model of the target object M According to the projection of the initial pose P of the target object on the target image I (or the image coordinate system), it can be regarded as a reprojected edge point, which can be expressed by the following formula (2). c It can be expressed as (u, v), which represents the object contour point (or edge point) projected onto the image coordinate system. As an example, u =10, v =20, indicating that the pixel position corresponding to the 10th horizontal pixel and the 20th vertical pixel on the target image I is the projection position of an edge point of the target object.

[0076] λ(X c ) function is about X C function, λ represents the projection, X C is the three-dimensional position of the target object in the camera coordinate system, X C =(X, Y, Z). x , f y , p x , p y is the camera internal parameter, R and T represent the rotation and displacement of the pose, X, Y, Z are the surface point X in the three-dimensional model of the target object M The three-dimensional position of x. c The pixel color at the pixel location, H e is a unit step function, which can be expressed by formula (3).

[0077] In formula (3), π represents the ratio of circumference to circumference, and b is a hyperparameter, for example, 0.5. f (S) represents the foreground probability, P b (S) represents the background probability, and both can be calculated using the color histogram method based on the posterior probability.

[0078] According to the above, Φ(x c ) represents the distance between the actual edge of each edge point and the reprojected edge, which can be expressed by formula (4), where C(contour) represents the actual edge contour of the object, Ω f represents the foreground area (i.e., the object projection outline), Ω b Represents the background area, both can be represented by Pf (S) and P b (S) is calculated. c∈C, c is a point in the actual edge contour C, C is composed of a series of points, and formula (4) d is calculated as x c The shortest distance to contour C. As an example, traverse the points in contour C and find the minimum distance.

[0079]

[0080]

[0081] The pose optimization equation can be expressed by formula (5), where P r represents the optimized pose (i.e. the pose result after optimization), P r For example, a 4×4 matrix can be expressed as [R r , T r ], R r represents the optimized rotation matrix, T r represents the optimized translation matrix, [Δθ r ] × represents Δθ r The antisymmetric matrix of is, for example, a 3×3 matrix. Therefore, the pose optimization offset can be expressed as θ=[θ r ,θ t ], where θ r represents the amount of rotation, θ t Indicates the amount of translation.

[0082]

[0083] In order to calculate the pose optimization offset θ, the target pose optimization energy function P(Φ|I) can be differentiated with respect to θ to obtain Equation (6).

[0084] Formula (7) and formula (8) are the specific expressions of the two parts in formula (6).

[0085]

[0086] The second part of formula (6) It can be quickly calculated by combining existing parameters with a fixed formula (the fixed formula can be given in advance, for example), and the first part The calculation of is more complicated, and the results calculated by the relevant algorithm are not accurate enough, so The reprojection calculation item corresponding to the target pose optimization energy function can be calculated using a deep learning model (e.g., a target pose optimization model). The deep learning model has a powerful function fitting capability, and its generalization and accuracy are significantly superior to related algorithms. Using a target pose optimization model based on deep learning to replace the computationally intensive and inaccurate parts of the related pose optimization equations can achieve faster calculations and higher accuracy. In some embodiments, The input consists of the reprojected edge x based on the initial pose c , and the foreground P f (S) and background P b (S). Among them, the foreground and background P f (S) and P b (S) The actual edge that can be extracted by the target edge detection model Find . Therefore The input can be replaced by x c and Targeted becomes calculate The input of the deep learning model (i.e., the target pose optimization model) is x c and

[0087] The target pose optimization model can be based on the Transformer architecture, with input x c and x c and By image pixel position Indicates the edge position x c and It is represented by n pixel positions, where n is a positive integer. As an example, suppose x c The image consists of 300 pixels with a resolution of (256, 256). Two adjacent points are located at (100, 100) and (103, 104). After normalization preprocessing, the positions of the two points become (0.391, 0.391) and (0.402, 0.406). At this point, the difference in the digital representation of the two points is very small, which is not conducive to the deep learning model (i.e., the target pose optimization model) to distinguish their positional relationship. Based on this, in x c and 2D position encoding can be performed before inputting the target pose optimization model. The encoding formula is shown in formula (9), where L is an integer. Assuming L = 10, the position dimension of a pixel changes from 2 to 40, and the position representation of adjacent boundary points is quite different, which is conducive to the deep learning model to distinguish the positions of different boundary points.

[0088]

[0089] Assume x c and It consists of 300 pixels with a dimension of (300, 2). After position encoding, its dimension becomes (300, 40). Then the position-encoded x c and Input the multi-layer fully connected network (MLP) and get and The dimension is (300, h), where h is an integer. As an example, h can be set to 128. Then and Perform cross attention encoding to obtain F, as shown in equations (10) and (11). Finally, the feature F is obtained through a multi-layer fully connected network (MLP)

[0090]

[0091] See also Figure 3 , Figure 3 This is a schematic diagram of the principle of a target posture optimization model provided in the embodiment of the present application, which is used to calculate The target pose optimization model calculates After that, the optimized object pose (i.e., optimized pose) can be calculated according to equations (5) and (6).

[0092] In some embodiments, the target pose optimization model can be trained by reprojection error. As an example, the reprojection error is expressed as Equation (12), where is the optimized pose, represents the rotation matrix of the optimized pose, Represents the translation matrix of the optimized pose. X i Indicates the position of each surface point of the 3D model of the target object, i is the serial number of the surface point. Assuming that the 3D model surface of the target object consists of n surface points, then X i represents the i-th surface point. gt , T gt ] is the true value of the pose. Among them, R gt is the rotation value of the pose truth, for example, it is a 3×3 matrix, T gt is the translation of the true pose value, for example, a three-dimensional vector. During training, x c The object pose truth value [R gt , T gt ]After adding a small amount of random perturbation error, the three-dimensional model is projected according to the posture with added noise.

[0093]

[0094] The above embodiment does not limit the manner in which the target pose optimization model calculates the reprojection calculation item. In some embodiments, the process of the target pose optimization model calculating the reprojection calculation item may include: performing cross-attention encoding based on the actual edge information and the reprojection edge information to obtain cross-attention feature information; performing full-connection processing on the cross-attention feature information to calculate the reprojection calculation item. As an example, the target pose optimization model may include an attention module and a fully connected layer (e.g., an output fully connected layer), the attention module being used to perform cross-attention encoding operations, and the fully connected layer being used to perform fully connected operations. The above embodiment uses the attention module to optimize the features of the edge positions of the input model to improve the accuracy of pose optimization.

[0095] In order to facilitate the target pose optimization model to distinguish the positions of adjacent edge points, in some embodiments, the cross-attention encoding based on the actual edge information and the reprojected edge information to obtain cross-attention feature information may include: performing position encoding processing on the actual edge information to obtain actual position encoding information; and, performing position encoding processing on the reprojected edge information to obtain reprojected position encoding information; and performing cross-attention encoding using the actual position encoding information and the reprojected position encoding information to obtain cross-attention feature information. As an example, the target pose optimization model may include a position encoding unit, which is used to perform position encoding operations. The above embodiment uses position encoding to perform position dimensionality upscaling on the edge position points of the input model, which is beneficial for the model to distinguish the positions of adjacent edge points.

[0096] Before position encoding, the edge information may be normalized. In some embodiments, the position encoding of the actual edge information to obtain actual position encoding information may include: normalizing the actual edge information to obtain actual normalized information; and position encoding the actual normalized information to obtain actual position encoding information. In some embodiments, the position encoding of the reprojected edge information to obtain reprojected position encoding information may include: normalizing the reprojected edge information to obtain reprojected normalized information; and position encoding the reprojected normalized information to obtain reprojected position encoding information. As an example, the target pose optimization model may include a normalization unit, which is used to perform a normalization operation.

[0097] In some embodiments, the cross-attention encoding using the actual position encoding information and the reprojection position encoding information to obtain cross-attention feature information may include: performing full-connection processing on the actual position encoding information to obtain actual fully-connected feature information; and performing full-connection processing on the reprojection position encoding information to obtain reprojection fully-connected feature information; and performing cross-attention encoding on the actual fully-connected feature information and the reprojection fully-connected feature information to obtain cross-attention feature information.

[0098] The present application also provides a posture optimization device, which includes an image receiving module, an edge detection module, a solution module and an optimization module. The image receiving module is used to receive a target image containing a target object. The edge detection module is used to perform edge detection on the target image to obtain the actual edge information of the target object. The solution module is used to minimize the target posture optimization energy function to calculate the posture optimization offset of the target object. The optimization module is used to optimize the initial posture of the target object using the posture optimization offset to obtain the optimized posture of the target object. The reprojection calculation item corresponding to the target posture optimization energy function is calculated using a target posture optimization model, and the target posture optimization model is a model based on deep learning. The input of the target posture optimization model includes the actual edge information and the reprojected edge information based on the initial posture.

[0099] In some embodiments, the edge detection module can be used to: perform edge detection on the target image using a deep learning-based target edge detection model to obtain actual edge information of the target object.

[0100] In some embodiments, the target edge detection model performs edge detection on the target image to obtain actual edge information of the target object, which may include: extracting image feature information corresponding to the target image; and decoding the image feature information to obtain actual edge information of the target object.

[0101] In some embodiments, the target edge detection model may include an encoder and a decoder; the encoder is used to extract image feature information corresponding to the target image; and the decoder is used to decode the image feature information to obtain actual edge information of the target object.

[0102] In some embodiments, a skip connection may be used between the encoder and the decoder.

[0103] In some embodiments, the process of calculating the reprojection calculation item by the target pose optimization model may include: performing cross-attention encoding based on the actual edge information and the reprojection edge information to obtain cross-attention feature information; and performing full connection processing on the cross-attention feature information to calculate the reprojection calculation item.

[0104] In some embodiments, the target pose optimization model can perform cross-attention encoding based on the actual edge information and the reprojected edge information in the following manner to obtain cross-attention feature information: performing position encoding processing on the actual edge information to obtain actual position encoding information; and performing position encoding processing on the reprojected edge information to obtain reprojection position encoding information; and performing cross-attention encoding using the actual position encoding information and the reprojection position encoding information to obtain cross-attention feature information.

[0105] In some embodiments, the target pose optimization model can use the actual position encoding information and the reprojection position encoding information to perform cross-attention encoding to obtain cross-attention feature information in the following manner: fully connect the actual position encoding information to obtain actual fully connected feature information; and fully connect the reprojection position encoding information to obtain reprojection fully connected feature information; cross-attention encode the actual fully connected feature information and the reprojection fully connected feature information to obtain cross-attention feature information.

[0106] In some embodiments, the target image may be an RGB image.

[0107] In some embodiments, the device can be applicable to one or more application scenarios in robotics, virtual reality, mixed reality, and intelligent manufacturing.

[0108] See also Figure 4 , Figure 4 This is a structural block diagram of a robot provided in an embodiment of the present application.

[0109] An embodiment of the present application also provides a robot, which includes a control module and an image sensor. The control module is used to execute any of the above methods to obtain an optimized posture of a target object; the image sensor is used to obtain a target image containing the target object.

[0110] In some embodiments, the robot may further include a laser sensor. The laser sensor may include a 2D laser sensor and / or a 3D laser sensor.

[0111] In some embodiments, the robot may also include an odometry and / or an IMU.

[0112] In some embodiments, the robot may further include one or more of an angle encoder, a torque sensor, and a PIR sensor.

[0113] In some embodiments, the robot may be a multi-jointed robot.

[0114] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above methods is implemented.

[0115] An embodiment of the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, implements any of the above methods.

[0116] The computer program product may be a portable compact disc read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the computer program product of the present application is not limited thereto, and the computer program product may be any combination of one or more computer-readable media.

[0117] An embodiment of the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements any of the above methods when executing the computer program.

[0118] See also Figure 5 , Figure 5 This is a structural block diagram of a computer device provided in an embodiment of the present application.

[0119] The embodiments of the present application do not limit the computer device, which may be, for example, a local computer device, a cloud computer device, a distributed computer device, etc.

[0120] The computer device may include: a memory 110, a processor 120, and a communication interface 130. The memory 110, the processor 120, and the communication interface 130 are connected via an internal connection path.

[0121] The memory 110 is used to store computer programs. In some implementations, the computer programs may include codes for implementing the methods of the embodiments of the present application.

[0122] The processor 120 is configured to execute the computer program stored in the memory 110 to control the communication interface 130 to receive input data and information and output data such as operation results. In some implementations, when the solutions of the embodiments of the present application are implemented through software or firmware, the computer program for implementing the solutions of the embodiments of the present application may be stored in the processor 120 and executed by the processor 120.

[0123] The memory 110 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (Programmable ROM, PROM), an erasable programmable read-only memory (Erasable PROM, EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory may be a random access memory (RAM). It should be noted that the memory 110 described herein is intended to include, but is not limited to, any memory of these and other suitable types. As an example, the memory 110 includes a random access memory (RAM), a cache memory and a read-only memory (ROM). Among them, the memory 110 stores a computer program, and the computer program can be executed by the processor 120 so that the processor 120 implements the steps of any of the above methods.

[0124] The processor 120 may be a central processing unit (CPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor, or the processor 120 may be any conventional processor.

[0125] During implementation, each step of the above method can be completed by an integrated logic circuit of the hardware in the processor 120 or by instructions in the form of software. The method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor 120. The software module can be located in a mature storage medium in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 110, and the processor 120 reads the information in the memory 110 and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0126] In some implementations, in addition to the hardware units described above, the computer device may also include software modules, where the software modules may be, for example, an operating system, a basic input and output system (BIOS), application software, etc.

[0127] An operating system manages the hardware and / or software resources of a computer device and is the core and cornerstone of the computer. It handles basic tasks such as managing and allocating memory, prioritizing the supply and demand of system resources, controlling input and output devices, operating the network, and managing the file system. To facilitate user operation, most operating systems provide an interface for users to interact with the system.

[0128] The BIOS is used to run hardware initialization during the power-on boot phase and provide runtime services for the operating system and applications. In some implementations, the BIOS can also monitor and display the processor temperature and execute functions such as adjusting temperature protection strategies.

[0129] Application software, also known as an application program, is software written for a specific user purpose. It is a major category of computer software. For example, application software might be a program used for power control, temperature management, and other purposes.

[0130] It should be noted that although some embodiments of this application take mobile robots as an example, this application can be applied to other bionic robots, such as AGVs, drones, etc., and this application is not limited to this.

[0131] It should be understood that the specific examples in this specification are only intended to help those skilled in the art better understand the implementation methods of the present application, rather than to limit the scope of protection of the present application.

[0132] It can be understood that in the various implementations of this specification, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application.

[0133] It can be understood that the various implementation methods described in this specification can be implemented individually or in combination, and this application is not limited to this.

[0134] Unless otherwise indicated, all technical and scientific terms used in this specification have the same meaning as commonly understood by those skilled in the art in the technical field of this specification. The terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit the scope of this specification. The term "and / or" used in this specification includes any and all combinations of one or more of the relevant listed items. The singular forms "a", "above", and "the" used in this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0135] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this specification.

[0136] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described embodiments may refer to the corresponding processes in other embodiments and will not be repeated here.

[0137] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0138] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the objectives of the technical solutions of this application.

[0139] In addition, each functional unit in each embodiment of this specification may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0140] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of this specification. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program code.

[0141] The above are only specific embodiments of this specification, but the scope of protection of this application is not limited to them. Any changes or substitutions that can be easily conceived by any person skilled in the art within the technical scope disclosed in this specification should be included in the scope of protection of this specification. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A posture optimization method, characterized in that: The method comprises: receiving a target image containing a target object; Performing edge detection on the target image to obtain actual edge information of the target object; Minimizing the target pose optimization energy function to calculate the pose optimization offset of the target object; Optimizing the initial pose of the target object using the pose optimization offset to obtain an optimized pose of the target object; Among them, the reprojection calculation item corresponding to the target pose optimization energy function is calculated using a target pose optimization model, and the target pose optimization model is a model based on deep learning. The input of the target pose optimization model includes the actual edge information and the reprojected edge information based on the initial pose.

2. The posture optimization method according to claim 1, characterized in that The performing edge detection on the target image to obtain actual edge information of the target object includes: The target image is subjected to edge detection using a deep learning-based target edge detection model to obtain actual edge information of the target object.

3. The posture optimization method according to claim 2, characterized in that The target edge detection model performs edge detection on the target image to obtain actual edge information of the target object, including: Extracting image feature information corresponding to the target image; The image feature information is decoded to obtain actual edge information of the target object.

4. The posture optimization method according to claim 3, characterized in that The target edge detection model includes an encoder and a decoder; The encoder is used to extract image feature information corresponding to the target image; The decoder is used to decode the image feature information to obtain actual edge information of the target object.

5. The posture optimization method according to claim 1, characterized in that The process of calculating the reprojection calculation item by the target pose optimization model includes: Performing cross-attention encoding based on the actual edge information and the reprojected edge information to obtain cross-attention feature information; The cross-attention feature information is fully connected to calculate the reprojection calculation item.

6. The posture optimization method according to claim 5, characterized in that The performing cross attention encoding based on the actual edge information and the reprojected edge information to obtain cross attention feature information includes: Performing position coding processing on the actual edge information to obtain actual position coding information; and performing position coding processing on the reprojected edge information to obtain reprojected position coding information; Cross-attention encoding is performed using the actual position encoding information and the re-projected position encoding information to obtain cross-attention feature information.

7. The posture optimization method according to claim 6, characterized in that: The cross-attention encoding is performed using the actual position encoding information and the re-projected position encoding information to obtain cross-attention feature information, including: Performing full connection processing on the actual position encoding information to obtain actual fully connected feature information; and performing full connection processing on the reprojected position encoding information to obtain reprojected fully connected feature information; Cross-attention encoding is performed on the actual fully-connected feature information and the reprojected fully-connected feature information to obtain cross-attention feature information.

8. The posture optimization method according to claim 1, characterized in that: The target image is an RGB image.

9. The posture optimization method according to claim 1, characterized in that: The method is applicable to one or more application scenarios in robotics, virtual reality, mixed reality and intelligent manufacturing.

10. A robot, characterized in that: The robot includes a control module and an image sensor, the control module is used to execute the method according to any one of claims 1 to 8 to obtain an optimized posture of a target object; the image sensor is used to obtain a target image containing the target object.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

12. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 9 when executing the computer program.

13. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images