Pose estimation method and apparatus, related devices and computer program product

Abstract

The present disclosure provides a pose estimation method and device, electronic equipment, computer readable storage medium and computer product, relating to the technical field of artificial intelligence, and the method comprises: obtaining model coordinate values of model key points on a model of a target object in a model coordinate system; obtaining a target image comprising the target object; performing key point bias prediction on the target image to determine position bias of each point in the target image in a camera coordinate system relative to a predicted key point; determining camera coordinate values of the predicted key point in the camera coordinate system according to the position bias of each point in the target image in the camera coordinate system relative to the predicted key point; splicing the model coordinate values of the model key points and the camera coordinate values of the predicted key points to obtain fusion coordinate values; and determining a pose of the target object in the camera coordinate system according to the fusion coordinate values and the model of the target object.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a pose estimation method, apparatus, electronic device, computer-readable storage medium, and computer product. Background Technology

[0002] Pose estimation has a wide range of applications and strong practical value in fields such as human-computer interaction, virtual reality, autonomous driving, and logistics classification. However, current research on pose estimation faces many problems, such as low accuracy and inability to determine the pose orientation.

[0003] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure. Summary of the Invention

[0004] The purpose of this disclosure is to provide an attitude estimation method, apparatus, electronic device, computer-readable storage medium, and computer product that can improve the accuracy of attitude estimation while determining the position and orientation of a target object.

[0005] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0006] This disclosure provides a pose estimation method, comprising: acquiring model coordinate values ​​of model keypoints on a target object in a model coordinate system; acquiring a target image, wherein the target image is obtained after image acquisition of the target object; performing keypoint offset prediction on the target image to determine the position offset of each point in the target image relative to the predicted keypoints in a camera coordinate system, wherein the predicted keypoints are keypoints predicted for the target object in the camera coordinate system, and the predicted keypoints correspond one-to-one with the model keypoints; determining the camera coordinate values ​​of the predicted keypoints in the camera coordinate system based on the position offsets of each point in the target image relative to the predicted keypoints; concatenating the model coordinate values ​​of the model keypoints with the camera coordinate values ​​of the predicted keypoints to obtain fused coordinate values; and determining the pose of the target object in the camera coordinate system based on the fused coordinate values ​​and the model of the target object.

[0007] This disclosure provides a pose estimation device, including: a model coordinate value acquisition module, a target image acquisition module, an offset prediction module, a camera coordinate value prediction module, a fused coordinate determination module, and a pose prediction module.

[0008] The module includes several modules: a model coordinate value acquisition module for acquiring the model coordinate values ​​of key points on the target object's model in the model coordinate system; a target image acquisition module for acquiring a target image obtained after image acquisition of the target object; an offset prediction module for performing key point offset prediction on the target image to determine the position offset of each point in the target image relative to the predicted key points in the camera coordinate system, wherein the predicted key points are key points predicted for the target object in the camera coordinate system and correspond one-to-one with the model key points; a camera coordinate value prediction module for determining the camera coordinate values ​​of the predicted key points in the camera coordinate system based on the position offsets of each point in the target image relative to the predicted key points; a fusion coordinate determination module for concatenating the model coordinate values ​​of the model key points with the camera coordinate values ​​of the predicted key points to obtain fusion coordinate values; and a pose prediction module for determining the pose of the target object in the camera coordinate system based on the fusion coordinate values ​​and the model of the target object.

[0009] In some embodiments, the camera coordinate prediction module includes: a semantic segmentation submodule, a position offset prediction submodule, a candidate camera coordinate determination submodule, and a camera coordinate prediction submodule.

[0010] The semantic segmentation submodule is used to perform semantic segmentation on the target image to determine multiple target points for describing the target object among various points in the target image; the position offset prediction submodule is used to determine the position offset of each target point relative to the predicted key point based on the position offset of each point relative to the predicted key point; the candidate camera coordinate value determination submodule is used to determine multiple candidate camera coordinate values ​​of the predicted key point based on the position offset of each target point relative to the predicted key point; and the camera coordinate value prediction submodule is used to determine the camera coordinate value of the predicted key point in the camera coordinate system based on the multiple candidate camera coordinate values.

[0011] In some embodiments, the camera coordinate value prediction submodule includes: a variance determination unit, a Gaussian standardization unit, a elimination unit, and a camera coordinate value determination unit.

[0012] The variance determination unit is used to determine the mean and variance of the coordinate values ​​corresponding to the plurality of candidate camera coordinate values; the Gaussian standardization unit is used to perform Gaussian standardization on each candidate camera coordinate value according to the mean and variance of the coordinate values, so as to determine the convergence confidence of each candidate camera coordinate value according to the Gaussian standardization result; the elimination unit is used to eliminate candidate camera coordinate values ​​with convergence confidence greater than a target threshold to obtain converged camera coordinate values; the camera coordinate value determination unit is used to determine the camera coordinate values ​​of the predicted keypoints in the camera coordinate system according to the converged camera coordinate values ​​and the convergence confidence of the converged camera coordinate values.

[0013] In some embodiments, the camera coordinate value determination unit includes a weighted summation subunit.

[0014] The weighted summation subunit is used to perform weighted summation on each converged camera coordinate value with the convergence confidence as the weight, so as to determine the camera coordinate value of the predicted key point in the camera coordinate system.

[0015] In some embodiments, the camera coordinate prediction submodule is configured to include: a mean determination unit or a clustering unit.

[0016] The mean determination unit is used to determine the mean of coordinate values ​​based on the multiple candidate camera coordinate values, and use the mean of coordinate values ​​as the camera coordinate values ​​of the predicted key points in the camera coordinate system; the clustering unit is used to perform clustering processing on the multiple candidate camera coordinate values, and use the coordinate values ​​corresponding to the cluster centers as the camera coordinate values ​​of the predicted key points in the camera coordinate system.

[0017] In some embodiments, the target image is acquired by a target image acquisition device, and the camera coordinate system is a coordinate system constructed based on the target image acquisition device; wherein, the candidate camera coordinate value determination submodule includes: an intrinsic parameter acquisition unit, a point coordinate determination unit, and a camera coordinate value determination unit.

[0018] The intrinsic parameter acquisition unit is used to acquire the intrinsic parameter information of the target image acquisition device; the point coordinate determination unit is used to process the target image using the intrinsic parameter information of the image acquisition device to determine the coordinate values ​​of each target point in the camera coordinate system; the camera coordinate value determination unit is used to determine the multiple candidate camera coordinate values ​​of the predicted key point based on the coordinate values ​​of each target point in the camera coordinate system and the positional offset of each target point relative to the predicted key point.

[0019] In some embodiments, the target image is a color depth map including target depth map information and target color map information; wherein, the bias prediction module includes: a color feature acquisition submodule, a depth feature acquisition submodule, a feature fusion submodule, and a keypoint bias prediction submodule.

[0020] The color feature acquisition submodule is used to extract features from the target color map information to obtain color features; the depth feature acquisition submodule is used to extract features from the target depth map information to obtain depth features; the feature fusion submodule is used to fuse the color features and the depth features to obtain color depth fusion features; and the keypoint offset prediction submodule is used to predict keypoint offsets based on the color depth fusion features to determine the position offset of each point in the target image relative to the predicted keypoint.

[0021] In some embodiments, the attitude prediction module includes: a prediction rotation and translation matrix determination submodule and an attitude prediction submodule.

[0022] The prediction rotation and translation matrix determination submodule is used to determine the prediction rotation and translation matrix between the model key points and the prediction key points based on the fused coordinate values; the attitude prediction submodule is used to rotate and translate the model of the target object based on the prediction rotation and translation matrix to determine the attitude of the target object in the camera coordinate system.

[0023] In some embodiments, the predicted rotation and translation matrix determination submodule includes a feature extraction unit and a prediction processing unit.

[0024] The feature extraction unit is used to extract features from the fused coordinate values ​​through a target convolutional structure to obtain fused coordinate features; the prediction processing unit is used to predict the fused coordinate features through a target classifier to determine the predicted rotation and translation matrix.

[0025] In some embodiments, the predicted rotation and translation matrix includes a rotation matrix, which includes a first rotation direction, a second rotation direction, and a third rotation direction; wherein, the prediction processing unit includes: a first rotation data prediction subunit, a first normalization subunit, a second normalization subunit, and a second rotation data prediction subunit.

[0026] The first rotation data prediction subunit is used to predict the fused coordinate features using a target classifier to determine the first rotation data of the rotation matrix in the first rotation direction and the second rotation data in the second rotation direction. The first normalization subunit is used to normalize the first rotation data to obtain first rotation normalized data. The second normalization subunit is used to perform a cross product between the first rotation normalized data and the second rotation data, and normalize the result of the cross product to obtain the third rotation normalized data of the rotation matrix in the third rotation direction. The second rotation data prediction subunit is used to perform a cross product between the first rotation normalized data and the third rotation normalized data to obtain the second rotation normalized data of the rotation matrix in the second rotation direction.

[0027] In some embodiments, the pose estimation method is performed by a target network model; wherein the pose estimation device further includes: an actual rotation and translation matrix acquisition module, a target loss value determination module, and a training module.

[0028] The actual rotation and translation matrix acquisition module is used to acquire the actual rotation and translation matrix between the model keypoints and the actual keypoints, wherein the actual keypoints are the keypoints of the target object in the target image in the actual pose in the camera coordinate system, and the model keypoints and the actual keypoints correspond one-to-one; the target loss value determination module is used to determine the target loss value based on the predicted rotation and translation matrix and the actual rotation and translation matrix; the training module is used to train the target network model based on the target loss value.

[0029] This disclosure provides an electronic device comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the attitude estimation method described above.

[0030] This disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the attitude estimation method as described in any of the preceding embodiments.

[0031] This disclosure provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned attitude estimation method.

[0032] The pose estimation method, apparatus, electronic device, computer-readable storage medium, and computer product provided in this disclosure, on the one hand, determine the pose of a target object in the camera coordinate system by fusing coordinate values ​​and a model of the target object. This allows for determining the target object's position in the camera coordinate system while simultaneously determining its orientation based on key points. Furthermore, the model avoids the problem of inaccurate pose estimation caused by occlusion of multiple targets in the target image. On the other hand, when determining predicted key points, this disclosure first predicts the positional offset of each point in the target image relative to the predicted key points, and then further determines the position of the predicted key points based on the positional offsets. Since the positional offset is a data point with a finite range, this disclosure avoids the data divergence problem caused by directly predicting the key point position, thereby improving the accuracy of key point position prediction.

[0033] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this disclosure. Attached Figure Description

[0034] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0035] Figure 1 A schematic diagram of an exemplary system architecture that can be applied to the attitude estimation method or attitude estimation apparatus of the embodiments of this disclosure is shown.

[0036] Figure 2 This is a flowchart illustrating an attitude estimation method according to an exemplary embodiment.

[0037] Figure 3 It is a model of a target object shown according to an exemplary embodiment.

[0038] Figure 4 This is a schematic diagram illustrating a target image according to an exemplary embodiment.

[0039] Figure 5 This is a schematic diagram of a point cloud of a target object according to an exemplary embodiment.

[0040] Figure 6 This is a flowchart illustrating a method for determining the camera coordinate values ​​of predicted keypoints based on the positional offsets of each point in a target image relative to the predicted keypoints in the camera coordinate system, according to an exemplary embodiment.

[0041] Figure 7 This is a flowchart illustrating a method for determining camera coordinate values ​​using the Gaussian standard voting method according to an exemplary embodiment.

[0042] Figure 8 This is a position offset prediction method illustrated according to an exemplary embodiment.

[0043] Figure 9 This is a flowchart illustrating a method for determining a predicted rotation and translation matrix according to an exemplary embodiment.

[0044] Figure 10 This is an attitude estimation method illustrated according to an exemplary embodiment.

[0045] Figure 11 This is a schematic diagram of an attitude estimation structure illustrated by an example.

[0046] Figure 12 This is a schematic diagram illustrating a pose estimation comparison result according to an exemplary embodiment.

[0047] Figure 13 This is a schematic diagram illustrating a pose estimation comparison result according to an exemplary embodiment.

[0048] Figure 14 This is a schematic diagram illustrating a pose estimation comparison result according to an exemplary embodiment.

[0049] Figure 15 This is a block diagram illustrating an attitude estimation device according to an exemplary embodiment.

[0050] Figure 16 A schematic diagram of the structure of an electronic device suitable for implementing embodiments of the present disclosure is shown. Detailed Implementation

[0051] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0052] The features, structures, or characteristics described in this disclosure can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more specific details omitted, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0053] The accompanying drawings are merely illustrative of this disclosure, and the same reference numerals in the drawings denote the same or similar parts, thus omitting repeated descriptions of them. Some block diagrams shown in the drawings do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0054] The flowchart shown in the accompanying drawings is merely illustrative and does not necessarily include all content and steps, nor does it require execution in the described order. For example, some steps may be broken down, while others may be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0055] In the description of this disclosure, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. "And / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" means one or more, and "multiple" means two or more. The words "first," "second," etc., do not limit the quantity or order of execution, and "first," "second," etc., do not necessarily imply differences; the words "contains," "includes," and "has" are used to indicate an open-ended meaning of inclusion and refer to the existence of additional elements / components / etc. besides those listed.

[0056] To better understand the above-mentioned objectives, features and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0057] The following is an explanation of some of the terms used in the embodiments of this disclosure, so that those skilled in the art can understand them.

[0058] The preceding text introduced some terms and concepts involved in the embodiments of this disclosure. The following text introduces the technical features involved in the embodiments of this disclosure.

[0059] FFB6D: A network for bidirectional fusion of point cloud and RGB (Red, Green, Blue) information.

[0060] PSPnet: A network that utilizes an image pyramid structure for semantic segmentation.

[0061] ResNet: A framework for image convolutional networks.

[0062] The technical solution disclosed herein uses computer vision (CV) technology in the field of artificial intelligence (AI) to predict the pose of the target object in the camera coordinate system based on the model of the target object and the image containing the target object.

[0063] Artificial intelligence (AI) encompasses the theories, methods, technologies, and application systems that utilize digital computers or computers-controlled machines to simulate, extend, and expand human intelligence, enabling them to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce new intelligent machines capable of reacting in a manner similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0064] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0065] Computer vision technology is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0066] The improved technical solution of this disclosure combines computer vision to process the target image to predict the position offset of each point in the target image relative to the predicted key points; then, based on the position offset, the position of the predicted key points in the camera coordinate system is determined; next, the model key points of the target object are concatenated with the predicted key points to predict the predicted rotation and translation matrix between the model key points and the predicted key points, so as to rotate and translate the model of the target object according to the predicted rotation and translation matrix to determine the pose of the target object in the camera coordinate system.

[0067] The exemplary embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0068] There are two main types of existing deep learning-based pose estimation:

[0069] 1. A classification method based on deep learning. The specific strategy is to divide the space into intervals, modeling the pose estimation problem as a non-discriminatory multi-class classification problem. After feature extraction, the output is the category of the spatial location to which the object's pose belongs.

[0070] 2. Direct Regression Method Based on Deep Learning. The specific strategy is to extract RGB or point cloud features using a convolutional network, and then directly regress the 6D pose of the object using the corresponding features. Here, 6D refers to 6 degrees of freedom, representing 3 degrees of freedom of displacement (also called translation) and 3 degrees of freedom of spatial rotation.

[0071] However, the first method treats the 6D pose estimation problem as a classification problem, and its pose estimation accuracy is poor because it is difficult to refine the entire space. The second method uses direct feature regression, but since it only supervises the pose information of the object and lacks further mining of the object's geometric structure information, its pose estimation accuracy is also poor.

[0072] This disclosure designs a suitable strategy and network structure that achieves end-to-end training and prediction from image to pose estimation while improving the accuracy of pose estimation.

[0073] Figure 1 A schematic diagram of an exemplary system architecture that can be applied to the attitude estimation method or attitude estimation apparatus of the embodiments of this disclosure is shown.

[0074] like Figure 1 As shown, the system architecture 100 may include image acquisition devices 101 and 102, terminal devices 103 and 104, a network 105, and a server 106. The network 105 serves as the medium for providing communication links between the acquisition devices 101 and 102, the terminal devices 103 and 104, and the server 106. The network 105 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0075] Users can use image acquisition devices 101 and 102 to acquire images of target objects and send the acquisition results to terminal devices 103 and 104 or server 106 via network 105 or other data transmission devices (such as USB). The image acquisition devices can be 2D or 3D image acquisition devices (such as point cloud image acquisition devices or RGBD image acquisition devices). The 2D image acquisition devices can be color image acquisition devices or black and white image acquisition devices; this disclosure does not impose any restrictions on this. Similarly, the 3D image acquisition devices can be point cloud image acquisition devices or RGBD image acquisition devices; this disclosure also does not impose any restrictions on this.

[0076] In some embodiments, the image acquisition device described above may be a color camera as shown in 101, or a depth camera (such as Kinect) as shown in 102. The image acquisition device may also be any device capable of image acquisition, such as a mobile phone or a computer, and this disclosure does not limit it.

[0077] In some embodiments, users can use terminal devices 103 and 104 to interact with server 106 via network 105 to receive or send messages, etc.

[0078] For example, users can use terminal devices 103 and 104 to receive target images acquired by image acquisition devices; users can also use terminal devices 103 and 104 to acquire images of target objects; users can also use terminal devices 103 and 104 to build and realize models of target objects; users can also use terminal devices 103 and 104 to send the acquired target images to the server for target object pose estimation; users can also use terminal devices 103 and 104 to perform target object pose estimation.

[0079] In some embodiments, user terminals 103 and 104 may, for example, perform pose estimation of the target object through the following process: obtaining the model coordinate values ​​of model keypoints on the target object's model in the model coordinate system; acquiring a target image, which is obtained after image acquisition of the target object; performing keypoint offset prediction on the target image to determine the position offset of each point in the target image relative to the predicted keypoints in the camera coordinate system, wherein the predicted keypoints are keypoints predicted for the target object in the camera coordinate system and correspond one-to-one with the model keypoints; determining the camera coordinate values ​​of the predicted keypoints in the camera coordinate system based on the position offsets of each point in the target image relative to the predicted keypoints; concatenating the model coordinate values ​​of the model keypoints with the camera coordinate values ​​of the predicted keypoints to obtain fused coordinate values; and determining the predicted rotation and translation matrix between the model keypoints and the predicted keypoints based on the fused coordinate values, so as to determine the pose of the target object in the camera coordinate system based on the target object's model and the predicted rotation and translation matrix.

[0080] Among them, terminal devices 103 and 104 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, desktop computers, wearable devices, virtual reality devices, smart home devices, etc.

[0081] Server 106 can be a server that provides various services, such as a backend management server that supports the devices operated by users using terminal devices 103 and 104. The backend management server can analyze and process received requests and other data, and feed the processing results back to the terminal devices.

[0082] A server can be a standalone physical server, a server cluster or a distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. This disclosure does not impose any restrictions on this.

[0083] Server 106 may, for example, acquire the model coordinates of key points on the model of the target object in the model coordinate system; server 106 may, for example, acquire a target image, which is obtained after image acquisition of the target object; server 106 may, for example, perform key point offset prediction on the target image to determine the position offset of each point in the target image relative to the predicted key points in the camera coordinate system, where the predicted key points are key points predicted for the target object in the camera coordinate system, and the predicted key points correspond one-to-one with the model key points; server 106 may, for example, determine the camera coordinates of the predicted key points in the camera coordinate system based on the position offsets of each point in the target image relative to the predicted key points; server 106 may, for example, concatenate the model coordinates of the model key points with the camera coordinates of the predicted key points to obtain fused coordinate values; server 106 may, for example, determine the pose of the target object in the camera coordinate system based on the fused coordinate values ​​and the model of the target object.

[0084] It should be understood that Figure 1 The number of image acquisition devices, terminal devices, networks, and servers shown in the diagram is merely illustrative. Server 106 can be a single physical server or a combination of multiple servers. Depending on actual needs, it can have any number of image acquisition devices, terminal devices, networks, and servers.

[0085] Under the above system architecture, this disclosure provides an attitude estimation method. This method can be executed by any electronic device with computing power, for example, the method can be performed by the aforementioned... Figure 1 The execution can be performed by a server or terminal device in the embodiments, or it can be performed by both a server and a terminal device. In the following embodiments, the server is used as the execution subject for illustration, but this disclosure is not limited to this.

[0086] Figure 2 This is a flowchart illustrating an attitude estimation method according to an exemplary embodiment.

[0087] Reference Figure 2 The attitude estimation method provided in this disclosure may include the following steps.

[0088] Step S202: Obtain the model coordinate values ​​of the model key points on the target object's model in the model coordinate system.

[0089] The target object can refer to any object that can be captured by images, such as express delivery, toys (e.g., toy dogs, toy cats), cardboard boxes, chairs, tables, robots, etc.

[0090] In some embodiments, the size, proportion, and positional offsets between key points of the target object can be fixed.

[0091] The model of the target object can refer to a model built in advance for the target object. The model can be a 2D (Dimension) model or a 3D model. This disclosure does not limit it.

[0092] In some embodiments, the model of the target object may be the same as or different from the size (and shape) of the target object. Those skilled in the art can make adjustments according to their own needs, and this disclosure does not limit this.

[0093] Figure 3 It is a 3D model of a target object shown according to an exemplary embodiment.

[0094] The target object could be a toy cat.

[0095] In some embodiments, a model coordinate system can be constructed based on the model of the target object. For example, the model coordinate system can be constructed with the center position of the target object as the origin and any three mutually perpendicular directions (or two perpendicular directions) as coordinate axes. This disclosure does not limit the method of constructing the model coordinate system, and those skilled in the art can construct it according to their own needs. It is understood that the coordinates of the target object's model in the model coordinate system remain unchanged.

[0096] In some embodiments, model key points (such as...) can be set in the model of the target object. Figure 3 The model key points (including black dots) can mark the key positions of the target object. The key points of the model can highlight the characteristic information of the target object (such as orientation information, body shape information, appearance information, etc.). Those skilled in the art can set the model key points in the model of the target object according to their own needs. This disclosure does not limit this.

[0097] It is understandable that the key points of the above model may include at least four.

[0098] like Figure 3 As shown, at least one key point can be specified in the toy cat model at the tips of the two ears, mouth, back of the head, toes, etc.

[0099] In some embodiments, after setting the model key points of the target object, the coordinate values ​​of the model key points in the model coordinate system (i.e., model coordinate values) can also be obtained.

[0100] Step S204: Obtain the target image. The target image is obtained after image acquisition of the target object.

[0101] In some embodiments, an image of the target object can be acquired.

[0102] The target object can be in any pose within any scene. For example, the target object can be in a certain pose within a parcel sorting scene. This disclosure does not restrict the orientation, position, or other aspects of the target object in the target image.

[0103] In some embodiments, the target image may be a color map (such as an RGB image), a black and white image, a depth map (such as an RGBD image), etc., and this disclosure does not limit it.

[0104] In some embodiments, an image acquisition device can be used to capture the target object to obtain an image of the target. In some embodiments, capturing an image of the target object (e.g., a toy cat) using an image acquisition device can obtain an image such as... Figure 4 or Figure 5 The image shown may include the target object (such as...) Figure 4 The toy cat in the Chinese frame, or... Figure 5 (A toy cat shown in the image using point clouds).

[0105] The image acquisition device can be any device capable of image acquisition, such as a mobile phone, computer, camera, Kinect camera (a type of depth camera), etc. This disclosure does not impose any restrictions on it.

[0106] In some embodiments, a camera coordinate system can be constructed based on the image acquisition device of the target image, or it can be constructed based on the environment in which the image acquisition device of the target image is located. It is understood that as long as the coordinate values ​​of the image acquisition device of the target image do not change in the constructed coordinate system, then that coordinate system can be the camera coordinate system in this application. For example, a camera coordinate system can be constructed with the bottom of the image acquisition device as the center and any three mutually perpendicular directions as coordinate axes. This disclosure does not limit the method of constructing the camera coordinate system.

[0107] In some embodiments, the model coordinate system and the camera coordinate system may be the same or different, and this disclosure does not impose any restrictions on this.

[0108] Step S206: Perform keypoint offset prediction on the target image to determine the position offset of each point in the target image relative to the predicted keypoint in the camera coordinate system. The predicted keypoint is the keypoint predicted for the target object in the camera coordinate system.

[0109] In some embodiments, the position offset may include displacement offset and orientation offset.

[0110] In some embodiments, the predicted key points mentioned above may refer to key points predicted for the target object in the camera coordinate system. The predicted key points may correspond one-to-one with the key points in the model. The one-to-one correspondence may include the same number of key points or the same key point position (e.g., if the ear tip position is specified as a key point in the model of the target object, then a predicted key point can also be predicted at the ear tip position of the target object).

[0111] In some embodiments, the target image can be processed to predict the position offset of each point in the target image (which may be a pixel or a point cloud in the target image) relative to each predicted key point in the camera coordinate system.

[0112] For example, assuming that the predicted keypoints include a first predicted keypoint and a second predicted keypoint, then the positional offset of each point in the target image relative to the first predicted keypoint can be predicted, and the positional offset of each point in the target image relative to the second predicted keypoint can also be predicted.

[0113] In some embodiments, the target image can be processed by a bias prediction structure in the target network model to predict the positional bias of each point in the target image (which may be a pixel or a point cloud in the target image) relative to each predicted keypoint in the camera coordinate system. The bias prediction structure can be any neural network structure, such as a network structure composed of convolutional layers, fully connected layers, and a classifier. In short, any network structure capable of predicting the positional bias of each point relative to the predicted keypoint is within the scope of this disclosure.

[0114] Step S208: Determine the camera coordinates of the predicted key points in the camera coordinate system based on the positional offset of each point in the target image relative to the predicted key points in the camera coordinate system.

[0115] In some embodiments, the coordinates of each point in the target image in the camera coordinate system can be determined first, then multiple candidate camera coordinates can be determined for the predicted key point based on the positional offset of each point relative to the predicted key point, and finally the camera coordinates of the predicted key point can be determined based on the multiple candidate camera coordinates.

[0116] Those skilled in the art can determine the coordinates of each point in the target image in the camera coordinate system based on relevant technologies, which will not be elaborated in this embodiment.

[0117] In some embodiments, the target network model described above may further include a camera coordinate value determination structure. This structure determines the camera coordinate values ​​of the predicted keypoints in the camera coordinate system based on the positional offsets of each point in the target image relative to the predicted keypoints in the camera coordinate system. This disclosure does not limit the specific network structure within the camera coordinate value determination structure; any network structure capable of implementing the camera coordinate value prediction process is within the scope of this disclosure.

[0118] Step S210: The model coordinate values ​​of the model key points are concatenated with the camera coordinate values ​​of the predicted key points to obtain the fused coordinate values.

[0119] In some embodiments, the model coordinates of the model keypoints and the camera coordinates of the predicted keypoints can be concatenated to obtain fused coordinates; alternatively, features can be extracted from the model coordinates of the model keypoints and the camera coordinates of the predicted keypoints, and then the extracted features can be concatenated to directly obtain fused features.

[0120] In some embodiments, the target network model further includes a splicing structure, which can splice the model coordinates of the model key points with the camera coordinates of the predicted key points to obtain fused coordinates; it can also extract features from the model coordinates of the model key points and the camera coordinates of the predicted key points, and then splice the extracted features to directly obtain fused features.

[0121] This disclosure does not limit the specific network structure in the above splicing structure, and any network structure that can realize the above splicing process is within the protection scope of this disclosure.

[0122] Step S212: Determine the pose of the target object in the camera coordinate system based on the fused coordinate values ​​and the model of the target object.

[0123] In some embodiments, the predicted rotation and translation matrix between model keypoints and predicted keypoints can be determined based on the fused coordinate values ​​(or fused features).

[0124] In some embodiments, the target network model may further include a rotation and translation matrix prediction structure, which can be used to process the fused coordinate values ​​to determine the predicted rotation and translation matrix between the model key points and the predicted key points.

[0125] In some embodiments, the rotation and translation matrix prediction structure described above may include a fully connected layer and two one-dimensional convolutional layers. Similarly, this disclosure does not limit the specific network structure in the above rotation and translation matrix prediction structure; any network structure capable of implementing the above rotation and translation matrix prediction process is within the scope of this disclosure.

[0126] In some embodiments, after obtaining the above-mentioned predicted rotation and translation matrix, the model of the target object can be rotated and translated using the predicted rotation and translation matrix to determine the pose of the target object in the camera coordinate system.

[0127] Through the above embodiments, on the one hand, by rotating and translating the model of the target object to determine the pose of the target object in the camera coordinate system, the problem of being unable to determine the pose of the target object in the camera coordinate system due to multiple target occlusions in the target image can be avoided; on the other hand, the positions of the predicted key points corresponding to the model key points of the target object are known, so the position and orientation of the target object in the camera coordinate system can be determined according to the predicted key points corresponding to the model key points (for example, if the position of the first predicted key point on the ear and the position of the second predicted key point on the tail of the target object are known, then the orientation of the target object is known, and the target object can be grasped or other operations can be performed according to the orientation); in addition, when predicting the predicted key points of the target object, this disclosure first predicts the positional offset (including direction offset and position offset) of each point in the target image relative to the predicted key points, and then determines the position of the predicted key points according to the position offset. Since the direction offset in the position offset is within a certain preset range, there is no divergence problem in the prediction of the direction offset. Compared with the divergence problem that may be caused by directly determining the position of the predicted key points, this disclosure can improve the accuracy of the predicted key point position prediction, thereby improving the accuracy of target pose estimation.

[0128] Figure 6 This is a flowchart illustrating a method for determining the camera coordinate values ​​of predicted keypoints based on the positional offsets of each point in a target image relative to the predicted keypoints in the camera coordinate system, according to an exemplary embodiment.

[0129] refer to Figure 6 The above method for determining camera coordinates may include the following steps.

[0130] Step S602: Perform semantic segmentation on the target image to determine multiple target points in each point of the target image for describing the target object.

[0131] The aforementioned target points can be used to describe the target object. For example, they can be points on the target object itself, or points on the target object itself and its surroundings. This disclosure does not impose any restrictions on this.

[0132] In some embodiments, semantic segmentation can be performed on the target image to determine an image block that includes the target object.

[0133] like Figure 4As shown, if the target image is a one- or two-dimensional image, semantic segmentation can be used to determine, for example, the target image itself. Figure 4 The image block shown in the box may include the target object, and the points included in the image block may be the aforementioned multiple target points.

[0134] In some embodiments, if the target image is a depth image, then semantic segmentation can be used to determine, for example, depth images. Figure 5 The image block shown includes the target object, and the points included in the image block can be the aforementioned multiple target points.

[0135] In other embodiments, if the target image is an RGBD image, which may include a depth map and a color map, then semantic segmentation can be performed on the color map of the target image to obtain a detection box that includes the target object. Then, the color map and the depth map within the detection box are fused. Finally, semantic segmentation is performed on the fused color depth map to determine a region that includes the target object in the fused color depth map, and the points in the region are used as the target points in this embodiment.

[0136] In other embodiments, if the target image is an RGBD image, semantic segmentation of the target image can be performed using the following methods: feature extraction is performed on the target color map information to obtain color features; feature extraction is performed on the target depth map information to obtain depth features; the color features and depth features are fused to obtain color-depth fused features; and semantic segmentation processing is performed on the color-depth fused features to determine multiple target points in the target image used to describe the target object.

[0137] It should be noted that this disclosure does not impose any restrictions on the method for determining the target point.

[0138] Step S604: Determine the position offset of each target point relative to the predicted key point based on the position offset of each point relative to the predicted key point.

[0139] In some embodiments, the positional offset of each target point relative to the predicted keypoint can be filtered from the positional offsets of each point relative to the predicted keypoint.

[0140] Step S606: Determine multiple candidate camera coordinate values ​​for the predicted key points based on the positional offset of each target point relative to the predicted key points.

[0141] In some embodiments, the target image may be acquired by a target image acquisition device, and the camera coordinate system may be a coordinate system constructed based on the target image acquisition device. Therefore, the intrinsic parameter information of the target image acquisition device can be obtained, and the target image can be processed using this intrinsic parameter information to determine the coordinate values ​​of each target point (or each point in the target image) in the camera coordinate system.

[0142] In some embodiments, the coordinates of each target point in the camera coordinate system are known. Then, by combining the position offsets (including orientation offsets and displacement offsets) of each target point relative to the predicted key point, multiple candidate camera coordinate values ​​of the predicted key point can be determined (i.e., one target point can correspond to one candidate camera coordinate value, and one predicted key point can correspond to multiple candidate camera coordinate values).

[0143] Step S608: Determine the camera coordinates of the predicted keypoints in the camera coordinate system based on multiple candidate camera coordinates.

[0144] In some embodiments, the camera coordinates of the predicted keypoints in the camera coordinate system can be determined by the following methods: determining an average coordinate value based on multiple candidate camera coordinate values, and using the average coordinate value as the camera coordinates of the predicted keypoints in the camera coordinate system; or, performing clustering processing on multiple candidate camera coordinate values, and using the coordinate value corresponding to the cluster center as the camera coordinates of the predicted keypoints in the camera coordinate system.

[0145] In other embodiments, keypoint voting can be used to vote on the multiple candidate camera coordinate values ​​to determine the predicted camera coordinate values ​​of the keypoints in the camera coordinate system. Keypoint voting can include methods such as Gaussian standard voting, mean-shift clustering, and Hough voting; this disclosure does not limit the scope of these voting methods.

[0146] The technical solution provided in this embodiment determines the target points corresponding to the target object from the target image through semantic segmentation, and then predicts the camera coordinates of multiple candidate keypoints based on the positional offset of the target points, thereby predicting the camera coordinates of the keypoints. In predicting the camera coordinates of keypoints, this embodiment focuses on the target points most closely related to the target object through semantic segmentation, and then predicts the keypoints based on the positional offset of the target points, improving the prediction accuracy of the keypoints and thus improving the pose estimation accuracy of the target object.

[0147] Figure 7 This is a flowchart illustrating a method for determining camera coordinate values ​​using the Gaussian standard voting method according to an exemplary embodiment.

[0148] refer to Figure 7The above-mentioned camera coordinate value prediction method may include the following steps.

[0149] In some embodiments, the target object may include M predicted keypoints, where M is an integer greater than or equal to 1.

[0150] Step S702: Determine the mean and variance of the coordinate values ​​corresponding to the coordinate values ​​of multiple candidate cameras.

[0151] In some embodiments, it can be achieved through Determine the mean coordinates of multiple candidate camera coordinates for the j-th predicted keypoint, where This represents the coordinates of the i-th candidate camera for the j-th predicted keypoint, where i is an integer greater than or equal to 1, and j is an integer greater than or equal to 1 and less than or equal to M. mean() is a function that calculates the mean.

[0152] In some embodiments, it can be achieved through std() is a function that calculates the variance of the coordinates of multiple candidate cameras for the j-th predicted keypoint.

[0153] Step S704: Gaussian standardization is performed on the coordinate values ​​of each candidate camera based on the mean and variance of the coordinate values, so as to determine the convergence confidence of each candidate camera coordinate value based on the Gaussian standardization results.

[0154] In some embodiments, the mean of the coordinate values ​​can be subtracted from the candidate camera coordinate values, and then the ratio of the result to the variance of the coordinate values ​​(i.e., the Gaussian normalized value) can be calculated. Finally, the convergence confidence corresponding to the candidate camera coordinate values ​​can be determined based on the ratio.

[0155]

[0156] In some embodiments, the j-th predicted key point can be Gaussian standardized using formula (1) to obtain the Gaussian standardized value corresponding to the j-th key point.

[0157] Step S706: Remove candidate camera coordinate values ​​with convergence confidence greater than the target threshold to obtain converged camera coordinate values.

[0158]

[0159] In some embodiments, candidate camera coordinates with Gaussian normalized values ​​greater than a certain threshold can be considered to be relatively discrete and do not contribute much to the prediction of key point locations, and can therefore be removed.

[0160] Therefore, as shown in formula (2), the convergence confidence of Gaussian normalized values ​​greater than or equal to a certain threshold (e.g., θ = 0.6) can be set to 0, so that the corresponding candidate camera coordinate values ​​can be used to determine the convergence confidence. Delete, and set the convergence confidence of candidate camera coordinates with Gaussian normalization values ​​less than the threshold to 1.

[0161] Step S708: Determine the camera coordinates of the predicted key points in the camera coordinate system based on the converged camera coordinates and the convergence confidence of the converged camera coordinates.

[0162] In some embodiments, as shown in formula (3), the convergence confidence level can be used. The weights are the coordinates of each convergent camera. A weighted summation process is performed to determine the camera coordinates of the j-th predicted keypoint in the camera coordinate system. c k j .

[0163]

[0164] The technical solution provided in this embodiment performs Gaussian standardization on the coordinate values ​​of each candidate camera based on the mean and variance of all candidate camera coordinate values. Then, candidate camera coordinates whose Gaussian standardized values ​​are greater than a certain threshold are removed to eliminate candidate camera coordinates with relatively discrete distribution, thereby improving the prediction accuracy of the location of key points.

[0165] Figure 8 This is a position offset prediction method illustrated according to an exemplary embodiment.

[0166] In some embodiments, the target image described above may be a color depth map that includes target depth map information and target color map information.

[0167] So, for reference Figure 8 The positional offset of each point in the target image relative to the predicted keypoint in the camera coordinate system can be determined using the following method.

[0168] Step S802: Extract features from the target color map information to obtain color features.

[0169] Step S804: Extract features from the target depth map information to obtain depth features.

[0170] Step S806: Perform feature fusion of color features and depth features to obtain color depth fusion features.

[0171] Step S808: Perform keypoint offset prediction on the color depth fusion features to determine the position offset of each point in the target image relative to the predicted keypoint.

[0172] In some embodiments, the fused coordinate values ​​can be feature-extracted through a target convolutional structure to obtain fused coordinate features, and then the fused coordinate features can be predicted by a target classifier to determine the predicted rotation and translation matrix.

[0173] The improved technical solution in this embodiment combines both the color information and the depth information (i.e., geometric information) of the target object when performing position offset prediction, thereby improving the accuracy of position offset prediction.

[0174] Figure 9 This is a flowchart illustrating a method for determining a predicted rotation and translation matrix according to an exemplary embodiment.

[0175] In some embodiments, the predicted rotation and translation matrix may include a rotation matrix and a translation matrix.

[0176] In the following embodiments, the method for determining the rotation matrix will be used as an example for explanation. The method for determining the translation matrix can be referred to the method for determining the rotation matrix, and will not be repeated in this embodiment.

[0177] In some embodiments, the rotation matrix may include a first rotation direction, a second rotation direction, and a third rotation direction.

[0178] refer to Figure 9 The predicted rotation and translation matrix can be determined using the following method.

[0179] Step S902: The fused coordinate features are predicted by the target classifier to determine the first rotation data in the first rotation direction and the second rotation data in the second rotation direction.

[0180] In some embodiments, the fused coordinate features can be processed by a classifier in the target network model to determine the first rotation data r1 of the rotation and translation matrix in the first rotation direction (e.g., the x-direction) and the second rotation data r2 of the rotation matrix in the second rotation direction.

[0181] Step S904: Normalize the first rotation data to obtain first rotation normalized data.

[0182] In some embodiments, the first rotation data r1 can be normalized using the formula φ(r1) to obtain the first rotation-normalized data R1. Here, φ() is the normalization function.

[0183] Step S906: Perform a cross product on the first rotation normalized data and the second rotation data, and normalize the result after the cross product to obtain the third rotation normalized data of the rotation matrix in the third rotation direction.

[0184] In some embodiments, the first rotation-normalized data R1 and the second rotation data r2 can be cross-multiplied using the formula φ(R1×r2), and the result after cross-multiplication can be normalized to obtain the third rotation-normalized data R3 of the rotation matrix in the third rotation direction.

[0185] Step S908: Perform a cross product of the first rotation-normalized data and the third rotation-normalized data to obtain the second rotation-normalized data of the rotation matrix in the second rotation direction.

[0186] In some embodiments, the first rotation-normalized data R1 and the third rotation-normalized data R3 can be cross-multiplied using the formula R3×R1 to obtain the second rotation-normalized data R2 of the rotation matrix in the second rotation direction.

[0187] The technical solution provided in this embodiment can first predict the data in two rotation directions in the rotation matrix, and then predict the data in the third direction based on the data in the first two rotation directions. This can make the three predicted directions completely perpendicular, which is consistent with the actual situation of the three perpendicular directions.

[0188] Figure 10 This is an attitude estimation method illustrated according to an exemplary embodiment.

[0189] In some embodiments, the pose estimation method described above is performed by the target network model.

[0190] So, for reference Figure 10 The above attitude estimation method may include the following steps.

[0191] Step S1002: Obtain the model coordinate values ​​of the model key points on the target object's model in the model coordinate system.

[0192] Step S1004: Obtain the target image. The target image is obtained after image acquisition of the target object.

[0193] Step S1006: Perform keypoint offset prediction on the target image through the offset prediction structure of the target network model, and determine the position offset of each point in the target image relative to the predicted keypoint in the camera coordinate system. The predicted keypoint is the keypoint predicted for the target object in the camera coordinate system.

[0194] Step S1008: The camera coordinate value determination structure of the target network model determines the camera coordinate value of the predicted key point in the camera coordinate system based on the position offset of each point in the target image relative to the predicted key point in the camera coordinate system.

[0195] Step S1010: The splicing structure of the target network model splices the model coordinate values ​​of the model key points with the camera coordinate values ​​of the predicted key points to obtain the fused coordinate values.

[0196] In step S1012, the rotation and translation matrix prediction structure of the target network model determines the predicted rotation and translation matrix between the model key points and the predicted key points based on the fused coordinate values, so as to determine the pose of the target object in the camera coordinate system based on the model of the target object and the predicted rotation and translation matrix.

[0197] In other embodiments, multiple rotation and translation matrices can be obtained based on the one-to-one correspondence between model keypoints and predicted keypoints. Then, the least squares method is used to obtain an optimal rotation and translation matrix. Finally, the pose of the target object in the camera coordinate system is determined based on the optimal rotation and translation matrix.

[0198] However, the least squares method described above is an iterative method, which involves a large amount of computation and has low accuracy.

[0199] The method described above, which concatenates the model key points with the predicted key points and then uses a rotation and translation matrix to predict the concatenated result directly, uses a machine learning model. It provides the result directly in one calculation, has few parameters, fast calculation speed, and high accuracy.

[0200] Step S1014: Obtain the actual rotation and translation matrix between the model keypoints and the actual keypoints, where the actual keypoints are the keypoints of the target object in the target image in the actual pose of the target object in the camera coordinate system.

[0201] Step S1016: Determine the target loss value based on the predicted rotation and translation matrix and the actual rotation and translation matrix.

[0202] Step S1018: Train the target network model based on the target loss value.

[0203] The technical solution provided in this embodiment directly supervises the rotation and translation matrix during the training process of the target network model, rather than supervising the intermediate variables of the rotation and translation matrix. Since the error of the rotation and translation matrix directly reflects the error in pose estimation, the pose estimation method provided in this embodiment can provide accurate pose estimation.

[0204] Figure 11 This is a schematic diagram of an attitude estimation structure illustrated by an example.

[0205] In some embodiments, the attitude estimation method provided in this embodiment can be applied to products such as robotic arm grasping and AR.

[0206] This embodiment mainly introduces how to accurately identify the pose of a known object model in the camera coordinate system using target images, including RGB images and depth maps. The RGB images and depth maps are primarily obtained by acquiring images of the target object using an RGBD camera.

[0207] In some embodiments, RGB images can be used to extract color features through a first feature extraction module, wherein the first feature extraction module can be a deep network with ResNet (a residual network) as the skeleton and PSPNet (a network that uses an image pyramid structure for semantic segmentation) as the head.

[0208] In some embodiments, a point cloud can be generated from a depth map, and then its features can be extracted by a second feature extraction module to obtain depth features.

[0209] The second feature extraction module can be a point cloud network, Randla-net (a method for semantic segmentation of large scenes).

[0210] In some embodiments, the two modal information can be interacted at each layer of the encoder and decoder of the RGB and point cloud network. Then, the point-by-point features of the RGB and point cloud are concatenated together through a feature fusion module, and finally point-by-point keypoint bias estimation and semantic segmentation estimation are performed.

[0211] Keypoint offset estimation involves estimating the offset (including orientation offset and unique offset) of each point in the target image relative to the predicted keypoint in the camera coordinate system.

[0212] Semantic segmentation can refer to identifying image blocks containing target objects from a target image.

[0213] In some embodiments, an object detection network (such as YOLOv3) can first be used to detect the location of the object in the RGB image, and the depth map within the corresponding object detection box can be extracted and converted into a point cloud. Then, depth features and color features are extracted, and point-by-point semantic segmentation and keypoint bias prediction tasks are completed. After completing the keypoint bias prediction, a candidate point set for each predicted keypoint can be obtained based on the results of semantic segmentation. Then, Gaussian normalization is used to vote on the candidate keypoint set (the specific voting process can be referred to the corresponding parts of formulas (1) to (3), which will not be repeated in this embodiment), and the keypoints obtained after voting are concatenated with the keypoints on the object model, and the object's pose is finally regressed using this information.

[0214] After obtaining the predicted key points using the above method, the model key points in the aggregate coordinate system and the predicted key points in the camera coordinate system can be aggregated. Then, the aggregated results are passed through a regression network to obtain the 6D pose of the object. The specific process can be referred to formula (4).

[0215] (R,t)=G(cat{( c k1, m k1),…,( c k j , m k j )},Θ) (4)

[0216] in, c k j Identify the predicted keypoint coordinates in the camera coordinate system. m k j This represents the coordinates of the model's keypoints in the model coordinate system. For function G, a two-dimensional convolutional layer with a fully connected layer can be used (see details...). Figure 11 This is achieved through the network structure within the dashed box.

[0217] In some embodiments, the first two columns of the rotation matrix can be used to supervise the object's pose as the object's rotation information. See [reference needed] for details. Figure 9 The illustrated embodiment is used to predict the rotation and translation matrix.

[0218] In some embodiments, the average distance L of the object point cloud can be used. pose As supervisory information.

[0219]

[0220] in, and The results of the predictions (i.e., the prediction of the rotation matrix) and the predicted translation matrix And the true annotation (i.e., the rotation matrix in the true rotation and translation matrix) Translation matrix x∈M represents the point cloud representation of the object in the object coordinate system.

[0221] The technical solution provided in this embodiment improves the pose estimation effect of objects and realizes end-to-end pose estimation, which can be further applied to self-supervised pose estimation tasks.

[0222] This embodiment can use the ADD distance and ADD AUC of object pose estimation to measure the accuracy of pose estimation. The ADD distance is defined as follows.

[0223]

[0224] in, and The predicted results and ground truth labels are shown separately. v∈O represents the point cloud representation of the object in the object coordinate system. For symmetrical objects, we use ADD-S as the evaluation criterion. ADD-S is defined as follows.

[0225]

[0226] After recognizing the 6D pose of an object through image information, the object's complete position information can be obtained. This allows for manipulation of the object using prior information from the object model. Example 1: In object grasping or sorting scenarios, this method can accurately locate the object's orientation and position in 3D space. Based on this visual information, a robotic arm can accurately grasp the target object. Simultaneously, the robotic arm can place such objects in designated locations within a real-world scene, controlling the object's pose after placement to prevent collisions with other objects in the environment. Example 2: In some AR applications, after recognizing the pose of a known object, the position and pose of corresponding special effects can be adjusted based on the object's pose.

[0227] This disclosure describes a deep learning method for end-to-end pose estimation based on keypoint prediction. This method uses predicted keypoint biases of the object as intermediate variables and employs standardization to filter out high-confidence keypoint information before feeding it into the pose estimation network. This method ensures both accuracy and efficiency in pose estimation and recognition while maintaining end-to-end computation.

[0228] The technical solution provided in this embodiment has achieved good results on the object pose estimation datasets Linemod, LinemodOcculusion, and YCB-V. For details, please refer to... Figures 12-14 The results are shown.

[0229] pass Figures 12-14 The comparison structure shown reveals that the technical solution provided in this embodiment has a higher attitude estimation accuracy compared to other different attitude estimation schemes in the attitude estimation process of different target objects.

[0230] The table contains several proper nouns, and a Chinese-English glossary of these proper nouns is provided below.

[0231] Experiments on the online modulo dataset: EXPERIMENT ON LINEMOD OCCULUSION DATASET.

[0232] Convolutional Neural Network for 6D Pose Estimation: POseCNN.

[0233] Pixel-wise Voting Network for 6DoF Pose Estimation (PvNet)

[0234] Coordinates-Based Disentangled Pose Network for Real-Time RGB-Based 6-DoF Object Pose (CDPN).

[0235] 6D pose object detector and refiner (DPOD).

[0236] PointFusion Network.

[0237] 6D Pose Estimation Network: Dense-fusion.

[0238] Global to Local Network for Real-time 6D Pose Estimation with Embedding Vector Features, G2L-Net.

[0239] A Deep Point-wise 3D Keypoints Voting Network for 6DoF Pose (PVN3D).

[0240] Experiments on the online modal occultation dataset: EXPERIMENT ON LINEMOD OCCULUSION DATASET.

[0241] 6D object pose estimation with hybrid representation: HybridPose.

[0242] Single-stage network.

[0243] Pix2Pose: Pixel-Wise Coordinate Regression of Objects for 6D Pose Estimation.

[0244] Experiments on the YCB-V dataset: PERIMENT ON THE YCB-V DATASET.

[0245] Yale-CMU-Berkeley (YCB) Object and Modelset.

[0246] Average Distance of Model Points (ADD).

[0247] Average Closest Point Distance (ADD-S).

[0248] Deep continuous fusion for multi-sensor 3D object detection (DCF).

[0249] Based on the same inventive concept, this disclosure also provides an attitude estimation device, as shown in the following embodiment. Since the principle by which this device embodiment solves the problem is similar to that of the above-described method embodiment, the implementation of this device embodiment can refer to the implementation of the above-described method embodiment, and repeated details will not be described again.

[0250] Figure 15 This is a block diagram illustrating an attitude estimation device according to an exemplary embodiment. (Refer to...) Figure 15 The attitude estimation device 1500 provided in this embodiment may include: a model coordinate value acquisition module 1501, a target image acquisition module 1502, an offset prediction module 1503, a camera coordinate value prediction module 1504, a fusion coordinate determination module 1505, and an attitude prediction module 1506.

[0251] The module includes several key components: a model coordinate value acquisition module 1501, which acquires the model coordinate values ​​of key points on the target object's model in the model coordinate system; a target image acquisition module 1502, which acquires the target image after image acquisition of the target object; an offset prediction module 1503, which performs key point offset prediction on the target image to determine the position offset of each point in the target image relative to the predicted key points in the camera coordinate system, where the predicted key points are key points predicted for the target object in the camera coordinate system and correspond one-to-one with the model key points; a camera coordinate value prediction module 1504, which determines the camera coordinate values ​​of the predicted key points in the camera coordinate system based on the position offsets of each point in the target image relative to the predicted key points; a fusion coordinate determination module 1505, which concatenates the model coordinate values ​​of the model key points with the camera coordinate values ​​of the predicted key points to obtain fusion coordinate values; and a pose prediction module 1506, which determines the pose of the target object in the camera coordinate system based on the fusion coordinate values ​​and the model of the target object.

[0252] It should be noted that the aforementioned model coordinate value acquisition module 1501, target image acquisition module 1502, bias prediction module 1503, camera coordinate value prediction module 1504, fusion coordinate determination module 1505, and pose prediction module 1506 correspond to S202 to S212 in the method embodiment. The examples and application scenarios implemented by these modules and their corresponding steps are the same, but they are not limited to the content disclosed in the above method embodiment. It should be noted that these modules, as part of the apparatus, can be executed in a computer system such as a set of computer-executable instructions.

[0253] In some embodiments, the camera coordinate prediction module 1504 may include: a semantic segmentation submodule, a position offset prediction submodule, a candidate camera coordinate determination submodule, and a camera coordinate prediction submodule.

[0254] The semantic segmentation submodule can be used to perform semantic segmentation on the target image to determine multiple target points used to describe the target object among various points in the target image; the position offset prediction submodule can be used to determine the position offset of each target point relative to the predicted key point based on the position offset of each point relative to the predicted key point; the candidate camera coordinate value determination submodule can be used to determine multiple candidate camera coordinate values ​​of the predicted key point based on the position offset of each target point relative to the predicted key point; and the camera coordinate value prediction submodule can be used to determine the camera coordinate value of the predicted key point in the camera coordinate system based on the multiple candidate camera coordinate values.

[0255] In some embodiments, the camera coordinate prediction submodule may include: a variance determination unit, a Gaussian standardization unit, a culling unit, and a camera coordinate determination unit.

[0256] The variance determination unit can be used to determine the mean and variance of coordinate values ​​corresponding to multiple candidate camera coordinate values; the Gaussian standardization unit can be used to perform Gaussian standardization on each candidate camera coordinate value according to the mean and variance of coordinate values, so as to determine the convergence confidence of each candidate camera coordinate value based on the Gaussian standardization result; the elimination unit can be used to eliminate candidate camera coordinate values ​​with convergence confidence greater than the target threshold to obtain converged camera coordinate values; and the camera coordinate value determination unit can be used to determine the camera coordinate values ​​of the predicted key points in the camera coordinate system based on the converged camera coordinate values ​​and the convergence confidence of the converged camera coordinate values.

[0257] In some embodiments, the camera coordinate value determination unit may include a weighted summation subunit.

[0258] The weighted summation subunit can be used to perform weighted summation of each converged camera coordinate value with the convergence confidence as the weight, so as to determine the camera coordinate value of the predicted key point in the camera coordinate system.

[0259] In some embodiments, the camera coordinate prediction submodule may include: a mean determination unit or a clustering unit.

[0260] The mean determination unit can be used to determine the mean of coordinate values ​​based on multiple candidate camera coordinate values, and use the mean of coordinate values ​​as the camera coordinate values ​​of the predicted key points in the camera coordinate system; the clustering unit can be used to perform clustering processing on multiple candidate camera coordinate values, and use the coordinate values ​​corresponding to the cluster centers as the camera coordinate values ​​of the predicted key points in the camera coordinate system.

[0261] In some embodiments, the target image is acquired by a target image acquisition device, and the camera coordinate system is a coordinate system constructed based on the target image acquisition device; wherein, the candidate camera coordinate value determination submodule may include: an intrinsic parameter acquisition unit, a point coordinate determination unit, and a camera coordinate value determination unit.

[0262] The intrinsic parameter acquisition unit can be used to acquire the intrinsic parameter information of the target image acquisition device; the point coordinate determination unit can be used to process the target image through the intrinsic parameter information of the image acquisition device to determine the coordinate values ​​of each target point in the camera coordinate system; the camera coordinate value determination unit can be used to determine multiple candidate camera coordinate values ​​of the predicted key point based on the coordinate values ​​of each target point in the camera coordinate system and the position offset of each target point relative to the predicted key point.

[0263] In some embodiments, the target image is a color depth map including target depth map information and target color map information; wherein, the bias prediction module 1503 may include: a color feature acquisition submodule, a depth feature acquisition submodule, a feature fusion submodule, and a key point bias prediction submodule.

[0264] The color feature acquisition submodule can be used to extract features from the target color map information to obtain color features; the depth feature acquisition submodule can be used to extract features from the target depth map information to obtain depth features; the feature fusion submodule can be used to fuse color features and depth features to obtain color-depth fusion features; and the keypoint offset prediction submodule can be used to predict keypoint offsets based on the color-depth fusion features to determine the position offset of each point in the target image relative to the predicted keypoint.

[0265] In some embodiments, the attitude prediction module 1506 includes: a prediction rotation and translation matrix determination submodule and an attitude prediction submodule.

[0266] The prediction rotation and translation matrix determination submodule is used to determine the prediction rotation and translation matrix between the model key points and the prediction key points based on the fused coordinate values; the attitude prediction submodule is used to rotate and translate the model of the target object based on the prediction rotation and translation matrix to determine the attitude of the target object in the camera coordinate system.

[0267] In some embodiments, the predicted rotation and translation matrix determination submodule includes a feature extraction unit and a prediction processing unit.

[0268] The feature extraction unit is used to extract features from the fused coordinate values ​​through a target convolutional structure to obtain fused coordinate features; the prediction processing unit is used to predict the fused coordinate features through a target classifier to determine the predicted rotation and translation matrix.

[0269] In some embodiments, the predicted rotation and translation matrix includes a rotation matrix, which includes a first rotation direction, a second rotation direction, and a third rotation direction; wherein, the prediction processing unit includes: a first rotation data prediction subunit, a first normalization subunit, a second normalization subunit, and a second rotation data prediction subunit.

[0270] The first rotation data prediction subunit is used to predict the fused coordinate features using a target classifier to determine the first rotation data of the rotation matrix in the first rotation direction and the second rotation data in the second rotation direction. The first normalization subunit is used to normalize the first rotation data to obtain first rotation normalized data. The second normalization subunit is used to perform a cross product between the first rotation normalized data and the second rotation data, and normalize the result of the cross product to obtain the third rotation normalized data of the rotation matrix in the third rotation direction. The second rotation data prediction subunit is used to perform a cross product between the first rotation normalized data and the third rotation normalized data to obtain the second rotation normalized data of the rotation matrix in the second rotation direction.

[0271] In some embodiments, the pose estimation method is performed by a target network model; wherein the pose estimation apparatus may further include: an actual rotation and translation matrix acquisition module, a target loss value determination module, and a training module.

[0272] The module for obtaining the actual rotation and translation matrix can be used to obtain the actual rotation and translation matrix between the model keypoints and the actual keypoints. The actual keypoints are the keypoints of the target object in the target image in the actual pose in the camera coordinate system, and there is a one-to-one correspondence between the model keypoints and the actual keypoints. The module for determining the target loss value can be used to determine the target loss value based on the predicted rotation and translation matrix and the actual rotation and translation matrix. The module for training can be used to train the target network model based on the target loss value.

[0273] Since the functions of the apparatus 1500 have been described in detail in their respective method embodiments, they will not be repeated here.

[0274] The modules and / or sub-modules and / or units described in the embodiments of this disclosure can be implemented in software or hardware. The described modules and / or sub-modules and / or units can also be located in a processor. The names of these modules and / or sub-modules and / or units do not, in some cases, constitute a limitation on the module and / or sub-module and / or unit itself.

[0275] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0276] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0277] Figure 16 A schematic diagram of the structure of an electronic device suitable for implementing embodiments of the present disclosure is shown. It should be noted that... Figure 16 The illustrated electronic device 1600 is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0278] like Figure 16 As shown, the electronic device 1600 includes a central processing unit (CPU) 1601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1602 or a program loaded from a storage section 1608 into a random access memory (RAM) 1603. The RAM 1603 also stores various programs and data required for the operation of the electronic device 1600. The CPU 1601, ROM 1602, and RAM 1603 are interconnected via a bus 1604. An input / output (I / O) interface 1605 is also connected to the bus 1604.

[0279] The following components are connected to I / O interface 1605: an input section 1606 including a keyboard, mouse, etc.; an output section 1607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1608 including a hard disk, etc.; and a communication section 1609 including a network interface card such as a LAN card, modem, etc. The communication section 1609 performs communication processing via a network such as the Internet. A drive 1610 is also connected to I / O interface 1605 as needed. Removable media 1611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1610 as needed so that computer programs read from them can be installed into storage section 1608 as needed.

[0280] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1609, and / or installed from removable medium 1611. When the computer program is executed by central processing unit (CPU) 1601, it performs the functions defined above in the system of this disclosure.

[0281] It should be noted that the computer-readable storage medium disclosed herein may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable storage medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0282] In another aspect, this disclosure also provides a computer-readable storage medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable storage medium carries one or more programs that, when executed by the device, enable the device to perform the following functions: acquiring model coordinate values ​​of model keypoints on a model of a target object in a model coordinate system; acquiring a target image, which is obtained after image acquisition of the target object; performing keypoint offset prediction on the target image to determine the position offset of each point in the target image relative to the predicted keypoints in a camera coordinate system, wherein the predicted keypoints are keypoints predicted for the target object in the camera coordinate system, and the predicted keypoints correspond one-to-one with the model keypoints; determining the camera coordinate values ​​of the predicted keypoints in the camera coordinate system based on the position offsets of each point in the target image relative to the predicted keypoints; concatenating the model coordinate values ​​of the model keypoints with the camera coordinate values ​​of the predicted keypoints to obtain fused coordinate values; and determining the pose of the target object in the camera coordinate system based on the fused coordinate values ​​and the model of the target object.

[0283] According to one aspect of this disclosure, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in various optional implementations of the above embodiments.

[0284] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions of the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) and includes several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or smart device, etc.) to execute the method according to the embodiments of this disclosure, for example... Figure 2 , Figure 6 , Figure 7 , Figure 8 , Figure 9 or Figure 10 One or more of the steps shown in the diagram.

[0285] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not claimed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0286] It should be understood that this disclosure is not limited to the detailed structures, drawing arrangements or implementations shown herein; rather, this disclosure is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.

Claims

1. A pose estimation method, characterized in that, include: Obtain the model coordinates of the key points on the target object's model in the model coordinate system; Acquire a target image, wherein the target image is obtained by acquiring an image of the target object; Perform keypoint offset prediction on the target image to determine the position offset of each point in the target image relative to the predicted keypoint in the camera coordinate system. The predicted keypoint is the keypoint predicted for the target object in the camera coordinate system. Based on the positional offset of each point in the target image relative to the predicted key point in the camera coordinate system, the camera coordinate values ​​of the predicted key point in the camera coordinate system are determined. The model coordinate values ​​of the model key points are concatenated with the camera coordinate values ​​of the predicted key points to obtain fused coordinate values; The fused coordinate values ​​are feature extracted using a target convolutional structure to obtain fused coordinate features. The fused coordinate features are predicted using a target classifier to determine the predicted rotation and translation matrix between the model keypoints and the predicted keypoints. The predicted rotation and translation matrix includes a rotation matrix with a first rotation direction, a second rotation direction, and a third rotation direction. The process involves: predicting the fused coordinate features using a target classifier to determine first rotation data in the first rotation direction and second rotation data in the second rotation direction; normalizing the first rotation data to obtain first rotation-normalized data; performing a cross product between the first and second rotation data and normalizing the result to obtain third rotation-normalized data of the rotation matrix in the third rotation direction; and performing a cross product between the first and third rotation-normalized data to obtain second rotation-normalized data of the rotation matrix in the second rotation direction. The target object model is rotated and translated according to the predicted rotation and translation matrix to determine the pose of the target object in the camera coordinate system.

2. The method according to claim 1, characterized in that, Based on the positional offset of each point in the target image relative to the predicted keypoint in the camera coordinate system, the camera coordinate values ​​of the predicted keypoint in the camera coordinate system are determined, including: Semantic segmentation is performed on the target image to determine multiple target points for describing the target object among various points in the target image; The positional offset of each target point relative to the predicted key point is determined based on the positional offset of each point relative to the predicted key point; Based on the positional offset of each target point relative to the predicted key point, multiple candidate camera coordinate values ​​of the predicted key point are determined; The camera coordinates of the predicted keypoint in the camera coordinate system are determined based on the multiple candidate camera coordinates.

3. The method according to claim 2, characterized in that, Determining the camera coordinates of the predicted keypoint in the camera coordinate system based on the plurality of candidate camera coordinates includes: Determine the mean and variance of the coordinate values ​​corresponding to the multiple candidate camera coordinate values; Gaussian standardization is performed on the coordinate values ​​of each candidate camera based on the mean and variance of the coordinate values, so as to determine the convergence confidence of each candidate camera coordinate value based on the Gaussian standardization results. Candidate camera coordinates with convergence confidence greater than the target threshold are removed to obtain converged camera coordinates. The camera coordinates of the predicted keypoints in the camera coordinate system are determined based on the converged camera coordinates and the convergence confidence of the converged camera coordinates.

4. The method according to claim 3, characterized in that, Determining the camera coordinates of the predicted keypoints in the camera coordinate system based on the converged camera coordinates and the convergence confidence of the converged camera coordinates includes: The convergence confidence level is used as the weight to perform a weighted summation of the coordinate values ​​of each converged camera to determine the camera coordinate values ​​of the predicted key points in the camera coordinate system.

5. The method according to claim 2, characterized in that, Determining the camera coordinates of the predicted keypoint in the camera coordinate system based on the plurality of candidate camera coordinates includes: The mean coordinate value is determined based on the multiple candidate camera coordinate values, and this mean coordinate value is used as the camera coordinate value of the predicted keypoint in the camera coordinate system; or... The multiple candidate camera coordinate values ​​are clustered, and the coordinate values ​​corresponding to the cluster centers are used as the camera coordinate values ​​of the predicted key points in the camera coordinate system.

6. The method according to claim 2, characterized in that, The target image is acquired by a target image acquisition device, and the camera coordinate system is a coordinate system constructed based on the target image acquisition device; wherein, determining multiple candidate camera coordinate values ​​of the predicted keypoint based on the positional offset of each target point relative to the predicted keypoint includes: Obtain the intrinsic parameter information of the target image acquisition device; The target image is processed using the intrinsic parameter information of the image acquisition device to determine the coordinate values ​​of each target point in the camera coordinate system; Based on the coordinate values ​​of each target point in the camera coordinate system and the positional offset of each target point relative to the predicted key point, the multiple candidate camera coordinate values ​​of the predicted key point are determined.

7. The method according to claim 1, characterized in that, The target image is a color depth map including target depth map information and target color map information; wherein, keypoint offset prediction is performed on the target image to determine the position offset of each point in the target image relative to the predicted keypoint in the camera coordinate system, including: Feature extraction is performed on the target color map information to obtain color features; Feature extraction is performed on the target depth map information to obtain depth features; The color features and the depth features are fused to obtain color-depth fused features; Keypoint offset prediction is performed on the color depth fusion features to determine the position offset of each point in the target image relative to the predicted keypoint.

8. The method according to claim 1, characterized in that, The pose estimation method is performed by the target network model; wherein, the pose estimation method further includes: Obtain the actual rotation and translation matrix between the model key points and the actual key points, wherein the actual key points are the key points of the target object in the target image in the actual pose of the target object in the camera coordinate system; The target loss value is determined based on the predicted rotation and translation matrix and the actual rotation and translation matrix; The target network model is trained based on the target loss value.

9. An attitude estimation device, characterized in that, include: The model coordinate value acquisition module is used to obtain the model coordinate values ​​of the model key points on the target object's model in the model coordinate system. The target image acquisition module is used to acquire a target image, which is obtained by image acquisition of the target object. The offset prediction module is used to perform keypoint offset prediction on the target image, and determine the position offset of each point in the target image relative to the predicted keypoint in the camera coordinate system. The predicted keypoint is the keypoint predicted for the target object in the camera coordinate system. The camera coordinate prediction module is used to determine the camera coordinate values ​​of the predicted key points in the camera coordinate system based on the positional offset of each point in the target image relative to the predicted key points in the camera coordinate system. The fusion coordinate determination module is used to concatenate the model coordinate values ​​of the model key points with the camera coordinate values ​​of the predicted key points to obtain fusion coordinate values; A pose prediction module, used to determine the pose of the target object in the camera coordinate system based on the fused coordinate values ​​and the model of the target object, includes: The prediction rotation and translation matrix determination submodule includes: The feature extraction unit is used to extract features from the fused coordinate values ​​through the target convolutional structure to obtain fused coordinate features; The prediction processing unit is used to perform prediction processing on the fused coordinate features through a target classifier to determine the prediction rotation and translation matrix between the model key points and the prediction key points; The pose prediction submodule is used to rotate and translate the model of the target object according to the predicted rotation and translation matrix, and to determine the pose of the target object in the camera coordinate system. The predicted rotation and translation matrix includes a rotation matrix, which includes a first rotation direction, a second rotation direction, and a third rotation direction; wherein, the prediction processing unit includes: The first rotation data prediction subunit is used to predict the fused coordinate features through the target classifier to determine the first rotation data of the rotation matrix in the first rotation direction and the second rotation data in the second rotation direction; The first normalization subunit is used to normalize the first rotation data to obtain the first rotation normalized data. The second normalization subunit is used to perform a cross product of the first rotation normalized data and the second rotation data, and to normalize the result after the cross product to obtain the third rotation normalized data of the rotation matrix in the third rotation direction. The second rotation data prediction subunit is used to perform a cross product of the first rotation normalized data and the third rotation normalized data to obtain the second rotation normalized data of the rotation matrix in the second rotation direction.

10. The apparatus according to claim 9, characterized in that, The pose estimation method is performed by the target network model; wherein, the apparatus further includes: The actual rotation and translation matrix acquisition module is used to acquire the actual rotation and translation matrix between the model key points and the actual key points, wherein the actual key points are the key points of the target object in the target image in the actual pose of the camera coordinate system. The target loss value determination module is used to determine the target loss value based on the predicted rotation and translation matrix and the actual rotation and translation matrix. The training module is used to train the target network model based on the target loss value.

11. An electronic device, characterized in that, include: Memory; as well as A processor coupled to the memory, the processor being used to execute the attitude estimation method as described in any one of claims 1-8 based on instructions stored in the memory.

12. A computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the attitude estimation method as described in any one of claims 1-8.

13. A computer program product comprising computer instructions stored in a computer-readable storage medium, characterized in that, When the computer instructions are executed by the processor, they implement the method of any one of claims 1-8.

Images