A pose estimation method, a pose estimation device, an electronic device, and a storage medium

Abstract

The application discloses a pose estimation method, a pose estimation device, an electronic device and a computer readable storage medium. The method comprises the following steps: target detection is performed on a to-be-processed image, and position information and category information of a target object contained in the to-be-processed image are determined; a first target two-dimensional feature of the target object is extracted according to the position information; in a plurality of object point cloud models obtained through offline training, a target point cloud model corresponding to the target object is determined according to the category information; and pose estimation is performed on the target object according to the first target two-dimensional feature and the target point cloud model, so that pose information of the target object is obtained. Through the scheme, the pose of the object can be quickly and accurately estimated.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a pose estimation method, a pose estimation device, an electronic device, and a computer readable storage medium. BACKGROUND

[0002] Object grasping and augmented reality are common work scenarios for robots. In these work scenarios, robots need to interact with different objects in the real environment. To achieve interaction with objects, robots usually obtain 3D position information of objects relative to the robot camera based on 2D target detection technology and depth information collected by an RGB-D camera.

[0003] However, in the case of only having 3D position information of an object, since the pose information of the object is unknown, when the object is placed obliquely or laterally, the robot may default to a normal flat pose of the object to perform interaction operations, resulting in misalignment of interaction. SUMMARY

[0004] The present application provides a pose estimation method, a pose estimation device, an electronic device, and a computer readable storage medium, which can quickly and accurately estimate the pose of an object.

[0005] In a first aspect, the present application provides a pose estimation method, comprising:

[0006] performing target detection on a to-be-processed image to determine position information and category information of a target object contained in the to-be-processed image;

[0007] extracting a first target two-dimensional feature of the target object according to the position information;

[0008] determining a target point cloud model corresponding to the target object from a plurality of object point cloud models trained offline according to the category information;

[0009] performing pose estimation on the target object according to the first target two-dimensional feature and the target point cloud model to obtain pose information of the target object.

[0010] In a second aspect, the present application provides a pose estimation device, comprising:

[0011] a detection module configured to perform target detection on a to-be-processed image to determine position information and category information of a target object contained in the to-be-processed image;

[0012] an extraction module configured to extract a first target two-dimensional feature of the target object according to the position information;

[0013] a determination module configured to determine a target point cloud model corresponding to the target object from a plurality of object point cloud models trained offline according to the category information.

[0014] an estimation module configured to perform pose estimation on the target object according to the first target two-dimensional feature and the target point cloud model to obtain pose information of the target object.

[0015] In a third aspect, the present application provides an electronic device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the method of the first aspect when executing the computer program.

[0016] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program implements the steps of the method of the first aspect when executed by a processor.

[0017] In a fifth aspect, the present application provides a computer program product, which comprises a computer program, and the computer program implements the steps of the method of the first aspect when executed by one or more processors.

[0018] Compared with the prior art, the present application has the beneficial effects that: the present scheme first uses a simple target detection algorithm to determine the position information and category information of the target object; then extracts the first target two-dimensional feature of the target object according to the position information, and calls the target point cloud model corresponding to the target object based on the category information; finally, the pose estimation of the target object can be performed according to the first target two-dimensional feature and the target point cloud model. Based on the above process, it can be known that the present scheme obtains the object point cloud models corresponding to objects of different categories through pre-off-line training, and only needs to correspondingly call according to the category information of the target object when pose estimation is needed, thereby improving the generalization ability and processing speed of pose estimation, and the real-time inference of pose estimation can be realized on embedded devices such as robots,

[0019] It can be understood that the beneficial effects of the second aspect to the fifth aspect can be referred to the related description in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0021] Figure 1 is an implementation flow diagram of the pose estimation method provided by the embodiments of the present application;

[0022] Figure 2 is a schematic diagram of a poly-attention layer provided by an embodiment of the present application;

[0023] Figure 3 is a schematic diagram of a self-attention layer provided by an embodiment of the present application;

[0024] Figure 4 is a schematic diagram of a cross-attention layer provided by an embodiment of the present application;

[0025] Figure 5 is a multi-input network infrastructure provided by an embodiment of the present application;

[0026] Figure 6 is an example flowchart of an implementation of a pose estimation method provided by an embodiment of the present application;

[0027] Figure 7 is an example flowchart of an implementation of an offline training process provided by an embodiment of the present application;

[0028] Figure 8 is a structural block diagram of a pose estimation apparatus provided by an embodiment of the present application;

[0029] Figure 9 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0030] In the following description, for the purpose of explanation and not limitation, specific details are set forth, such as particular system configurations, techniques, etc., in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

[0031] A pose estimation method provided by an embodiment of the present application is described below. The pose estimation method can be applied to an electronic device. Specifically, the electronic device can be a robot; of course, the electronic device can also be another device, such as a user terminal or a server, which establishes a communication connection with the robot and is capable of controlling the robot, and the present application is not limited in this regard. For ease of description, the pose estimation method is explained and described below by taking the case where the pose estimation method is applied to a robot. Please refer to Figure 1 The pose estimation method in the present application includes:

[0032] In step 101, target detection is performed on a to-be-processed image to determine position information and category information of a target object contained in the to-be-processed image.

[0033] The robot is equipped with a camera which can keep running synchronously after the robot is started. In the case that the robot has interaction demand, it can obtain the environment image of the environment where it is located through the camera, and the environment image can be determined as the to-be-processed image. Of course, the robot can also obtain the to-be-processed image based on the camera in the simulation platform such as webots. The embodiment of the present application does not limit the acquisition mode of the to-be-processed image.

[0034] The robot can perform target detection on the to-be-processed image, and the target detection algorithm used by the robot is not limited here. Through target detection, the position information and the category information of the target object contained in the to-be-processed image can be determined.

[0035] The position information specifically refers to the coordinates of a specified vertex (for example, the top-left vertex) of the bounding box of the target object in the to-be-processed image, and the size of the bounding box. For example, the position information can be specifically in the form of (x, y, w, h), where x is the horizontal coordinate of the specified vertex of the bounding box of the target object in the to-be-processed image, y is the vertical coordinate of the specified vertex of the bounding box of the target object in the to-be-processed image, w is the width of the bounding box of the target object, and h is the height of the bounding box of the target object.

[0036] The category information specifically refers to the category ID of the object category to which the target object belongs. It should be noted that the object categories are divided in detail in the embodiment of the present application. For example, the beverage of A brand is an object category, and the beverage of B brand is another object category, and so on. Each object category actually only has one object, and the shape and other characteristics of the object are basically fixed.

[0037] It can be understood that the number of target objects is not limited in the embodiment of the present application. In the case that multiple objects are placed in the environment where the camera is shooting, through target detection, multiple objects contained in the corresponding to-be-processed image can be determined; the robot can determine part or all of the multiple objects as target objects according to its own interaction demand, and can perform each of the subsequent steps on each target object to obtain the pose information of each target object.

[0038] In step 102, a first target two-dimensional feature of the target object is extracted according to the position information.

[0039] The main purpose of the robot is to interact with the target object, and the image to be processed often contains other information in addition to the target object currently considered, such as background information and information of other target objects. To avoid interference or influence of these other information on the pose estimation of the target object currently considered, the robot can extract the two-dimensional features of the target object according to the obtained position information of the target object. For convenience of distinction, the two-dimensional features are referred to as first target two-dimensional features.

[0040] In step 103, among the plurality of object point cloud models obtained through offline training, a target point cloud model corresponding to the target object is determined according to the category information.

[0041] The robot has obtained, through offline training, a plurality of object point cloud models corresponding to a plurality of objects, wherein the plurality of objects belong to different object categories, that is, the category information of the plurality of objects is different. Based on this, the robot can find, among the plurality of object point cloud models obtained through offline training, an object point cloud model corresponding to the target object according to the category information of the target object currently considered. For convenience of distinction, the object point cloud model is referred to as a target point cloud model.

[0042] In step 104, the pose of the target object is estimated according to the first target two-dimensional features and the target point cloud model, and the pose information of the target object is obtained.

[0043] The target point cloud model contains information of the target object in three-dimensional space, and the first target two-dimensional features contain information of the target object in two-dimensional plane; the robot can combine the two and perform pose estimation through matching between the two, thereby obtaining the pose information of the target object. It can be understood that the pose information actually contains three-dimensional position information and three-dimensional attitude information of the target object; that is, the pose information is specifically 6D pose information.

[0044] In some embodiments, in order to improve the accuracy of pose estimation and obtain a more accurate pose estimation result, each object point cloud model is associated with two-dimensional features of the corresponding object; that is, when the object point cloud model of each object is trained offline, the two-dimensional features of the object are also associated and saved with the object point cloud model. Based on this optimization, step 104 can specifically include:

[0045] A1, determining target three-dimensional features of the target object according to the target point cloud model.

[0046] As described above, the target point cloud model contains information of the target object in three-dimensional space, so the robot can extract three-dimensional features of the target object according to the target point cloud model. For convenience of distinction, the three-dimensional features are referred to as target three-dimensional features.

[0047] A2, calling a pose estimation model adopting a graph attention mechanism according to the first target two-dimensional feature, the target three-dimensional feature, and the second target two-dimensional feature, to obtain pose information of the target object output by the pose estimation model.

[0048] The embodiment of the present application optimizes the pose estimation model adopting the ordinary pose estimation algorithm, and applies the graph attention mechanism on the basis of the pose estimation model. Through the graph attention mechanism, each layer of the pose estimation model can focus on the node that can highlight the importance of the feature in the processing process. The input of the optimized pose estimation model includes: the first target two-dimensional feature, the target three-dimensional feature, and the second target two-dimensional feature, wherein the second target two-dimensional feature is the two-dimensional feature associated with the target point cloud model, which will not be described here.

[0049] For example only, the pose estimation algorithm adopted by the pose estimation model can be a PnP (Perspective-n-Point) algorithm, and the embodiment of the present application does not limit this.

[0050] For example only, in the embodiment of the present application, the two-dimensional features concerned by the robot can include color features and texture features, etc.; and the three-dimensional features concerned by the robot can include the features of the point cloud in the three-dimensional space.

[0051] In some embodiments, the pose estimation model adopting the graph attention mechanism includes an aggregation attention layer, a self-attention layer, and a cross-attention layer; wherein the aggregation attention layer is used to strengthen the relationship between the second target two-dimensional feature and the target three-dimensional feature; the self-attention layer is used to strengthen the relationship between each target three-dimensional feature and the relationship between each first target two-dimensional feature, that is, to strengthen the relationship between the features in the same dimension; and the cross-attention layer is used to strengthen the relationship between the target three-dimensional feature and the first target two-dimensional feature.

[0052] Let the second target two-dimensional feature be F t 2D , the target three-dimensional feature be F 3D , the weight layer corresponding to the second target two-dimensional feature be w 2d , and the weight layer corresponding to the target three-dimensional feature be w 3d , then the aggregation attention layer can be defined as follows:

[0053] Agg(F t 2D ,F 3D )=F 3D +α·F t 2D

[0054] α=softmax(w 2D ·F t 2D +w3D • F 3D )

[0055] Let the first target two-dimensional feature be denoted as F The aggregation attention layer, the self-attention layer and the cross-attention layer can be defined as follows, respectively:

[0056]

[0057] Please refer to Figure 2 , Figure 3 and Figure 4 , Figure 2 The schematic of the aggregation attention layer is given, Figure 3 The schematic of the self-attention layer is given, Figure 4 The schematic of the cross-attention layer is given. As can be seen from the drawings, the aggregation attention layer utilizes the existing mutual connection between the nodes in the graph, enhances the effective information in the graph, and retains more distinctive features; the self-attention layer utilizes the features of adjacent nodes to assign weights to the correlation between different nodes in the same class.

[0058] Please refer to Figure 5 , Figure 5 The multi-input network basic structure of any attention layer of the graph attention mechanism is given. Among them, the Conv layer performs convolution operation; the MatMul layer performs matrix multiplication operation; the Concat layer performs splicing operation; value and x represent the input features, which are different according to the different current attention layers. Specifically, for the aggregation attention layer, one of value and x is the second target two-dimensional feature F t 2D , and the other is the target three-dimensional feature F 3D ; for the three-dimensional self-attention layer, both value and x are the target three-dimensional feature F 3D ; for the two-dimensional self-attention layer, both value and x are the first target two-dimensional feature For the cross-attention layer, one of value and x is the first target two-dimensional feature , and the other is the target three-dimensional feature F 3D .

[0059] In some embodiments, the robot can also obtain the full Figure Two dimensional feature of the image to be processed through the target detection on the image to be processed in step 101; that is, the input of the target detection also includes the full Figure Two dimensional feature of the image to be processed. Considering that the target object does not actually occupy the entire frame of the image to be processed, step 102 can include:

[0060] B1, determining the object region corresponding to the target object in the image to be processed according to the position information.

[0061] From the definition of the position information, it can be seen that the position information actually describes the position of the bounding box of the target object in the to-be-processed image. Thus, by the position information, the robot can determine the region framed by the bounding box in the to-be-processed image, which is the object region corresponding to the target object in the to-be-processed image.

[0062] B2, extracting, according to the object region, a first target two-dimensional feature of the target object from the full- Figure Two dimensional feature.

[0063] Obviously, the full- Figure Two dimensional feature contains two-dimensional features of the entire frame of the to-be-processed image, which leads to the presence of some feature information irrelevant to the target object in the full- Figure Two dimensional feature. Based on this, the robot can extract corresponding two-dimensional features from the full- Figure Two dimensional feature according to the determined object region, and these two-dimensional features are the first target two-dimensional features of the target object.

[0064] Considering the case that the edge of the target object may not be completely included in the bounding box, the robot can first expand the object region based on a preset expansion ratio to obtain a target region. It can be understood that the target region is larger than the object region, so that the object region can be completely contained in the target region. Then, the robot only needs to determine the two-dimensional features corresponding to the target region in the full- Figure Two dimensional feature as the first target two-dimensional features of the target object.

[0065] It can be understood that by the above operation, the data processing amount of the robot can be reduced, and the pose estimation of multiple target objects in the to-be-processed image can also be realized, reducing the interference caused by the features of different target objects.

[0066] Based on the above-described steps, please refer to Figure 6 , Figure 6 for a flow example of the pose estimation method proposed in the embodiments of the present application.

[0067] In some embodiments, the robot or other types of electronic devices can obtain the object point cloud model of each object through offline training in advance. The process of offline training includes:

[0068] C1, obtaining multiple images of an object, the multiple images being obtained by a camera continuously shooting around the object.

[0069] A user can control the camera to face the object and surround the object for one round. During the process of the camera surrounding the object, the camera can continuously shoot, thereby obtaining multiple images surrounding the object for one round.

[0070] C2. Perform 3D reconstruction on multiple images to obtain the object's point cloud model.

[0071] Robots or other types of electronic devices can perform 3D reconstruction from multiple images. As an example only, SFM (Structure from Motion) or other 3D reconstruction methods can be used; no specific method is specified here. Through 3D reconstruction, a point cloud model of the object can be obtained.

[0072] C3. Name the object point cloud model according to the object's category information.

[0073] To ensure that object category information can be associated with the object's point cloud model, the object point cloud model can be named according to its category information. For example, if the category ID of an object is 00001, then the object point cloud model can be named 00001. This allows the robot to quickly find the target point cloud model corresponding to the target object based on its category information.

[0074] In some embodiments, the above offline training process may further include:

[0075] D1. Perform image enhancement on multiple images to obtain enhanced images.

[0076] For illustrative purposes only, image enhancement methods include, but are not limited to, mixup and mosaic techniques; they are not limited here.

[0077] D2. Extract the two-dimensional features of objects from the enhanced multiple images.

[0078] The two-dimensional features of objects in multiple enhanced images can be extracted using any lightweight object detection algorithm. For example, this lightweight object detection algorithm could be MobileNetv3 or YOLO, etc., and is not limited here. Robots or other types of electronic devices can use publicly available pre-trained models of this object detection algorithm, or they can train the model using commonly available training data.

[0079] In practical applications, if a publicly available pre-trained model of the object detection algorithm is used, the model can be fine-tuned based on multiple enhanced images. Only a few rounds of training are needed to quickly obtain a model that can recognize the current object and has stronger generalization ability. Since this model is a lightweight object detection model, only a small number of images (i.e., multiple images obtained around the object) are required to complete the fine-tuning of the model. Therefore, this fine-tuning operation can be performed directly on a robot equipped with a graphics processing unit (GPU).

[0080] D3, a relationship between the object point cloud model of the object and the two-dimensional feature of the object.

[0081] In the offline training process, the robot or other types of electronic devices can save the relationship between the object point cloud model of the object and the two-dimensional feature of the object for subsequent application of the object point cloud model.

[0082] It can be understood that in the offline training process, for any object, the three-dimensional feature of the object can be obtained through the obtained object point cloud model; the two-dimensional feature of the object can be obtained through a target detection algorithm or other ways; the above two types of data can be used as inputs of the pose estimation model to be trained, and the pose estimation model is used for feature matching of the two-dimensional feature and the three-dimensional feature to generate a correspondence graph, so as to realize the training of the pose estimation model. Please refer to Figure 7 , Figure 7 The flowchart of the offline training process proposed in the embodiments of the present application is given.

[0083] As can be seen from the above, in the embodiments of the present application, first, the position information and the category information of the target object are determined using a simple target detection algorithm; then the first target two-dimensional feature of the target object is extracted according to the position information, and the target point cloud model corresponding to the target object is called based on the category information; finally, the pose of the target object can be estimated according to the first target two-dimensional feature and the target point cloud model. Based on the above process, it can be known that the object point cloud model corresponding to multiple objects of different categories is obtained through pre-offline training in the embodiments of the present application, and when pose estimation is needed, the corresponding calling is only needed according to the category information of the target object, thereby improving the generalization ability and processing speed of the pose estimation, and the real-time inference of the pose estimation can be realized on the embedded device such as the robot,

[0084] It should be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0085] Corresponding to the pose estimation method provided in the above, the embodiments of the present application also provide a pose estimation device. The above pose estimation device can be integrated into an electronic device. Specifically, the electronic device can be a robot; of course, the electronic device can also be other devices that establish communication connection with the robot and can control the robot, and the embodiments of the present application do not limit this. Please refer to Figure 8 , the pose estimation device 8 in the embodiments of the present application includes:

[0086] The detection module 801 is configured to perform target detection on the to-be-processed image, and determine position information and category information of a target object contained in the to-be-processed image.

[0087] The extraction module 802 is configured to extract a first target two-dimensional feature of the target object according to the position information.

[0088] The determination module 803 is configured to determine, from a plurality of object point cloud models obtained through offline training, a target point cloud model corresponding to the target object according to the category information.

[0089] The estimation module 804 is configured to perform pose estimation on the target object according to the first target two-dimensional feature and the target point cloud model, and obtain pose information of the target object.

[0090] In some embodiments, each object point cloud model is associated with a two-dimensional feature of a corresponding object; and the estimation module 804 includes:

[0091] The first determination unit is configured to determine a target three-dimensional feature of the target object according to the target point cloud model.

[0092] The calling unit is configured to call a pose estimation model adopting a graph attention mechanism according to the first target two-dimensional feature, the target three-dimensional feature, and a second target two-dimensional feature, to obtain the pose information of the target object output by the pose estimation model, where the second target two-dimensional feature is the two-dimensional feature associated with the target point cloud model.

[0093] In some embodiments, the pose estimation model adopting the graph attention mechanism includes an aggregation attention layer, a self-attention layer, and a cross-attention layer.

[0094] The aggregation attention layer is configured to strengthen the relationship between the second target two-dimensional feature and the target three-dimensional feature.

[0095] The self-attention layer is configured to strengthen the relationship between each target three-dimensional feature and the relationship between each first target two-dimensional feature.

[0096] The cross-attention layer is configured to strengthen the relationship between the target three-dimensional feature and the first target two-dimensional feature.

[0097] In some embodiments, the target detection further determines a global feature of the to-be-processed image; Figure Two The extraction module 802 includes:

[0098] The second determination unit is configured to determine an object region corresponding to the target object in the to-be-processed image according to the position information.

[0099] The first extraction unit is configured to extract the first target two-dimensional feature of the target object from the global feature according to the object region. Figure Two ​

[0100] In some embodiments, the extraction unit comprises:

[0101] The region expansion subunit is configured to expand the object region outward based on a preset expansion ratio to obtain a target region.

[0102] The feature determination subunit is configured to determine, in the full-dimensional features, a two-dimensional feature corresponding to the target region as a first target two-dimensional feature of the target object. Figure Two

[0103] In some embodiments, the pose estimation device 8 further comprises a training module configured to obtain, for each object, an object point cloud model corresponding to the object through offline training.

[0104] The training module comprises:

[0105] The acquisition unit is configured to acquire a plurality of images of the object, the plurality of images being obtained by a camera continuously capturing around the object.

[0106] The reconstruction unit is configured to perform three-dimensional reconstruction on the plurality of images to obtain an object point cloud model of the object.

[0107] The naming unit is configured to name the object point cloud model according to category information of the object.

[0108] In some embodiments, the training module further comprises:

[0109] The enhancement unit is configured to perform image enhancement on the plurality of images to obtain enhanced plurality of images.

[0110] The second extraction unit is configured to extract a two-dimensional feature of the object in the enhanced plurality of images.

[0111] The saving unit is configured to save an association between the object point cloud model of the object and the two-dimensional feature of the object.

[0112] As can be seen from the above, in the embodiments of the present application, first, a simple target detection algorithm is used to determine the position information and category information of the target object; then, the first target two-dimensional feature of the target object is extracted according to the position information, and the target point cloud model corresponding to the target object is called based on the category information; finally, the pose of the target object can be estimated according to the first target two-dimensional feature and the target point cloud model. Based on the above process, it can be seen that the object point cloud models corresponding to objects of different categories are obtained through offline training in advance, and when pose estimation is needed, the corresponding calling is only needed according to the category information of the target object, thereby improving the generalization ability and processing speed of pose estimation, and the real-time inference of pose estimation can be realized on embedded devices such as robots,

[0113] ​Corresponding to the pose estimation method provided above, the embodiments of the present application further provide an electronic device. Please refer to Figure 9 The electronic device 9 in the embodiments of the present application includes a memory 901, one or more processors 902 (only one is shown in the figure) Figure 9 and a computer program stored in the memory 901 and executable on the processor. Among them: the memory 901 is used to store software programs and modules, and the processor 902 executes various functions and data processing by running the software programs and units stored in the memory 901, so as to obtain the resources corresponding to the above-mentioned preset event. Specifically, the processor 902 realizes the following steps by running the above-mentioned computer program stored in the memory 901:

[0114] perform target detection on the to-be-processed image to determine the position information and category information of the target object contained in the to-be-processed image;

[0115] extract a first target two-dimensional feature of the target object according to the position information;

[0116] Among a plurality of object point cloud models obtained by offline training, a target point cloud model corresponding to the target object is determined according to the category information;

[0117] perform pose estimation on the target object according to the first target two-dimensional feature and the target point cloud model to obtain pose information of the target object.

[0118] Suppose the above is the first possible implementation, then in the second possible implementation provided on the basis of the first possible implementation, each object point cloud model is associated with a two-dimensional feature of the corresponding object; the pose estimation on the target object according to the first target two-dimensional feature and the target point cloud model to obtain the pose information of the target object, including:

[0119] determine a target three-dimensional feature of the target object according to the target point cloud model;

[0120] call a pose estimation model using a graph attention mechanism according to the first target two-dimensional feature, the target three-dimensional feature and a second target two-dimensional feature to obtain the pose information of the target object output by the pose estimation model, wherein the second target two-dimensional feature is the two-dimensional feature associated with the target point cloud model.

[0121] In the third possible implementation provided on the basis of the above-mentioned second possible implementation, the pose estimation model using the graph attention mechanism includes an aggregation attention layer, a self-attention layer and a cross-attention layer;

[0122] Among them, the aggregation attention layer is used to strengthen the relationship between the second target two-dimensional feature and the target three-dimensional feature;

[0123] The self-attention layer is configured to strengthen the relationship between the target three-dimensional features and the relationship between the first target two-dimensional features;

[0124] The cross-attention layer is configured to strengthen the relationship between the target three-dimensional features and the first target two-dimensional features.

[0125] In a fourth possible implementation provided on the basis of the first possible implementation, the target detection further determines the full-dimensional features of the to-be-processed image Figure Two The first target two-dimensional features of the target object are extracted according to the position information, including:

[0126] According to the position information, the object region corresponding to the target object in the to-be-processed image is determined;

[0127] According to the object region, the first target two-dimensional features of the target object are extracted from the full-dimensional features. Figure Two

[0128] In a fifth possible implementation provided on the basis of the fourth possible implementation, the first target two-dimensional features of the target object are extracted from the full-dimensional features according to the object region, including: Figure Two

[0129] The object region is expanded outward based on a preset expansion ratio to obtain a target region;

[0130] In the full-dimensional features, the two-dimensional features corresponding to the target region are determined as the first target two-dimensional features of the target object. Figure Two

[0131] In a sixth possible implementation provided on the basis of the first possible implementation, or on the basis of the second possible implementation, or on the basis of the third possible implementation, or on the basis of the fourth possible implementation, or on the basis of the fifth possible implementation, the offline training process of the object point cloud model corresponding to each object includes:

[0132] A plurality of images of the object are obtained, and the plurality of images are obtained by a camera continuously capturing around the object;

[0133] The plurality of images are three-dimensionally reconstructed to obtain the object point cloud model of the object;

[0134] According to the category information of the object, the object point cloud model is named.

[0135] In a seventh possible implementation provided on the basis of the sixth possible implementation, the offline training process of the object point cloud model corresponding to the object further includes:

[0136] ​​​image enhancement is performed on the plurality of images to obtain a plurality of enhanced images;

[0137] two-dimensional features of the object in the plurality of enhanced images are extracted;

[0138] a correlation between the object point cloud model of the object and the two-dimensional features of the object is saved.

[0139] It should be understood that, in the embodiments of the present application, the processor 902 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or can also be any conventional processor.

[0140] The memory 901 can include read-only memory and random access memory, and provide instructions and data to the processor 902. Part or all of the memory 901 can also include non-volatile random access memory. For example, the memory 901 can also store device type information.

[0141] As can be seen from the above, in the embodiments of the present application, first, a simple target detection algorithm is used to determine the position information and category information of the target object; then the first target two-dimensional features of the target object are extracted according to the position information, and the target point cloud model corresponding to the target object is called based on the category information; finally, the pose estimation of the target object can be performed according to the first target two-dimensional features and the target point cloud model. Based on the above process, it can be known that the object point cloud model corresponding to a plurality of objects of different categories is obtained by pre-off-line training in the embodiments of the present application, and when pose estimation is needed, only the corresponding calling according to the category information of the target object is needed, thereby improving the generalization ability and processing speed of pose estimation, and the real-time inference of pose estimation can be realized on embedded devices such as robots,

[0142] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is taken as an example, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the above device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit or module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The above integrated unit can be realized in the form of hardware or software functional unit. In addition, the specific name of each functional unit or module is only for easy distinction, and does not limit the protection scope of the application. The specific working process of the unit or module in the above system can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

[0143] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0144] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the application.

[0145] In the embodiments provided in the present application, it should be understood that the disclosed device and method can be implemented in other ways. For example, the system embodiments described above are only schematic, for example, the division of the above modules or units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0146] The units described above as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0147] The above integrated unit, if realized in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing associated hardware. The above-mentioned computer program can be stored in a computer readable storage medium, and when executed by a processor, can realize the steps of the above-mentioned various method embodiments. The above-mentioned computer program includes computer program code, which can be in the form of source code, object code, an executable file, or some intermediate form, etc. The above-mentioned computer readable storage medium can include any entity or device capable of carrying the above-mentioned computer program code, a recording medium, a U disk, a mobile hard disk, a magnetic disk, an optical disk, a computer readable memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the content included in the above-mentioned computer readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable storage medium does not include electrical carrier signals and telecommunication signals.

[0148] The above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A pose estimation method, characterized in that, include: Target detection is performed on the image to be processed to determine the location information and category information of the target objects contained in the image to be processed. The category information is the category ID of the object category to which the target object belongs. There is only one type of object under each object category, and the shape features of the object remain fixed. Extract the first two-dimensional feature of the target object based on the location information; Among multiple offline trained object point cloud models, a target point cloud model corresponding to the target object is determined based on the category information. Based on the first target's two-dimensional features and the target's point cloud model, the pose of the target object is estimated to obtain the target object's pose information.

2. The pose estimation method as described in claim 1, characterized in that, Each of the object point cloud models is associated with the corresponding two-dimensional features of the object; the step of estimating the pose of the target object based on the first target two-dimensional features and the target point cloud model to obtain the pose information of the target object includes: Based on the target point cloud model, determine the target three-dimensional features of the target object; Based on the first target two-dimensional features, the target three-dimensional features, and the second target two-dimensional features, a pose estimation model employing a graph attention mechanism is invoked to obtain the pose information of the target object output by the pose estimation model, wherein the second target two-dimensional features are the two-dimensional features associated with the target point cloud model.

3. The pose estimation method as described in claim 2, characterized in that, The pose estimation model employing graph attention mechanism includes: a convergent attention layer, a self-attention layer, and a cross-attention layer; The aggregated attention layer is used to enhance the relationship between the second target's two-dimensional features and the target's three-dimensional features; The self-attention layer is used to enhance the relationship between the three-dimensional features of each target and the relationship between the two-dimensional features of each first target; The cross-attention layer is used to enhance the relationship between the target's three-dimensional features and the first target's two-dimensional features.

4. The pose estimation method as described in claim 1, characterized in that, The target detection also identifies the full-image two-dimensional features of the image to be processed; The step of extracting the first two-dimensional feature of the target object based on the location information includes: Based on the location information, the object region corresponding to the target object in the image to be processed is determined; Based on the object region, extract the first target two-dimensional feature of the target object from the two-dimensional features of the full image.

5. The pose estimation method as described in claim 4, characterized in that, The step of extracting the first target two-dimensional feature of the target object from the two-dimensional features of the entire image based on the object region includes: The object region is expanded outward based on a preset expansion ratio to obtain the target region; In the full-image two-dimensional features, the two-dimensional features corresponding to the target region are determined as the first target two-dimensional features of the target object.

6. The pose estimation method according to any one of claims 1 to 5, characterized in that, For each object, the offline training process of the object point cloud model corresponding to that object includes: Acquire multiple images of the object, the multiple images being continuously captured by a camera surrounding the object; The multiple images are reconstructed in three dimensions to obtain the object point cloud model of the object; Name the point cloud model of the object based on the object's category information.

7. The pose estimation method as described in claim 6, characterized in that, The offline training process for the point cloud model corresponding to the object also includes: Image enhancement is performed on the multiple images to obtain enhanced multiple images; Extract the two-dimensional features of the object from the enhanced multiple images; Save the association between the object's point cloud model and the object's two-dimensional features.

8. A pose estimation device, characterized in that, include: The detection module is used to perform target detection on the image to be processed, and to determine the location information and category information of the target objects contained in the image to be processed. The category information is the category ID of the object category to which the target object belongs. There is only one type of object under each object category, and the shape features of the object remain fixed. The extraction module is used to extract the first two-dimensional features of the target object based on the location information; The determination module is used to determine the target point cloud model corresponding to the target object from multiple offline trained object point cloud models based on the category information. The estimation module is used to estimate the pose of the target object based on the first target's two-dimensional features and the target point cloud model, so as to obtain the pose information of the target object.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images