Pose determination method and device and electronic equipment

Abstract

A pose determination method and apparatus, and an electronic device relate to the technical field of computer vision, and the method determines a relative pose between a complemented point cloud and an initial point cloud based on the complemented point cloud, and can more accurately determine a target pose of a first target according to the relative pose and the initial pose of the first target. In the method, the electronic device firstly obtains an RGB image including a first target and a depth image corresponding to the RGB image, and obtains an initial point cloud of the first target. And then, according to the RGB image and the depth image, determining a first point cloud of the first target, obtaining RGB features of the RGB image and point cloud features of the first point cloud, and according to the RGB features and the point cloud features, obtaining a complemented point cloud of the first target. And finally, determining a relative pose between the complemented point cloud and the initial point cloud, and determining a target pose of the first target according to the relative pose and the initial point cloud.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method, device and electronic device for determining a posture. Background Art

[0002] 6D pose refers to the pose of an object with 6 degrees of freedom, including 3 degrees of freedom translation parameters and 3 degrees of freedom spatial rotation parameters. The acquisition and estimation of the 6D pose of an object can be applied to robotics, autonomous driving, virtual reality (VR) / augmented reality (AR) and other fields. Based on the 6D pose of an object, the position of the object in space can be determined more accurately.

[0003] In the current object 6D pose estimation method, convolutional neural networks (CNN) and PointNet networks are used to extract RGB features and point cloud features from the image of the target object, and then the RGB features and point cloud features are spliced ​​for each key pixel in the image to obtain the 6D pose of the target object. However, the accuracy of the 6D pose estimated in this way is low, which makes it difficult for the 6D pose to meet the actual scene requirements in different fields. Summary of the invention

[0004] The embodiments of the present application provide a posture determination method, device and electronic device, which can determine the relative posture between the completed point cloud and the initial point cloud based on the relative posture and the initial posture of the first target, and can more accurately determine the target posture of the first target based on the relative posture and the initial posture of the first target.

[0005] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, a method for determining a posture is provided, in which an electronic device first obtains an RGB image of a first target and a depth image corresponding to the RGB image, and obtains an initial point cloud of the first target. Afterwards, a first point cloud of the first target is determined based on the RGB image and the depth image, and RGB features of the RGB image and point cloud features of the first point cloud are obtained, and a completed point cloud of the first target is obtained based on the RGB features and the point cloud features. Finally, a relative posture between the completed point cloud and the initial point cloud is determined, and a target posture of the first target is determined based on the relative posture and the initial point cloud.

[0007] In the above method, after obtaining the RGB image and the corresponding depth image and the first point cloud of the first target, the completed point cloud of the first target can be obtained based on the RGB features of the RGB image and the point cloud features of the first point cloud, that is, the first point cloud is completed, improved or the point cloud information is enhanced, so as to obtain the completed point cloud. Compared with the first point cloud, the completed point cloud includes more three-dimensional points and can more accurately characterize the position and posture of the first target, as well as the geometric information. Therefore, the relative posture between the completed point cloud and the initial point cloud is determined based on the completed point cloud, and the target posture of the first target can be more accurately determined based on the relative posture and the initial posture of the first target.

[0008] In one implementation method of the first aspect, when the electronic device determines the target posture of the first target, it can transform the initial point cloud according to the relative posture to obtain the transformed target point cloud of the first target, and then determine the target posture of the first target based on the target point cloud.

[0009] In the above implementation, the purpose of transforming the initial point cloud is to further adjust the pose of the initial point cloud so that the pose of the target point cloud obtained after adjustment is closer to the actual pose of the first target.

[0010] In an implementation manner of the first aspect, the electronic device acquires at least two image groups including a first target, wherein one image group corresponds to a camera perspective, and one image group includes an RGB image and a depth image corresponding to the RGB image. Afterwards, the electronic device acquires one image group from the at least two image groups, thereby obtaining an RGB image including the first target and a depth image corresponding to the RGB image.

[0011] In an implementation of the first aspect, the electronic device determines an initial point cloud of the first target based on the depth images in the at least two image groups. The electronic device can determine the initial point cloud of the first target in combination with multiple depth images acquired from different camera perspectives. Such an initial point cloud can be regarded as a relatively accurate point cloud. Afterwards, in the subsequent posture determination process, the electronic device can use the initial point cloud as a reference or as a standard point cloud, and on this basis further obtain a more accurate target point cloud and target posture.

[0012] In an implementation of the first aspect, when the electronic device determines the first point cloud of the first target, the electronic device determines the position of the first target in the depth image according to the RGB image and the depth image. Afterwards, the electronic device determines the first point cloud of the first target according to the camera intrinsic parameter and the position of the first target in the depth image.

[0013] In an implementation of the first aspect, when the electronic device obtains the completed point cloud of the first target, the RGB feature and the point cloud feature are merged to obtain the first feature. Afterwards, the electronic device obtains the completed point cloud of the first target based on the first feature and the RGB feature.

[0014] In the above implementation, the completed point cloud refers to the point cloud after the first point cloud is completed, because the first point cloud is determined by the electronic device based on an RGB image and a depth image, which is equivalent to the first point cloud being only a point cloud of the visible part of the first target at one angle. Such a point cloud is not complete and cannot well reflect the geometric information of the first target. Therefore, in order to obtain a point cloud that can better reflect the geometric information of the first target, the electronic device completes the first point cloud to make the completed point cloud more complete, thereby enhancing the point cloud information compared to the first point cloud.

[0015] In an implementable manner of the first aspect, when the electronic device determines the relative pose between the completed point cloud and the initial point cloud, at least one first key point in the completed point cloud is obtained. For each first key point, within a range with the first key point as the center and a preset length as the radius, the electronic device obtains a second point cloud corresponding to the first key point, wherein the second point cloud includes RGB information and position information of each point in the second point cloud. For each second point cloud, the electronic device determines a third point cloud corresponding to the second point cloud in the initial point cloud, and determines the relative pose between the second point cloud and the third point cloud, wherein the third point cloud includes RGB information and position information of each point in the third point cloud. The electronic device determines the registration error of the second point cloud based on the RGB information and position information of each point in the second point cloud and the RGB information and position information of each point in the third point cloud, and, if the registration error of each second point cloud is less than or equal to the first preset error, the electronic device determines the relative pose between the completed point cloud and the initial point cloud based on the relative pose between each second point cloud and the third point cloud.

[0016] In an implementation manner of the first aspect, the second point cloud includes at least one second key point; each second key point corresponds to a third key point in the third point cloud. When the electronic device determines the registration error of the second point cloud, for each second key point, the distance difference between the second key point and the third key point is determined based on the position information of the second key point and the position information of the corresponding third key point. And, the RGB difference between the second key point and the third key point is determined based on the RGB information of the second key point and the RGB information of the corresponding third key point. Afterwards, the electronic device determines the registration error of the second key point based on the distance difference and the RGB difference, and determines the registration error of the second point cloud based on the registration error of each second key point.

[0017] In an implementation of the first aspect, the RGB information of each point in the second point cloud and the third point cloud includes color values ​​corresponding to the three colors of red, green, and blue. When the electronic device determines the RGB difference between the second key point and the third key point, it determines the red difference, green difference, and blue difference between the red, green, and blue color values ​​in the RGB information of the second key point and the corresponding red, green, and blue color values ​​in the RGB information of the third key point. And, the electronic device obtains the sum of the red difference, the green difference, and the blue difference after square operations, respectively, and the sum represents the RGB difference between the second key point and the corresponding third key point.

[0018] Among the above-mentioned implementation methods, the key points in the completed point cloud can be optimized and adjusted based on the RGB information and position information of the points in the point cloud. Due to the addition of RGB information, when determining the registration error of the point cloud, the influence of multiple aspects (position information, RGB information) can be combined to more accurately determine the relative pose between the completed point cloud and the initial point cloud, and based on the relative pose, more accurately determine the target pose of the first target, while also achieving the purpose of optimizing the pose of the completed point cloud (i.e., the target pose of the first target).

[0019] In one implementation method of the first aspect, if there is a second point cloud with a registration error greater than a first preset error, the electronic device adjusts the position of the first key point corresponding to the second point cloud according to the registration error, and the electronic device again obtains the registration error of the new second point cloud corresponding to the first key point after the adjusted position, until the registration error of the new second point cloud is less than or equal to the first preset error.

[0020] In this implementation, the electronic device can adjust the position of each first key point, so as to achieve the purpose of optimizing the first key point. And, if the registration error of the second point cloud is less than or equal to the first preset error, it means that the first key corresponding to the second point cloud is in the optimal position or has been optimized. In this case, the pose estimation or determination based on the completed point cloud including the first key point is more accurate.

[0021] In an implementation manner of the first aspect, the electronic device fits the relative pose between each second point cloud and the third point cloud based on a least squares method, thereby obtaining the relative pose between the completed point cloud and the initial point cloud.

[0022] In an implementation manner of the first aspect, when the electronic device adjusts the position of the first key point, operations such as translation and / or rotation may be performed.

[0023] In a second aspect, a posture determination device is provided, comprising: an information acquisition module, used to acquire an RGB image of a first target and a depth image corresponding to the RGB image, and to acquire an initial point cloud of the first target. A point cloud acquisition module, used to determine a first point cloud of the first target based on the RGB image and the depth image. A point cloud completion module, used to acquire RGB features of the RGB image and point cloud features of the first point cloud, and to obtain a completed point cloud of the first target based on the RGB features and the point cloud features. A posture acquisition module, used to determine the relative posture between the completed point cloud and the initial point cloud, and to determine the target posture of the first target based on the relative posture and the initial point cloud.

[0024] In an implementation manner of the second aspect, the posture acquisition module is further used to transform the initial point cloud according to the relative posture to obtain the transformed target point cloud of the first target; and determine the target posture of the first target according to the target point cloud.

[0025] In an implementation manner of the second aspect, the information acquisition module is further used to acquire at least two image groups including the first target, one image group corresponding to a camera perspective, and one image group including an RGB image and a depth image corresponding to the RGB image; and acquire one image group from the at least two image groups.

[0026] In an implementation manner of the second aspect, the information acquisition module is further used to determine an initial point cloud of the first target based on the depth images in the at least two image groups.

[0027] In an implementation manner of the second aspect, the point cloud acquisition module is further used to determine the position of the first target in the depth image based on the RGB image and the depth image; and determine the first point cloud of the first target based on the camera intrinsic parameters and the position of the first target in the depth image.

[0028] In an implementation manner of the second aspect, the point cloud completion module is further used to fuse the RGB feature and the point cloud feature to obtain the first feature; and obtain the completed point cloud of the first target based on the first feature and the RGB feature.

[0029] In an implementable manner of the second aspect, the posture acquisition module is also used to acquire at least one first key point in the completed point cloud; for each first key point, a second point cloud corresponding to the first key point is acquired within a range with the first key point as the center and a preset length as the radius; the second point cloud includes the RGB information and position information of each point in the second point cloud; for each second point cloud, a third point cloud corresponding to the second point cloud in the initial point cloud is determined, and the relative posture between the second point cloud and the third point cloud is determined; the third point cloud includes the RGB information and position information of each point in the third point cloud; based on the RGB information and position information of each point in the second point cloud, and the RGB information and position information of each point in the third point cloud, the registration error of the second point cloud is determined; if the registration error of each second point cloud is less than or equal to the first preset error, the relative posture between the completed point cloud and the initial point cloud is determined based on the relative posture between each second point cloud and the third point cloud.

[0030] In an implementation method of the second aspect, the second point cloud includes at least one second key point; each second key point corresponds to a third key point in the third point cloud; the posture acquisition module is also used to determine, for each second key point, the distance difference between the second key point and the third key point based on the position information of the second key point and the corresponding position information of the third key point; determine the RGB difference between the second key point and the third key point based on the RGB information of the second key point and the corresponding RGB information of the third key point; determine the alignment error of the second key point based on the distance difference and the RGB difference; determine the alignment error of the second point cloud based on the alignment error of each second key point.

[0031] In an implementation method of the second aspect, the RGB information of each point in the second point cloud and the third point cloud includes color values ​​corresponding to the three colors of red, green and blue, respectively; the posture acquisition module is also used to respectively determine the red difference, green difference and blue difference between the red, green and blue color values ​​in the RGB information of the second key point and the corresponding red, green and blue color values ​​in the RGB information of the third key point; obtain the sum of the red difference, green difference and blue difference after square operation, and the sum represents the RGB difference between the second key point and the corresponding third key point.

[0032] In an implementable manner of the second aspect, the posture acquisition module is also used to adjust the position of the first key point corresponding to the second point cloud according to the registration error if there is a second point cloud with a registration error greater than the first preset error; and again obtain the registration error of the new second point cloud corresponding to the first key point after the adjusted position, until the registration error of the new second point cloud is less than or equal to the first preset error.

[0033] In a third aspect, an electronic device is provided, comprising a memory and one or more processors; the memory is coupled to the processor; wherein computer program code is stored in the memory, and the computer program code comprises computer instructions, and when the computer instructions are executed by the processor, the electronic device executes a posture determination method as in the first aspect and any implementation thereof.

[0034] In a fourth aspect, a computer-readable storage medium is provided, comprising computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes a posture determination method as in the first aspect and any implementation thereof.

[0035] In a fifth aspect, a computer program product is provided. When the computer program product is run on a computer, the computer executes the posture determination method as in the first aspect and any implementation thereof.

[0036] It can be understood that the beneficial effects that can be achieved by the posture determination device of the second aspect, the electronic device of the third aspect, the computer-readable storage medium of the fourth aspect, and the computer program product of the fifth aspect can be referred to the beneficial effects in the first aspect and any possible design method thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic diagram of a pose estimation method based on feature matching shown in an embodiment of the present application;

[0038] Figure 2 A schematic diagram of a posture estimation method based on deep learning shown in an embodiment of the present application;

[0039] Figure 3 A schematic diagram of the connection between an electronic device and a camera shown in an embodiment of the present application;

[0040] Figure 4 A schematic diagram of an application scenario shown in an embodiment of the present application;

[0041] Figure 5 Schematic diagram of the process of the posture determination method shown in the embodiment of the present application Figure 1 ;

[0042] Figure 6 A schematic diagram of an image group shown in an embodiment of the present application;

[0043] Figure 7 Schematic diagram of the process of the posture determination method shown in the embodiment of the present application Figure 2 ;

[0044] Figure 8 Schematic diagram of the process of the posture determination method shown in the embodiment of the present application Figure 3 ;

[0045] Fig. 9 Schematic diagram of the process of the posture determination method shown in the embodiment of the present application Figure 4 ;

[0046] Fig.10 Schematic diagram of the process of the posture determination method shown in the embodiment of the present application Figure 5 ;

[0047] Fig.11 Schematic diagram of the process of the posture determination method shown in the embodiment of the present application Figure 6 ;

[0048] Fig.12 A schematic diagram of the structure of a posture determination device shown in an embodiment of the present application;

[0049] Fig.13 This is a schematic diagram of the structure of an electronic device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. Among them, in the description of the present application, unless otherwise specified, " / " indicates that the objects associated before and after are in an "or" relationship, for example, A / B can represent A or B; "and / or" in the present application is only a description of the association relationship of the associated objects, indicating that there can be three relationships, for example, A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. And, in the description of the present application, unless otherwise specified, "multiple" refers to two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple. In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first", "second" and the like are used to distinguish the same items or similar items with substantially the same functions and effects. Those skilled in the art will understand that the words "first", "second" and the like do not limit the quantity and execution order, and the words "first", "second" and the like do not necessarily limit the differences. At the same time, in the embodiments of the present application, the words "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or design solutions. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete manner for ease of understanding.

[0051] In addition, the business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. A person of ordinary skill in the art can appreciate that with the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0052] The pose represents the position and posture of a person or object in space, and the pose can generally include 3D and 6D poses. Among them, the 3D pose uses three degrees of freedom translation parameters to represent the pose, for example, the translation parameters in the three directions (or degrees of freedom) of x, y, and z in a specified coordinate system are used to represent the pose. The 6D pose refers to the use of six degrees of freedom parameters to represent the pose, and the six degrees of freedom include, for example, the aforementioned three degrees of freedom translation parameters, and three degrees of freedom spatial rotation parameters. The acquisition and estimation of the 6D pose of an object can be applied to robotics, autonomous driving, virtual reality (VR) / augmented reality (AR) and other fields. Based on the 6D pose of the object, the position of the object in space can be determined more accurately.

[0053] At present, the methods for estimating the 6D pose of an object can be roughly divided into feature matching-based methods (such as the DenseFusion method) and deep learning-based methods (such as the FFB6D method).

[0054] Among them, see Figure 1 As shown in (a) in the figure, in the feature matching-based method, a fusion method of pixel-level RGB features and point cloud features is designed, that is, for the image of the target object, convolutional neural networks (CNN) and point cloud network PointNet are used to extract RGB features and point cloud features respectively, and then DenseFusion and other methods are used to splice the RGB features and point cloud features for each key pixel in the image. After that, global features are extracted based on the multilayer perceptron (MLP), and the 6D pose of the target object is obtained based on the global features and spliced ​​features. And, after obtaining the 6D pose, see Figure 1 As shown in (b), the 6D pose predicted or obtained in the previous round is regarded as the standard pose of the target object, the RGB features and point cloud features are re-extracted, and the two are spliced ​​again to obtain a new 6D pose of the target object. According to the 6D pose obtained last time, the point cloud of the target object corresponding to the new 6D pose is rotated, and the error or residual pose between the point cloud corresponding to the new 6D pose and the point cloud corresponding to the previous 6D pose is determined. Continue to repeat Figure 1In the step (b), the position of the point cloud is continuously adjusted. After K rounds of iterations, the final 6D pose of the target object can be obtained. This method can be regarded as a process of 6D pose optimization, or as an iterative optimization process of the above pose estimation method.

[0055] However, in the above 6D pose estimation method, the fusion method for features of different modalities (such as RGB features and point cloud features) is just a simple splicing. The 6D accuracy determined based on this method is low, which makes it difficult for the 6D pose to meet the actual scene requirements in different fields. In addition, iterative optimization only uses the 6D pose estimation method to optimize itself. This optimization method has low generalization ability and is only used for the current scene. It is difficult to apply to other scenes. For example, when the lighting conditions in other scenes are different from those in the current scene, the 6D pose estimation method obtained by this optimization method will not estimate a very accurate 6D pose in other scenes.

[0056] See also Figure 2 As shown in the figure, in the deep learning-based method, for each key pixel in the image, a deep neural network is used to fuse RGB features and point cloud features, and then the displacement between the pixel and the pre-selected preset key point is predicted one by one, and the coordinates of the key pixel are obtained by voting clustering. Finally, based on the coordinates of each key pixel, the least squares method is used to obtain the 6D pose of the target object.

[0057] However, this 6D pose estimation method does not have an iterative optimization process, nor can it guarantee that the estimated 6D pose has high accuracy, nor can it meet the actual scenario requirements in different fields.

[0058] Based on the above content, an embodiment of the present application provides a method for determining a posture. In this method, first, an RGB image including a first target and a depth image corresponding to the RGB image are obtained, and an initial point cloud of the first target is obtained. Afterwards, the first point cloud of the first target is determined based on the RGB image and the depth image, and the RGB features of the RGB image and the point cloud features of the first point cloud are obtained, and the completed point cloud of the first target is obtained based on the RGB features and the point cloud features. Finally, the relative posture between the completed point cloud and the initial point cloud is determined, and the target posture of the first target is determined based on the relative posture and the initial point cloud.

[0059] In the posture determination method of the embodiment of the present application, after obtaining the RGB image and the corresponding depth image and the first point cloud of the first target, the completed point cloud of the first target can be obtained based on the RGB features of the RGB image and the point cloud features of the first point cloud, that is, the first point cloud is completed, improved or the point cloud information is enhanced, so as to obtain the completed point cloud. Compared with the first point cloud, the completed point cloud includes more three-dimensional points and can more accurately characterize the position and posture of the first target, as well as the geometric information. Therefore, the relative posture between the completed point cloud and the initial point cloud is determined based on the completed point cloud, and according to the relative posture and the initial posture of the first target, the target posture of the first target can be more accurately determined.

[0060] The above-mentioned posture determination method provided in the embodiment of the present application can be applied to electronic devices with computing and processing capabilities, such as computers, servers, hosts, VR devices, AR devices, robots, and vehicles. If the VR device, AR device, robot, vehicle, and other devices do not have computing and processing capabilities, then the electronic device that implements the above-mentioned posture determination method is connected to the VR device, AR device, robot, vehicle, and other devices, and the posture determination result of the target is sent to these devices. For example, the VR device, AR device, robot, vehicle, etc. can be connected to the server, and RGB images and depth images, etc. are sent to the server. The server determines the posture of the target through the above-mentioned posture determination method, and sends the posture determination result to the VR device, AR device, robot, vehicle, etc.

[0061] For some possible application scenarios, see Figure 3 As shown in (a) of FIG. 1 , the electronic device 301 may be connected to an RGB camera 302, and the RGB camera 302 is used to capture an RGB image including the first target from different camera angles. Figure 3 As shown in (b), the RGB camera can take RGB images A1, RGB images A2, RGB images A3, etc. at different camera perspectives at equal distances on a sphere with a fixed length as the radius and the first target A as the center. After the RGB camera 302 takes the RGB image, it sends the RGB image to the electronic device 301. After the electronic device 301 receives the RGB images under multiple perspectives, it can use a depth estimation algorithm to obtain a depth image corresponding to each RGB image, for example, obtain a corresponding depth image a1 for RGB image A1, obtain a corresponding depth image a2 for RGB image A2, and obtain a corresponding depth image a3 for RGB image A3.

[0062] In other possible application scenarios, the electronic device 301 may also be connected to a depth camera (such as RGB-D), which may obtain the RGB image and depth image of the first target A from the same camera perspective, and send the RGB image and depth image to the electronic device 301, respectively.

[0063] The electronic device 301 in the embodiment of the present application can determine the position and posture of targets in actual scenes in different fields. Figure 4 As shown, for VR or AR scenes, an RGB camera or a depth camera is deployed on the VR or AR device to capture the environment around the user of the VR or AR device, so that the VR or AR device can estimate the posture (such as 3D posture, 6D posture, etc.) of objects around the user based on the images captured by the camera, thereby realizing the environmental perception function. Figure 4 As shown, for industrial robot scenarios, RGB cameras or depth cameras are deployed on or around the robotic arms of industrial robots to capture images of objects that the robotic arms need to operate, so that the robot or the electronic device connected to the camera can estimate the pose (such as 3D pose, 6D pose, etc.) of the object to be operated based on the images captured by the camera, and then guide the robotic arm to accurately grasp the object. For another example, see Figure 4 As shown, for the intelligent driving scenario, an RGB camera or a depth camera is deployed on the intelligent driving vehicle to capture the scene around the vehicle. The vehicle or an electronic device connected to the camera estimates the posture (such as 3D posture, 6D posture, etc.) of surrounding vehicles or pedestrians based on the images captured by the camera, thereby realizing vehicle distance estimation and scene perception functions, and providing a basis for the driving and decision-making of the intelligent driving vehicle.

[0064] In other possible application scenarios, the electronic device 301 can also determine the position for a known model, wherein the known model can be a computer aided design (CAD) model, etc. Moreover, the model can be any person or object in actual scenes in different fields. In this case, the electronic device 301 can display an image of a known model, and collect RGB images and depth images of the known model from multiple perspectives at equal distances through an RGB camera or a depth camera on a sphere with a radius of a fixed length and centered on the known model. Alternatively, after obtaining the known model, the electronic device 301 rotates the known model at a fixed angle, and obtains the known model image after each rotation, thereby achieving the purpose of collecting RGB images and depth images of the known model from multiple perspectives.

[0065] Take the method for determining the posture in the embodiment of the present application as an example, which is applied to the above-mentioned electronic device. Figure 5As shown, the posture determination method may include the following steps S501-S504.

[0066] S501: The electronic device obtains an RGB image of a first target and a depth image corresponding to the RGB image, and obtains an initial point cloud of the first target.

[0067] In some embodiments, the electronic device may first acquire at least two image groups including the first target, wherein one image group corresponds to one camera perspective, and one image group includes an RGB image and a depth image corresponding to the RGB image, for example Figure 6 As shown. It can be understood that the depth image can be obtained by a depth camera, or it can be determined by an electronic device through a depth estimation algorithm based on the RGB image, and there is no specific limitation on this in this embodiment. Among them, the depth image includes the depth information of each pixel in the corresponding RGB image, for example, the distance between each pixel and the camera (such as), the distance between each pixel and a reference point, etc.

[0068] Afterwards, the electronic device acquires one image group from the at least two image groups, thereby obtaining an RGB image of the first target and a depth image corresponding to the RGB image. And, the acquired one image group is any one of the at least two image groups.

[0069] In addition, the image including the first target (such as an RGB image, a depth image, etc.) also includes other objects. The purpose of the embodiment of the present application is to determine the position and posture of the first target in the acquired image. The first target can be any object in the image (such as Figure 6 The banana shown in the dotted box) or any character.

[0070] In some embodiments, the electronic device may also determine the initial point cloud of the first target based on the depth images in the at least two image groups obtained above. It can be seen that the electronic device can determine the initial point cloud of the first target by combining multiple depth images obtained from different camera perspectives. Such an initial point cloud can be regarded as a relatively accurate point cloud. Afterwards, in the subsequent posture determination process, the electronic device can use the initial point cloud as a reference or as a standard point cloud, and on this basis further obtain a more accurate target point cloud and target posture.

[0071] This step S501 can be regarded as a preparation stage or a preprocessing stage for posture determination.

[0072] S502: The electronic device determines a first point cloud of a first target based on the RGB image and the depth image.

[0073] In some embodiments, when determining the first point cloud of the first target, the electronic device may first determine the position of the first target in the depth image based on the RGB image and the depth image, and determine the first point cloud of the first target based on the camera intrinsic parameters and the position of the first target in the depth image. Determining the position of the first target in the depth image may also be referred to as instance segmentation of the first target. Exemplarily, the position of the first target in the depth image may be shown as a rectangular frame in the depth image.

[0074] For some possible application scenarios, see Figure 7 As shown, the electronic device can use the pose estimation network to achieve instance segmentation of the first target. Specifically, the electronic device can input the RGB image and the depth image into the pose estimation network, and the pose estimation network determines the position of the first target in the depth image through a preset pose estimation algorithm, and at the same time, can output the first pose of the first target. The first pose here is the estimated pose of the first target by the pose estimation network.

[0075] Afterwards, see Figure 7 As shown, the electronic device then determines the first point cloud of the first target based on the camera intrinsic parameters and the position of the first target in the depth image. Alternatively, it can also be expressed as the electronic device converts the segmentation result obtained by the pose estimation network into a point cloud of the visible part of the first target based on the camera intrinsic parameters.

[0076] It can be understood that the camera coordinate system can be determined by the camera intrinsic parameters, so the point cloud of the visible part of the first target is the point cloud in the camera coordinate system.

[0077] For other possible application scenarios, see Figure 7 As shown, after obtaining the first point cloud, the electronic device can also delete outliers in the first point cloud to make the first point cloud more accurate. Also, illustratively, the electronic device can remove outliers in the first point cloud based on the MeanShift clustering method, which is not specifically limited in the embodiments of the present application.

[0078] In some embodiments, the first pose is used to convert the point cloud in the camera coordinate system to the object coordinate system. In the relevant content of step S502, the electronic device has acquired the first point cloud of the first target. In order to ensure the consistency of the first target information comparison in the subsequent determination of the pose, the electronic device converts the first point cloud from the camera coordinate system to the object coordinate system according to the first pose after acquiring the first point cloud.

[0079] S503: The electronic device obtains RGB features of the RGB image and point cloud features of the first point cloud, and obtains a completed point cloud of the first target according to the RGB features and the point cloud features.

[0080] In some embodiments, the electronic device may fuse the RGB features and the point cloud features to first obtain the first feature, and then obtain the completed point cloud of the first target based on the first feature and the RGB feature.

[0081] Furthermore, in some possible application scenarios, the electronic device can obtain a completed point cloud of the first target based on a point cloud completion network, and the input of the point cloud completion network is an RGB image and a first point cloud, and the output is a completed point cloud.

[0082] It can be understood that the completed point cloud refers to the point cloud after the completion of the first point cloud, because the first point cloud is determined by the electronic device based on an RGB image and a depth image, which is equivalent to the first point cloud being only a point cloud of the visible part of the first target at one angle. Such a point cloud is not complete and cannot well reflect the geometric information of the first target. Therefore, in order to obtain a point cloud that can better reflect the geometric information of the first target, the electronic device completes the first point cloud to make the completed point cloud more complete, thereby enhancing the point cloud information compared to the first point cloud.

[0083] Based on this, the complement operation for the first point cloud in the embodiment of the present application can also be called a perfection operation, a point cloud information enhancement operation, etc., and the embodiment of the present application does not make any specific limitations on this.

[0084] For example, see Figure 8 As shown, the point cloud completion network includes a feature extraction unit and a point cloud completion unit. The feature extraction unit may also include an RGB feature extraction unit, a full flow bidirectional fusion network (FFB) unit, a DenseFusion unit, and a point cloud feature extraction unit. Among them, the RGB feature extraction unit and the point cloud feature extraction unit extract the RGB features of the RGB image and the point cloud features of the first point cloud respectively. The FFB unit and the DenseFusion unit can fuse the RGB features and the point cloud features to obtain the first feature.

[0085] The feature extraction unit sends the first feature and the RGB feature to the point cloud completion unit, and the point cloud completion unit completes the first point cloud according to the first feature and the RGB feature to obtain the completed point cloud of the first target.

[0086] In some possible application scenarios, the above-mentioned point cloud feature extraction unit can also be called a point cloud encoder (such as a PCN ENC module), and the point cloud completion unit can also be called a point cloud decoder (such as a PCN DEC module).

[0087] In other embodiments, in order to improve the geometric integrity and pose accuracy of the completed point cloud, the electronic device may further iteratively optimize the point cloud completion network when training the point cloud completion network. Fig. 9 As shown, the electronic device can introduce a key point detection unit. Iterative optimization of the point cloud completion network can also be regarded as a process of updating or optimizing the parameters of the feature extraction unit and the point cloud completion unit using the key point detection unit. The key point detection unit includes MLP. The main function of the key point detection unit is: in each round of training, multiple key points of the first target are determined respectively, and the feature extraction unit and the point cloud completion unit are used to obtain the completed point cloud for the key points determined in this round, and the key point error Lof and the point cloud error Lcd of each round before and after are determined by MLP, and the parameters of each unit in the point cloud completion network (such as FFB unit, DenseFusion unit, point cloud feature extraction unit, point cloud completion unit, etc.) are adjusted based on Lof and Lcd to minimize Lof and Lcd, thereby enhancing the utilization of the information of the input first point cloud by the point cloud completion network, improving the pose accuracy of the output completed point cloud, and ensuring the stability of the completed point cloud.

[0088] In addition, the above-mentioned iterative optimization of the point cloud completion network is equivalent to utilizing other methods or units (such as the above-mentioned key point detection unit) other than the point cloud completion network, and does not use the point cloud completion network to iteratively optimize itself. Therefore, this optimization method has a high generalization ability and can be used for point cloud completion networks in a variety of scenarios, fully considering the characteristics of different scenarios.

[0089] S504: The electronic device determines a relative pose between the completed point cloud and the initial point cloud, and determines a target pose of the first target according to the relative pose and the initial point cloud.

[0090] In some embodiments, when determining the target posture of the first target, the electronic device can transform the initial point cloud according to the relative posture to obtain the transformed target point cloud of the first target, and determine the target posture of the first target according to the target point cloud.

[0091] Exemplarily, the initial point cloud is transformed according to the relative pose, and the relative pose may be superimposed on the initial pose corresponding to the initial point cloud, so as to achieve the purpose of transforming the initial point cloud. The purpose of transforming the initial point cloud is to further adjust the pose of the initial point cloud so that the pose of the target point cloud obtained after adjustment is closer to the actual pose of the first target.

[0092] In some embodiments, see Fig.10 As shown, when the electronic device determines the relative position between the completed point cloud and the initial point cloud, it can first obtain at least one first key point in the completed point cloud (such as Fig.10Then, using the initial point cloud determined in the above step S501, at least one first key point is registered and optimized to obtain the relative pose between the completed point cloud and the initial point cloud.

[0093] In some possible application scenarios, when registering each first key point, the electronic device can obtain the second point cloud corresponding to the first key point within a range with a preset length as the radius and the first key point as the center, wherein the second point cloud includes the RGB information and position information of each point in the second point cloud. It can be understood that each point in the second point cloud can find the corresponding pixel point in the RGB image, and the pixel point in the RGB image includes RGB information, that is, color information, so each point in the second point cloud also includes RGB information. In addition, each point in the second point cloud also has its own position information, such as three-dimensional space coordinates.

[0094] Afterwards, the electronic device determines the third point cloud corresponding to the second point cloud in the initial point cloud for each second point cloud, and determines the relative position between the second point cloud and the third point cloud, wherein the third point cloud includes the RGB information and position information of each point in the third point cloud. Also, each point in the third point cloud can find the corresponding pixel point in the RGB image, and the pixel point in the RGB image includes RGB information, that is, color information, so each point in the third point cloud also includes RGB information. Moreover, each point in the third point cloud also has its own position information, such as three-dimensional space coordinates.

[0095] The electronic device then determines the registration error of the second point cloud based on the RGB information and position information of each point in the second point cloud and the RGB information and position information of each point in the third point cloud. And, if the registration error of each second point cloud is less than or equal to the first preset error, the electronic device determines the relative pose between the completed point cloud and the initial point cloud based on the relative pose between each second point cloud and the third point cloud.

[0096] In some possible application scenarios, when obtaining the second point cloud corresponding to the first key point, the electronic device may first select at least one second key point within a range with the first key point as the center and a preset length as the radius, thereby generating the second point cloud using the at least one second key point, that is, the second point cloud includes at least one second key point. Afterwards, the electronic device finds the third key point corresponding to each second key point in the initial point cloud, thereby obtaining at least one third key point, and generates the third point cloud using the at least one third key point, that is, the third point cloud includes at least one third key point.

[0097] Afterwards, the electronic device can further determine the registration error of the second point cloud based on the RGB information and position information of each point in the second point cloud (ie, each second key point) and the RGB information and position information of each point in the third point cloud (ie, the third key point).

[0098] Among them, for each second key point, the electronic device determines the distance difference between the second key point and the third key point according to the position information of the second key point and the position information of the corresponding third key point, for example, by taking the difference between the three-dimensional space coordinates of the second key point and the three-dimensional space coordinates of the third key point, and then taking the absolute value; and, the electronic device then determines the RGB difference between the second key point and the third key point according to the RGB information of the second key point and the RGB information of the corresponding third key point, wherein the RGB information includes color values ​​corresponding to the three colors of red, green, and blue, respectively, which are usually values ​​between 0-255.

[0099] Furthermore, the electronic device can respectively determine the red difference ΔR, green difference ΔG, and blue difference ΔB between the red, green, and blue color values ​​in the RGB information of the second key point and the corresponding red, green, and blue color values ​​in the RGB information of the third key point, and then obtain the sum ΔR of the red difference, the green difference, and the blue difference after square operation. 2 +ΔG 2 +ΔB 2 , then this sum value can represent the RGB difference between the second key point and the corresponding third key point.

[0100] When determining the registration error of the second point cloud, the electronic device can determine the registration error of the second key point according to the distance difference and RGB difference between the second key point and the third key point. Finally, the electronic device determines the registration error of the second point cloud according to the registration error of each second key point.

[0101] Exemplarily, the electronic device calculates an average value of the registration error of each second key point to obtain the registration error of the second point cloud.

[0102] In some possible application scenarios, if the registration error of each second point cloud is less than or equal to the first preset error, it can indicate that the completed point cloud and the initial point cloud are successfully registered, and then the electronic device can determine the relative pose between the completed point cloud and the initial point cloud based on the relative pose between each second point cloud and the third point cloud. Exemplarily, the electronic device can fit the relative pose between each second point cloud and the third point cloud using the least squares method, polynomial fitting, etc. to obtain the relative pose between the completed point cloud and the initial point cloud.

[0103] If there is a second point cloud whose registration error is greater than the first preset error, it means that the second point cloud has not been successfully registered with the corresponding third point cloud, and then the electronic device can adjust the position of the first key point corresponding to the second point cloud according to the registration error of the second point cloud. It can be understood that after the position of the first key point is adjusted, the position of the second key point in the second point cloud will also change, and then there is a new relative pose between the second point cloud and the corresponding third point cloud. Afterwards, the electronic device can obtain the registration error of the new second point cloud corresponding to the first key point after the position is adjusted again according to the content of the aforementioned embodiment, until the registration error of the new second point cloud is less than or equal to the first preset error. Afterwards, when the registration error of each second point cloud is less than or equal to the first preset error, the electronic device can determine the relative pose between the completed point cloud and the initial point cloud according to the current relative pose corresponding to each second point cloud.

[0104] Referring to the above method, the electronic device can adjust the position of each first key point, so as to achieve the purpose of optimizing the first key point. And, if the registration error of the second point cloud is less than or equal to the first preset error, it means that the first key corresponding to the second point cloud is in the optimal position or has been optimized. In this case, the pose estimation or determination based on the completed point cloud including the first key point is more accurate.

[0105] And, it can be understood that adjusting the position of the first key point may include: translating and / or rotating the first key point, and the specific adjustment method can be determined according to the registration error of the second point cloud.

[0106] In the above process of determining the registration error of the second point cloud, in addition to comparing the positions of the key points, the comparison of RGB information is also introduced. Through the comparison of RGB information, the difference or error between the key points can be more accurately determined, thereby more accurately determining the point cloud registration error corresponding to the key points and the relative pose between the completed point cloud and the initial point cloud.

[0107] In some embodiments, when the electronic device obtains at least one first key point in the completed point cloud, refer to Fig.11 As shown, first, for an image group, the electronic device obtains at least one two-dimensional feature point 1101 of the first target from the RGB image of the image group, and obtains the depth information corresponding to each two-dimensional feature point 1101 from the depth image, wherein at least one two-dimensional feature point 1101 is a part of all the pixels in the RGB image. Also, in some possible application scenarios, the electronic device can obtain the above-mentioned at least one two-dimensional feature point 1101 from the RGB image based on the SIFT algorithm, which is not specifically limited in the embodiments of the present application.

[0108] Afterwards, see Fig.11As shown, for at least two image groups acquired by the electronic device, the electronic device maps at least one two-dimensional feature point 1101 to a three-dimensional space based on the camera intrinsic parameters, at least one two-dimensional feature point 1101 in each RGB image, and the depth information corresponding to each two-dimensional feature point 1101 in each RGB image, to obtain at least one three-dimensional feature point 1102. The camera intrinsic parameters refer to the intrinsic parameters of the RGB camera that shoots the RGB image, or the intrinsic parameters of the depth camera that shoots the RGB image and the depth image.

[0109] The electronic device can match the two-dimensional feature points 1101 between each RGB image, determine multiple two-dimensional feature points 1101 belonging to the same target point on the first target, and based on the depth information of each two-dimensional feature point 1101 and the camera intrinsic parameters, map the multiple two-dimensional feature points belonging to the same target point to the three-dimensional space, so that the target point becomes a three-dimensional feature point 1102, and has three-dimensional space position information or three-dimensional space coordinates.

[0110] In addition, the camera coordinate system can be determined through the camera intrinsic parameters. The depth information of the two-dimensional feature point 1101 can represent the distance between the two-dimensional feature point 1101 and the camera. Then the three-dimensional space coordinates or three-dimensional space position information of the three-dimensional feature point 1102 determined based on the two-dimensional feature point 1101 represents the coordinates or position information of the three-dimensional feature point 1102 in the camera coordinate system.

[0111] In the above manner, the electronic device can map all the matched two-dimensional feature points 1101 in the multiple RGB images to the three-dimensional space, thereby obtaining at least one three-dimensional feature point 1102 of the first target.

[0112] Finally, see Fig.11 As shown, the electronic device then obtains at least one first key point 1103 of the first target from at least one three-dimensional feature point 1102. Among them, at least one first key point 1103 can be randomly selected, and, in some possible application scenarios, the electronic device can randomly select at least one first key point 1103 from the at least one three-dimensional feature point 1102 based on the FPS algorithm, which is not specifically limited in the embodiments of the present application.

[0113] In addition, in some possible application scenarios, after the above step S502, the first point cloud is in the object coordinate system, and the initial point cloud is still in the camera coordinate system. Therefore, in step S504, in order to ensure the accuracy of the registration error and the relative pose determination, the electronic device can also transform the completed point cloud according to the first pose of the first target obtained previously. Since the first pose represents the estimated pose of the first target in the camera coordinate system, after the completed point cloud is transformed according to the first pose, it can be transformed from the current object coordinate system back to the camera coordinate system, and compared with the initial point cloud in the camera coordinate system to determine the relative pose.

[0114] Then, it can be understood that the target posture of the first target subsequently determined by the electronic device is also the target posture of the first target in the camera coordinates.

[0115] The posture determination method provided in the above embodiment can obtain the completed point cloud of the first target based on the RGB features of the RGB image and the point cloud features of the first point cloud after obtaining the RGB image and the corresponding depth image and the first point cloud of the first target, that is, the first point cloud is completed, improved or the point cloud information is enhanced, so as to obtain the completed point cloud. Compared with the first point cloud, the completed point cloud includes more three-dimensional points and can more accurately characterize the position and posture of the first target, as well as the geometric information. Therefore, the relative posture between the completed point cloud and the initial point cloud is determined based on the completed point cloud, and the target posture of the first target can be more accurately determined based on the relative posture and the initial posture of the first target.

[0116] In addition, the above method can also optimize and adjust the key points in the completed point cloud based on the RGB information and position information of the points in the point cloud. Due to the addition of RGB information, when determining the registration error of the point cloud, the influence of multiple aspects (position information, RGB information) can be combined to more accurately determine the relative pose between the completed point cloud and the initial point cloud, and more accurately determine the target pose of the first target based on the relative pose, while also achieving the purpose of optimizing the pose of the completed point cloud (i.e., the target pose of the first target).

[0117] In the above embodiments, the initial point cloud of the first target is used as a reference, and the initial point cloud is transformed according to the relative posture. In some other embodiments, the completed point cloud of the first target can also be used as a reference, and the completed point cloud is transformed according to the relative posture to obtain the target point cloud of the first target. This can also achieve the purpose of more accurately determining the posture.

[0118] In some schemes, multiple embodiments of the present application can be combined, and the combined scheme can be implemented. Optionally, some operations in the process of each method embodiment are optionally combined, and / or the order of some operations is optionally changed. In addition, the execution order between the steps of each process is only exemplary and does not constitute a restriction on the execution order between the steps. There can also be other execution orders between the steps. It is not intended to show that the execution order is the only order in which these operations can be performed. A person of ordinary skill in the art will think of a variety of ways to reorder the operations described in the embodiments of the present application. In addition, it should be noted that the process details involved in a certain embodiment of the present application are also applicable to other embodiments in a similar manner, or different embodiments can be used in combination.

[0119] In addition, some steps in the method embodiment may be equivalently replaced by other possible steps. Alternatively, some steps in the method embodiment may be optional and may be deleted in certain usage scenarios. Alternatively, other possible steps may be added to the method embodiment.

[0120] Furthermore, the various method embodiments may be implemented separately or in combination.

[0121] It is understandable that in order to implement the above functions, the aforementioned electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of the present application.

[0122] In this embodiment, the electronic device can be divided into functional modules according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0123] The present application also provides a posture determination device, see Fig.12 As shown, the device may include an information acquisition module 1201, a point cloud acquisition module 1202, a point cloud completion module 1203 and a posture acquisition module 1204.

[0124] The information acquisition module 1201 is used to acquire an RGB image of the first target and a depth image corresponding to the RGB image, and to acquire an initial point cloud of the first target.

[0125] The point cloud acquisition module 1202 is used to determine a first point cloud of a first target according to the RGB image and the depth image.

[0126] The point cloud completion module 1203 is used to obtain the RGB features of the RGB image and the point cloud features of the first point cloud, and obtain the completed point cloud of the first target according to the RGB features and the point cloud features.

[0127] The pose acquisition module 1204 is used to determine the relative pose between the completed point cloud and the initial point cloud, and determine the target pose of the first target according to the relative pose and the initial point cloud.

[0128] In some embodiments, the posture acquisition module 1204 is further used to transform the initial point cloud according to the relative posture to obtain the transformed target point cloud of the first target; and determine the target posture of the first target according to the target point cloud.

[0129] In some embodiments, the information acquisition module 1201 is further used to acquire at least two image groups including the first target, one image group corresponds to a camera perspective, and one image group includes an RGB image and a depth image corresponding to the RGB image; and acquire one image group from the at least two image groups.

[0130] In some embodiments, the information acquisition module 1201 is further configured to determine an initial point cloud of the first target based on the depth images in the at least two image groups.

[0131] In some embodiments, the point cloud acquisition module 1202 is further used to determine the position of the first target in the depth image based on the RGB image and the depth image; and determine the first point cloud of the first target based on the camera intrinsic parameters and the position of the first target in the depth image.

[0132] In some embodiments, the point cloud completion module 1203 is further used to fuse the RGB features and the point cloud features to obtain the first feature; and obtain the completed point cloud of the first target based on the first feature and the RGB feature.

[0133] In some embodiments, the posture acquisition module 1204 is also used to acquire at least one first key point in the completed point cloud; for each first key point, a second point cloud corresponding to the first key point is acquired within a range with the first key point as the center and a preset length as the radius; the second point cloud includes the RGB information and position information of each point in the second point cloud; for each second point cloud, a third point cloud corresponding to the second point cloud in the initial point cloud is determined, and the relative posture between the second point cloud and the third point cloud is determined; the third point cloud includes the RGB information and position information of each point in the third point cloud; based on the RGB information and position information of each point in the second point cloud, and the RGB information and position information of each point in the third point cloud, the registration error of the second point cloud is determined; if the registration error of each second point cloud is less than or equal to the first preset error, the relative posture between the completed point cloud and the initial point cloud is determined based on the relative posture between each second point cloud and the third point cloud.

[0134] In some embodiments, the second point cloud includes at least one second key point; each second key point corresponds to a third key point in the third point cloud; the posture acquisition module 1204 is also used to determine, for each second key point, the distance difference between the second key point and the third key point based on the position information of the second key point and the position information of the corresponding third key point; determine the RGB difference between the second key point and the third key point based on the RGB information of the second key point and the RGB information of the corresponding third key point; determine the alignment error of the second key point based on the distance difference and the RGB difference; determine the alignment error of the second point cloud based on the alignment error of each second key point.

[0135] In some embodiments, the RGB information of each point in the second point cloud and the third point cloud includes color values ​​corresponding to the three colors of red, green and blue, respectively; the posture acquisition module 1204 is also used to respectively determine the red difference, green difference and blue difference between the red, green and blue color values ​​in the RGB information of the second key point and the corresponding red, green and blue color values ​​in the RGB information of the third key point; obtain the sum of the red difference, green difference and blue difference after square operation, and the sum represents the RGB difference between the second key point and the corresponding third key point.

[0136] In some embodiments, the posture acquisition module 1204 is also used to adjust the position of the first key point corresponding to the second point cloud according to the registration error if there is a second point cloud with a registration error greater than the first preset error; and again obtain the registration error of the new second point cloud corresponding to the first key point after the adjusted position, until the registration error of the new second point cloud is less than or equal to the first preset error.

[0137] The present application also provides an electronic device, such as Fig.13As shown, the electronic device may include one or more processors 1301 , a memory 1302 and a communication interface 1303 .

[0138] The memory 1302 and the communication interface 1303 are coupled to the processor 1301. For example, the memory 1302, the communication interface 1303 and the processor 1301 may be coupled together via a bus 1304.

[0139] The communication interface 1303 is used for data transmission with other devices. The memory 1302 stores computer program code. The computer program code includes computer instructions. When the computer instructions are executed by the processor 1301, the electronic device executes the posture determination method in the embodiment of the present application.

[0140] Among them, the processor 1301 can be a processor or a controller, for example, it can be a central processing unit (CPU), a general processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It can implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the contents of the embodiments of the present application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0141] The bus 1304 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus 1304 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.13 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0142] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program code is stored. When the processor executes the computer program code, the electronic device executes the relevant method steps in the method embodiment.

[0143] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, it enables the computer to execute the relevant method steps in the above method embodiment.

[0144] Among them, the posture determination device, electronic device, computer storage medium or computer program product provided in this application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0145] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0146] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0147] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0148] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0149] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0150] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for determining a posture, characterized in that: The method comprises: Acquire an RGB image of a first target and a depth image corresponding to the RGB image, and acquire an initial point cloud of the first target; Determine a first point cloud of the first target according to the RGB image and the depth image; Acquire RGB features of the RGB image and point cloud features of the first point cloud, and obtain a completed point cloud of the first target according to the RGB features and the point cloud features; A relative pose between the completed point cloud and the initial point cloud is determined, and a target pose of the first target is determined based on the relative pose and the initial point cloud.

2. The method according to claim 1, characterized in that: The step of determining the target pose of the first target according to the relative pose and the initial point cloud comprises: According to the relative position and posture, transforming the initial point cloud to obtain a transformed target point cloud of the first target; Determine a target pose of the first target according to the target point cloud.

3. The method according to claim 1 or 2, characterized in that: The acquiring includes an RGB image of the first target and a depth image corresponding to the RGB image, including: Acquire at least two image groups including a first target, wherein one of the image groups corresponds to a camera view, and one of the image groups includes an RGB image and a depth image corresponding to the RGB image; A group of images is obtained from the at least two groups of images.

4. The method according to claim 3, characterized in that: The obtaining of the initial point cloud of the first target includes: An initial point cloud of the first target is determined according to the depth images in the at least two image groups.

5. The method according to any one of claims 1 to 4, characterized in that: The step of determining a first point cloud of the first target according to the RGB image and the depth image includes: Determine a position of the first target in the depth image according to the RGB image and the depth image; A first point cloud of the first target is determined according to a camera intrinsic parameter and a position of the first target in the depth image.

6. The method according to any one of claims 1 to 5, characterized in that: The obtaining, according to the RGB features and the point cloud features, a completed point cloud of the first target includes: The RGB feature and the point cloud feature are combined to obtain a first feature; A completed point cloud of the first target is obtained according to the first feature and the RGB feature.

7. The method according to any one of claims 1 to 6, characterized in that: The determining the relative position between the completed point cloud and the initial point cloud comprises: Acquire at least one first key point in the completed point cloud; For each first key point, within a range with the first key point as the center and a preset length as the radius, a second point cloud corresponding to the first key point is obtained; the second point cloud includes RGB information and position information of each point in the second point cloud; For each of the second point clouds, determine a third point cloud corresponding to the second point cloud in the initial point cloud, and determine a relative pose between the second point cloud and the third point cloud; the third point cloud includes RGB information and position information of each point in the third point cloud; Determine a registration error of the second point cloud according to the RGB information and position information of each point in the second point cloud and the RGB information and position information of each point in the third point cloud; If the registration error of each of the second point clouds is less than or equal to the first preset error, the relative pose between the completed point cloud and the initial point cloud is determined according to the relative pose between each of the second point clouds and the third point cloud.

8. The method according to claim 7, characterized in that The second point cloud includes at least one second key point; each of the second key points corresponds to a third key point in the third point cloud; and determining the registration error of the second point cloud according to the RGB information and position information of each point in the second point cloud and the RGB information and position information of each point in the third point cloud includes: For each of the second key points, Determine a distance difference between the second key point and the third key point according to the position information of the second key point and the corresponding position information of the third key point; Determine an RGB difference between the second key point and the third key point according to the RGB information of the second key point and the corresponding RGB information of the third key point; Determining a registration error of the second key point according to the distance difference and the RGB difference; A registration error of the second point cloud is determined according to the registration error of each of the second key points.

9. The method according to claim 8, characterized in that The RGB information of each point in the second point cloud and the third point cloud includes color values ​​corresponding to red, green, and blue respectively; and determining the RGB difference between the second key point and the third key point according to the RGB information of the second key point and the corresponding RGB information of the third key point includes: Respectively determine the red difference, green difference, and blue difference between the red, green, and blue color values ​​in the RGB information of the second key point and the corresponding red, green, and blue color values ​​in the RGB information of the third key point; Obtain a sum of the red difference, the green difference, and the blue difference after square operations are respectively performed, wherein the sum represents an RGB difference between the second key point and the corresponding third key point.

10. The method according to any one of claims 7 to 9, characterized in that: The method further comprises: If there is a second point cloud with a registration error greater than the first preset error, adjusting the position of the first key point corresponding to the second point cloud according to the registration error; The registration error of the new second point cloud corresponding to the first key point after the position adjustment is obtained again, until the registration error of the new second point cloud is less than or equal to the first preset error.

11. A posture determination device, characterized in that: include: An information acquisition module, used to acquire an RGB image of a first target and a depth image corresponding to the RGB image, and to acquire an initial point cloud of the first target; A point cloud acquisition module, used to determine a first point cloud of the first target according to the RGB image and the depth image; A point cloud completion module, used to obtain RGB features of the RGB image and point cloud features of the first point cloud, and obtain a completed point cloud of the first target according to the RGB features and the point cloud features; The pose acquisition module is used to determine the relative pose between the completed point cloud and the initial point cloud, and determine the target pose of the first target according to the relative pose and the initial point cloud.

12. The device according to claim 11, characterized in that The posture acquisition module is further used to transform the initial point cloud according to the relative posture to obtain a transformed target point cloud of the first target; and determine the target posture of the first target according to the target point cloud.

13. The device according to claim 11 or 12, characterized in that The information acquisition module is further used to acquire at least two image groups including the first target, one of the image groups corresponds to a camera perspective, and one of the image groups includes an RGB image and a depth image corresponding to the RGB image; A group of images is obtained from the at least two groups of images.

14. The device according to claim 13, characterized in that The information acquisition module is further used to determine the initial point cloud of the first target based on the depth images in the at least two image groups.

15. The device according to any one of claims 11 to 14, characterized in that: The point cloud acquisition module is also used to determine the position of the first target in the depth image based on the RGB image and the depth image; and determine the first point cloud of the first target based on the camera intrinsic parameters and the position of the first target in the depth image.

16. The device according to any one of claims 11 to 15, characterized in that: The point cloud completion module is also used to fuse the RGB feature and the point cloud feature to obtain a first feature; and obtain a completed point cloud of the first target based on the first feature and the RGB feature.

17. The device according to any one of claims 11 to 16, characterized in that: The posture acquisition module is further used to acquire at least one first key point in the completed point cloud; for each first key point, within a range with the first key point as the center and a preset length as the radius, acquire a second point cloud corresponding to the first key point; the second point cloud includes RGB information and position information of each point in the second point cloud; for each second point cloud, determine a third point cloud corresponding to the second point cloud in the initial point cloud, and determine the relative posture between the second point cloud and the third point cloud; the third point cloud includes RGB information and position information of each point in the third point cloud; Determine a registration error of the second point cloud according to the RGB information and position information of each point in the second point cloud and the RGB information and position information of each point in the third point cloud; If the registration error of each of the second point clouds is less than or equal to the first preset error, the relative pose between the completed point cloud and the initial point cloud is determined according to the relative pose between each of the second point clouds and the third point cloud.

18. The device according to claim 17, characterized in that The second point cloud includes at least one second key point; each of the second key points corresponds to a third key point in the third point cloud; The posture acquisition module is further used to determine, for each of the second key points, a distance difference between the second key point and the third key point according to the position information of the second key point and the position information of the corresponding third key point; Determine an RGB difference between the second key point and the third key point according to the RGB information of the second key point and the corresponding RGB information of the third key point; Determine the registration error of the second key point according to the distance difference and the RGB difference; and determine the registration error of the second point cloud according to the registration error of each of the second key points.

19. The device according to claim 18, characterized in that The RGB information of each point in the second point cloud and the third point cloud includes color values ​​corresponding to red, green and blue respectively; The posture acquisition module is further used to respectively determine the red difference, green difference, and blue difference between the red, green, and blue color values ​​in the RGB information of the second key point and the corresponding red, green, and blue color values ​​in the RGB information of the third key point; Obtain a sum of the red difference, the green difference, and the blue difference after square operations are respectively performed, wherein the sum represents an RGB difference between the second key point and the corresponding third key point.

20. The device according to any one of claims 17 to 19, characterized in that The posture acquisition module is also used to adjust the position of the first key point corresponding to the second point cloud according to the registration error if there is a second point cloud with a registration error greater than the first preset error; and again obtain the registration error of the new second point cloud corresponding to the first key point after the adjusted position until the registration error of the new second point cloud is less than or equal to the first preset error.

21. An electronic device, characterized in that: It includes a memory and one or more processors; the memory is coupled to the processor; the memory stores computer program code, and the computer program code includes computer instructions. When the computer instructions are executed by the processor, the electronic device executes the posture determination method as described in any one of claims 1-10.

Images