Pose determination method and apparatus, and electronic device

Abstract

A pose determination method and apparatus, and an electronic device, relating to the technical field of computer vision. According to the method, a relative pose between a completed point cloud and an initial point cloud is determined on the basis of the completed point cloud, and a target pose of a first target can be more accurately determined on the basis of the relative pose and an initial pose of the first target. The method comprises: an electronic device first acquiring an RGB image comprising a first target and a depth image corresponding to the RGB image, and acquiring an initial point cloud of the first target; then determining a first point cloud of the first target on the basis of the RGB image and the depth image, acquiring an RGB feature of the RGB image and a point cloud feature of the first point cloud, and acquiring a completed point cloud of the first target on the basis of the RGB feature and the point cloud feature; and finally determining a relative pose between the completed point cloud and the initial point cloud, and determining a target pose of the first target on the basis of the relative pose and the initial point cloud.

Description

Position determination method, device and electronic equipment

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on September 4, 2023, with application number 202311137496.2 and application name “Posture determination method, device and electronic device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computer vision technology, and in particular to a method, device and electronic device for determining posture. Background Art

[0003] 6D pose refers to the pose of an object with six degrees of freedom (DOF), consisting of three DOF translation parameters and three DOF rotation parameters. Acquiring and estimating an object's 6D pose can be applied to fields such as robotics, autonomous driving, and virtual reality (VR) / augmented reality (AR). Based on an object's 6D pose, its position in space can be more accurately determined.

[0004] Current methods for estimating the 6D pose of an object use convolutional neural networks (CNNs) and PointNet networks to extract RGB features and point cloud features from the target object's image, respectively. These features are then combined for each key pixel in the image to obtain the 6D pose of the target object. However, the accuracy of the 6D pose estimated in this way is low, making it difficult to meet the actual needs of different scenarios in different fields.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a posture determination method, device and electronic device, which determine the relative posture between the completed point cloud and the initial point cloud, and based on the relative posture and the initial posture of the first target, can more accurately determine the target posture of the first target.

[0007] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0008] In a first aspect, a method for determining a pose is provided. In this method, an electronic device first obtains an RGB image of a first target and a depth image corresponding to the RGB image, as well as an initial point cloud of the first target. Next, a first point cloud of the first target is determined based on the RGB image and the depth image, and RGB features of the RGB image and point cloud features of the first point cloud are obtained. Furthermore, a completed point cloud of the first target is obtained based on the RGB features and the point cloud features. Finally, a relative pose between the completed point cloud and the initial point cloud is determined, and a target pose of the first target is determined based on the relative pose and the initial point cloud.

[0009] In the above method, after obtaining the RGB image and the corresponding depth image and the first point cloud of the first target, a completed point cloud of the first target can be obtained based on the RGB features of the RGB image and the point cloud features of the first point cloud. That is, the first point cloud is completed, improved, or the point cloud information is enhanced, thereby obtaining a completed point cloud. Compared with the first point cloud, the completed point cloud includes more three-dimensional points and can more accurately characterize the position and posture, as well as geometric information, of the first target. Therefore, the relative pose between the completed point cloud and the initial point cloud is determined based on the completed point cloud, and based on the relative pose and the initial pose of the first target, the target pose of the first target can be more accurately determined.

[0010] In one implementation method of the first aspect, when the electronic device determines the target posture of the first target, it can transform the initial point cloud according to the relative posture to obtain the transformed target point cloud of the first target, and then determine the target posture of the first target based on the target point cloud.

[0011] In the above implementation, the purpose of transforming the initial point cloud is to further adjust the pose of the initial point cloud so that the pose of the target point cloud obtained after adjustment is closer to the actual pose of the first target.

[0012] In one implementation of the first aspect, an electronic device acquires at least two image groups including a first object, wherein one image group corresponds to a camera viewpoint and one image group includes an RGB image and a depth image corresponding to the RGB image. The electronic device then acquires one image group from the at least two image groups to obtain the RGB image of the first object and the depth image corresponding to the RGB image.

[0013] In one implementation of the first aspect, the electronic device determines an initial point cloud of the first target based on depth images in at least two image groups. The electronic device may determine the initial point cloud of the first target by combining multiple depth images acquired from different camera perspectives. Such an initial point cloud can be considered a relatively accurate point cloud. Subsequently, in the subsequent pose determination process, the electronic device may use the initial point cloud as a reference or standard point cloud to further obtain a more accurate target point cloud and target pose.

[0014] In one implementation of the first aspect, when determining the first point cloud of the first object, the electronic device determines a position of the first object in the depth image based on the RGB image and the depth image. Thereafter, the electronic device determines the first point cloud of the first object based on camera intrinsic parameters and the position of the first object in the depth image.

[0015] In one implementation of the first aspect, when the electronic device obtains the completed point cloud of the first object, it fuses the RGB feature with the point cloud feature to obtain the first feature. Thereafter, the electronic device obtains the completed point cloud of the first object based on the first feature and the RGB feature.

[0016] In the above implementation method, the completed point cloud refers to the point cloud after the first point cloud is completed, because the first point cloud is determined by the electronic device based on an RGB image and a depth image, which is equivalent to the first point cloud being only the point cloud of the visible part of the first target at one angle. Such a point cloud is incomplete and cannot well reflect the geometric information of the first target. Therefore, in order to obtain a point cloud that can better reflect the geometric information of the first target, the electronic device completes the first point cloud to make the completed point cloud more complete, and enhances the point cloud information compared to the first point cloud.

[0017] In one implementation of the first aspect, when determining the relative pose between a completed point cloud and an initial point cloud, the electronic device obtains at least one first key point in the completed point cloud. For each first key point, the electronic device obtains a second point cloud corresponding to the first key point within a radius centered at the first key point and having a preset length, wherein the second point cloud includes RGB information and position information for each point in the second point cloud. For each second point cloud, the electronic device determines a third point cloud corresponding to the second point cloud in the initial point cloud, and determines the relative pose between the second point cloud and the third point cloud, wherein the third point cloud includes RGB information and position information for each point in the third point cloud. The electronic device determines a registration error of the second point cloud based on the RGB information and position information of each point in the second point cloud and the RGB information and position information of each point in the third point cloud. If the registration error of each second point cloud is less than or equal to the first preset error, the electronic device determines the relative pose between the completed point cloud and the initial point cloud based on the relative pose between each second point cloud and the third point cloud.

[0018] In one implementation of the first aspect, the second point cloud includes at least one second key point; each second key point corresponds to a third key point in the third point cloud. When the electronic device determines the registration error of the second point cloud, for each second key point, the electronic device determines the distance difference between the second key point and the third key point based on the position information of the second key point and the position information of the corresponding third key point. Furthermore, the electronic device determines the RGB difference between the second key point and the third key point based on the RGB information of the second key point and the RGB information of the corresponding third key point. Thereafter, the electronic device determines the registration error of the second key point based on the distance difference and the RGB difference, and determines the registration error of the second point cloud based on the registration error of each second key point.

[0019] In one implementation of the first aspect, the RGB information for each point in the second point cloud and the third point cloud includes color values ​​corresponding to red, green, and blue, respectively. When determining the RGB difference between the second key point and the third key point, the electronic device determines the red difference, green difference, and blue difference between the red, green, and blue color values ​​in the RGB information of the second key point and the corresponding red, green, and blue color values ​​in the RGB information of the third key point. Furthermore, the electronic device obtains the sum of the red difference, green difference, and blue difference after performing square operations on each of the values, where the sum represents the RGB difference between the second key point and the corresponding third key point.

[0020] Among the above-mentioned implementation methods, the key points in the completed point cloud can be optimized and adjusted based on the RGB information and position information of the points in the point cloud. Due to the addition of RGB information, when determining the registration error of the point cloud, the influence of multiple aspects (position information, RGB information) can be combined, so as to more accurately determine the relative pose between the completed point cloud and the initial point cloud, and more accurately determine the target pose of the first target based on the relative pose, while also achieving the purpose of optimizing the pose of the completed point cloud (that is, the target pose of the first target).

[0021] In one implementation method of the first aspect, if there is a second point cloud whose registration error is greater than a first preset error, the electronic device adjusts the position of the first key point corresponding to the second point cloud according to the registration error, and the electronic device again obtains the registration error of the new second point cloud corresponding to the first key point after the adjusted position, until the registration error of the new second point cloud is less than or equal to the first preset error.

[0022] In this implementation, the electronic device can adjust the position of each first key point to optimize the first key point. Furthermore, if the registration error of the second point cloud is less than or equal to the first predetermined error, it indicates that the first key point corresponding to the second point cloud is in the optimal position or has been optimized. In this case, the pose estimation or determination based on the completed point cloud including the first key point is more accurate.

[0023] In an implementation manner of the first aspect, the electronic device fits the relative pose between each second point cloud and the third point cloud based on a least squares method, thereby obtaining the relative pose between the completed point cloud and the initial point cloud.

[0024] In an implementation manner of the first aspect, when the electronic device adjusts the position of the first key point, it can perform operations such as translation and / or rotation.

[0025] In a second aspect, a posture determination device is provided, comprising: an information acquisition module for acquiring an RGB image of a first target and a depth image corresponding to the RGB image, and acquiring an initial point cloud of the first target. A point cloud acquisition module for determining a first point cloud of the first target based on the RGB image and the depth image. A point cloud completion module for acquiring RGB features of the RGB image and point cloud features of the first point cloud, and obtaining a completed point cloud of the first target based on the RGB features and the point cloud features. A posture acquisition module for determining the relative posture between the completed point cloud and the initial point cloud, and determining the target posture of the first target based on the relative posture and the initial point cloud.

[0026] In an implementation manner of the second aspect, the posture acquisition module is further used to transform the initial point cloud according to the relative posture to obtain the transformed target point cloud of the first target; and determine the target posture of the first target according to the target point cloud.

[0027] In an implementation manner of the second aspect, the information acquisition module is further used to acquire at least two image groups including the first target, one image group corresponding to a camera perspective, and one image group including an RGB image and a depth image corresponding to the RGB image; and acquire one image group from the at least two image groups.

[0028] In an implementation manner of the second aspect, the information acquisition module is further configured to determine an initial point cloud of the first target based on the depth images in the at least two image groups.

[0029] In an implementation manner of the second aspect, the point cloud acquisition module is further used to determine the position of the first target in the depth image based on the RGB image and the depth image; and determine the first point cloud of the first target based on the camera intrinsic parameters and the position of the first target in the depth image.

[0030] In an implementation manner of the second aspect, the point cloud completion module is further used to fuse the RGB features and the point cloud features to obtain a first feature; and obtain a completed point cloud of the first target based on the first feature and the RGB feature.

[0031] In an implementation method of the second aspect, the posture acquisition module is also used to obtain at least one first key point in the completed point cloud; for each first key point, a second point cloud corresponding to the first key point is obtained within a range with the first key point as the center and a preset length as the radius; the second point cloud includes the RGB information and position information of each point in the second point cloud; for each second point cloud, a third point cloud corresponding to the second point cloud in the initial point cloud is determined, and the relative posture between the second point cloud and the third point cloud is determined; the third point cloud includes the RGB information and position information of each point in the third point cloud; based on the RGB information and position information of each point in the second point cloud, and the RGB information and position information of each point in the third point cloud, the alignment error of the second point cloud is determined; if the alignment error of each second point cloud is less than or equal to the first preset error, the relative posture between the completed point cloud and the initial point cloud is determined based on the relative posture between each second point cloud and the third point cloud.

[0032] In an implementation method of the second aspect, the second point cloud includes at least one second key point; each second key point corresponds to a third key point in the third point cloud; the posture acquisition module is also used to determine, for each second key point, the distance difference between the second key point and the third key point based on the position information of the second key point and the position information of the corresponding third key point; determine the RGB difference between the second key point and the third key point based on the RGB information of the second key point and the RGB information of the corresponding third key point; determine the alignment error of the second key point based on the distance difference and the RGB difference; determine the alignment error of the second point cloud based on the alignment error of each second key point.

[0033] In one implementation method of the second aspect, the RGB information of each point in the second point cloud and the third point cloud includes color values ​​corresponding to the three colors of red, green, and blue, respectively; the posture acquisition module is also used to respectively determine the red, green, and blue color values ​​in the RGB information of the second key point, and the red difference, green difference, and blue difference between the corresponding red, green, and blue color values ​​in the RGB information of the third key point; obtain the sum of the red difference, green difference, and blue difference after square operation, and the sum represents the RGB difference between the second key point and the corresponding third key point.

[0034] In one implementation method of the second aspect, the posture acquisition module is also used to adjust the position of the first key point corresponding to the second point cloud according to the registration error if there is a second point cloud with a registration error greater than the first preset error; and again obtain the registration error of the new second point cloud corresponding to the first key point after the adjusted position, until the registration error of the new second point cloud is less than or equal to the first preset error.

[0035] In a third aspect, an electronic device is provided, comprising a memory and one or more processors; the memory is coupled to the processor; wherein computer program code is stored in the memory, and the computer program code comprises computer instructions, which, when executed by the processor, enable the electronic device to execute the posture determination method as in the first aspect and any implementation thereof.

[0036] In a fourth aspect, a computer-readable storage medium is provided, comprising computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the posture determination method as in the first aspect and any implementation thereof.

[0037] In a fifth aspect, a computer program product is provided. When the computer program product is run on a computer, the computer is caused to execute the posture determination method as in the first aspect and any implementation thereof.

[0038] It can be understood that the beneficial effects that can be achieved by the posture determination device of the second aspect, the electronic device of the third aspect, the computer-readable storage medium of the fourth aspect, and the computer program product of the fifth aspect can be referred to the beneficial effects of the first aspect and any possible design method thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] FIG1 is a schematic diagram of a pose estimation method based on feature matching according to an embodiment of the present application;

[0040] FIG2 is a schematic diagram of a pose estimation method based on deep learning according to an embodiment of the present application;

[0041] FIG3 is a schematic diagram showing the connection between an electronic device and a camera according to an embodiment of the present application;

[0042] FIG4 is a schematic diagram of an application scenario shown in an embodiment of the present application;

[0043] FIG5 is a flowchart of a method for determining a posture according to an embodiment of the present application;

[0044] FIG6 is a schematic diagram of an image group shown in an embodiment of the present application;

[0045] FIG7 is a second flow chart of a method for determining a posture according to an embodiment of the present application;

[0046] FIG8 is a third flow chart of a method for determining a posture according to an embodiment of the present application;

[0047] FIG9 is a fourth flow chart of a method for determining a posture according to an embodiment of the present application;

[0048] FIG10 is a fifth flow chart of a method for determining a posture according to an embodiment of the present application;

[0049] FIG11 is a sixth flow chart of a method for determining a posture according to an embodiment of the present application;

[0050] FIG12 is a schematic diagram of the structure of a posture determination device shown in an embodiment of the present application;

[0051] FIG13 is a schematic structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. Among them, in the description of the present application, unless otherwise specified, " / " indicates that the objects associated before and after are in an "or" relationship. For example, A / B can represent A or B; "and / or" in the present application is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. In addition, in the description of the present application, unless otherwise specified, "multiple" refers to two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple. In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. Those skilled in the art will understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit differences. At the same time, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or design schemes. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way for easy understanding.

[0053] In addition, the business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Ordinary technicians in this field can know that with the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0054] The pose represents the position and posture of a person or object in space, and the pose can generally include 3D and 6D poses. Among them, the 3D pose uses three-degree-of-freedom translation parameters to represent the pose, for example, the translation parameters in the three directions (or degrees of freedom) of x, y, and z in a specified coordinate system are used to represent the pose. The 6D pose refers to the use of six-degree-of-freedom parameters to represent the pose, and the six degrees of freedom include, for example, the aforementioned three-degree-of-freedom translation parameters and three-degree-of-freedom spatial rotation parameters. The acquisition and estimation of the 6D pose of an object can be applied to fields such as robotics, autonomous driving, virtual reality (VR) / augmented reality (AR), and based on the 6D pose of the object, the position of the object in space can be determined more accurately.

[0055] At present, the methods for estimating the 6D pose of an object can be roughly divided into feature matching-based methods (such as the DenseFusion method) and deep learning-based methods (such as the FFB6D method).

[0056] As shown in (a) of Figure 1, in the feature matching-based method, a fusion method of pixel-level RGB features and point cloud features is designed, that is, for the image of the target object, convolutional neural networks (CNN) and point cloud networks PointNet are used to extract RGB features and point cloud features respectively, and then DenseFusion and other methods are used to splice the RGB features and point cloud features for each key pixel in the image. After that, global features are extracted based on the Multilayer Perceptron (MLP), and the 6D pose of the target object is obtained based on the global features and spliced ​​features. And, after obtaining the 6D pose, as shown in Figure 1 (b), the 6D pose predicted or obtained in the previous round is regarded as the standard pose of the target object, the RGB features and point cloud features are re-extracted, and the two are spliced ​​again to obtain the new 6D pose of the target object. According to the 6D pose obtained last time, the point cloud of the target object corresponding to the new 6D pose is rotated, and the error or residual pose between the point cloud corresponding to the new 6D pose and the point cloud corresponding to the previous 6D pose is determined. Continue to repeat the steps shown in Figure 1 (b), constantly adjusting the position of the point cloud. After K rounds of iterations, the final 6D pose of the target object can be obtained. This method can be regarded as a process for optimizing the 6D pose, or as a process for iterative optimization of the above-mentioned pose estimation method.

[0057] However, in the above-mentioned 6D pose estimation method, the fusion method for features of different modalities (such as RGB features and point cloud features) is simply splicing. The 6D accuracy determined based on this method is low, which makes it difficult for the 6D pose to meet the actual scene requirements of different fields. In addition, iterative optimization only uses the 6D pose estimation method to optimize itself. This optimization method has low generalization ability and is only used for the current scene. It is difficult to apply to other scenes. For example, when the lighting conditions in other scenes are different from those in the current scene, the 6D pose estimation method obtained by this optimization method will not estimate a very accurate 6D pose in other scenes.

[0058] As shown in Figure 2, in the deep learning-based method, for each key pixel in the image, a deep neural network is used to fuse RGB features and point cloud features. Then, the displacement between the pixel and the pre-selected preset key point is predicted one by one, and the coordinates of the key pixel are obtained by voting clustering. Finally, based on the coordinates of each key pixel, the least squares method is used to obtain the 6D pose of the target object.

[0059] However, this 6D pose estimation method does not have an iterative optimization process, nor can it guarantee that the estimated 6D pose has high accuracy, nor can it meet the actual scenario requirements in different fields.

[0060] Based on the above content, an embodiment of the present application provides a posture determination method. In this method, first, an RGB image of a first target and a depth image corresponding to the RGB image are obtained, and an initial point cloud of the first target is obtained. Then, based on the RGB image and the depth image, a first point cloud of the first target is determined, and the RGB features of the RGB image and the point cloud features of the first point cloud are obtained, and based on the RGB features and the point cloud features, a completed point cloud of the first target is obtained. Finally, the relative posture between the completed point cloud and the initial point cloud is determined, and the target posture of the first target is determined based on the relative posture and the initial point cloud.

[0061] In the posture determination method of the embodiment of the present application, after obtaining the RGB image and the corresponding depth image and the first point cloud of the first target, the completed point cloud of the first target can be obtained based on the RGB features of the RGB image and the point cloud features of the first point cloud, that is, the first point cloud is completed, improved, or the point cloud information is enhanced, thereby obtaining a completed point cloud. Compared with the first point cloud, the completed point cloud includes more three-dimensional points and can more accurately characterize the position and posture, as well as the geometric information, of the first target. Therefore, the relative posture between the completed point cloud and the initial point cloud is determined based on the completed point cloud, and based on the relative posture and the initial posture of the first target, the target posture of the first target can be more accurately determined.

[0062] The above-mentioned posture determination method provided in the embodiment of the present application can be applied to electronic devices with computing and processing capabilities, such as computers, servers, hosts, VR devices, AR devices, robots, and vehicles. If the VR device, AR device, robot, vehicle, and other devices do not have computing and processing capabilities, then the electronic device that implements the above-mentioned posture determination method is connected to the VR device, AR device, robot, vehicle, and other devices, and the posture determination result of the target is sent to these devices. For example, the VR device, AR device, robot, vehicle, etc. can be connected to the server, and RGB images and depth images, etc. are sent to the server. The server determines the posture of the target through the above-mentioned posture determination method, and sends the posture determination result to the VR device, AR device, robot, vehicle, etc.

[0063] In some possible application scenarios, as shown in FIG3(a), electronic device 301 may be connected to RGB camera 302, which is used to capture RGB images including a first target from different camera perspectives. During capture, as shown in FIG3(b), the RGB camera may capture RGB images A1, A2, A3, and so on, from different camera perspectives at equal intervals on a sphere with a fixed radius and centered on first target A. After capturing the RGB images, RGB camera 302 transmits the RGB images to electronic device 301. After receiving the RGB images from multiple perspectives, electronic device 301 may utilize a depth estimation algorithm to obtain a corresponding depth image for each RGB image. For example, depth image a1 may be obtained for RGB image A1, depth image a2 may be obtained for RGB image A2, and depth image a3 may be obtained for RGB image A3.

[0064] In other possible application scenarios, the electronic device 301 may also be connected to a depth camera (such as RGB-D), which may obtain the RGB image and depth image of the first target A from the same camera perspective, and send the RGB image and depth image to the electronic device 301, respectively.

[0065] The electronic device 301 in the embodiment of the present application can determine the posture for targets in actual scenes in different fields. For example, referring to FIG4 , for VR or AR scenes, an RGB camera or a depth camera is deployed on the VR or AR device to shoot the environment around the user of the VR or AR device, so that the VR or AR device can estimate the posture (such as 3D posture, 6D posture, etc.) of objects around the user based on the image captured by the camera, thereby realizing the environmental perception function. For another example, referring to FIG4 , for industrial robot scenes, an RGB camera or a depth camera is deployed on the manipulator or around the manipulator of the industrial robot to shoot the image of the object that the manipulator needs to operate, so that the robot or the electronic device connected to the camera can estimate the posture (such as 3D posture, 6D posture, etc.) of the object to be operated based on the image captured by the camera, thereby guiding the manipulator to accurately grasp the object. As another example, referring to FIG4 , for intelligent driving scenarios, an RGB camera or a depth camera is deployed on an intelligent driving vehicle to capture scenes around the vehicle. The vehicle or an electronic device connected to the camera estimates the posture (such as 3D posture, 6D posture, etc.) of surrounding vehicles or pedestrians based on the images captured by the camera, thereby realizing vehicle distance estimation and scene perception functions, and providing a basis for the driving and decision-making of the intelligent driving vehicle.

[0066] In other possible application scenarios, the electronic device 301 can also determine the position of a known model, wherein the known model can be a computer aided design (CAD) model, etc. Moreover, the model can be any person or object in actual scenes in different fields. In this case, the electronic device 301 can display an image of the known model, and on a sphere with a fixed length as the radius and centered on the known model, collect RGB images and depth images of the known model from multiple perspectives through an RGB camera or a depth camera at equal distances. Alternatively, after obtaining the known model, the electronic device 301 rotates the known model at a fixed angle and obtains the known model image after each rotation, thereby achieving the purpose of collecting RGB images and depth images of the known model from multiple perspectives.

[0067] Taking the posture determination method in the embodiment of the present application applied to the above-mentioned electronic device as an example, as shown in Figure 5, the posture determination method may include the following steps S501-S504.

[0068] S501: The electronic device obtains an RGB image of a first object and a depth image corresponding to the RGB image, and obtains an initial point cloud of the first object.

[0069] In some embodiments, the electronic device may first acquire at least two image groups including a first target, wherein one image group corresponds to a camera perspective, and one image group includes an RGB image and a depth image corresponding to the RGB image, as shown in FIG6 . It is understandable that the depth image may be acquired by a depth camera, or may be determined by the electronic device through a depth estimation algorithm based on the RGB image, and this is not specifically limited in this embodiment. The depth image includes the depth information of each pixel in the corresponding RGB image, for example, the distance between each pixel and the camera (such as), the distance between each pixel and a reference point, etc.

[0070] Afterwards, the electronic device acquires an image group from the at least two image groups, thereby obtaining an RGB image of the first object and a depth image corresponding to the RGB image. Furthermore, the acquired image group is any one of the at least two image groups.

[0071] In addition, the image including the first target (such as an RGB image, a depth image, etc.) also includes other objects. The purpose of the embodiment of the present application is to determine the posture of the first target in the acquired image. The first target can be any object in the image (such as the banana shown in the dotted box in Figure 6) or any person.

[0072] In some embodiments, the electronic device may further determine an initial point cloud of the first target based on the depth images in the at least two image groups obtained above. Thus, the electronic device may determine an initial point cloud of the first target by combining multiple depth images obtained from different camera perspectives. Such an initial point cloud may be considered a relatively accurate point cloud. Subsequently, in the subsequent pose determination process, the electronic device may use the initial point cloud as a reference or standard point cloud, based on which to further obtain a more accurate target point cloud and target pose.

[0073] This step S501 can be regarded as a preparation stage or a preprocessing stage for posture determination.

[0074] S502: The electronic device determines a first point cloud of a first target based on the RGB image and the depth image.

[0075] In some embodiments, when determining the first point cloud of the first target, the electronic device may first determine the position of the first target in the depth image based on the RGB image and the depth image, and determine the first point cloud of the first target based on the camera intrinsic parameters and the position of the first target in the depth image. Determining the position of the first target in the depth image may also be referred to as instance segmentation of the first target. For example, the position of the first target in the depth image may be shown as a rectangular box in the depth image.

[0076] In some possible application scenarios, as shown in FIG7 , an electronic device can utilize a pose estimation network to implement instance segmentation of a first target. Specifically, the electronic device can input an RGB image and a depth image into the pose estimation network. The pose estimation network determines the position of the first target in the depth image using a preset pose estimation algorithm, and simultaneously outputs a first pose of the first target. The first pose here is the pose estimated by the pose estimation network for the first target.

[0077] Then, as shown in Figure 7, the electronic device determines a first point cloud for the first object based on the camera intrinsic parameters and the position of the first object in the depth image. Alternatively, the electronic device converts the segmentation results obtained by the pose estimation network into a point cloud of the visible portion of the first object based on the camera intrinsic parameters.

[0078] It can be understood that the camera coordinate system can be determined by the camera intrinsic parameters, so the point cloud of the visible part of the first target is the point cloud in the camera coordinate system.

[0079] In other possible application scenarios, as shown in FIG7 , after obtaining the first point cloud, the electronic device may further delete outliers in the first point cloud to make the first point cloud more accurate. Furthermore, the electronic device may, for example, remove outliers in the first point cloud based on a MeanShift clustering method, which is not specifically limited in the embodiments of the present application.

[0080] In some embodiments, the first pose is used to convert the point cloud in camera coordinates to object coordinates. In step S502, the electronic device has already acquired a first point cloud of the first target. To ensure consistency in the comparison of the first target information during the subsequent determination of the pose, the electronic device, after acquiring the first point cloud, converts the first point cloud from the camera coordinate system to the object coordinate system based on the first pose.

[0081] S503: The electronic device obtains RGB features of the RGB image and point cloud features of the first point cloud, and obtains a completed point cloud of the first target based on the RGB features and the point cloud features.

[0082] In some embodiments, the electronic device may fuse RGB features and point cloud features to first obtain a first feature, and then obtain a completed point cloud of the first target based on the first feature and the RGB feature.

[0083] Furthermore, in some possible application scenarios, the electronic device can obtain a completed point cloud of the first target based on a point cloud completion network, and the input of the point cloud completion network is an RGB image and a first point cloud, and the output is a completed point cloud.

[0084] It can be understood that the completed point cloud refers to the point cloud after the first point cloud is completed, because the first point cloud is determined by the electronic device based on an RGB image and a depth image, which is equivalent to the first point cloud being only the point cloud of the visible part of the first target at one angle. Such a point cloud is not complete and cannot well reflect the geometric information of the first target. Therefore, in order to obtain a point cloud that can better reflect the geometric information of the first target, the electronic device completes the first point cloud to make the completed point cloud more complete, and enhance the point cloud information compared to the first point cloud.

[0085] Based on this, the complement operation for the first point cloud in the embodiment of the present application can also be called a perfection operation, a point cloud information enhancement operation, etc., and the embodiment of the present application does not make any specific limitations on this.

[0086] Exemplarily, as shown in FIG8 , the point cloud completion network includes a feature extraction unit and a point cloud completion unit. The feature extraction unit may further include an RGB feature extraction unit, a full flow bidirectional fusion network (FFB) unit, a DenseFusion unit, and a point cloud feature extraction unit. The RGB feature extraction unit and the point cloud feature extraction unit extract RGB features of the RGB image and point cloud features of the first point cloud, respectively. The FFB unit and the DenseFusion unit may fuse the RGB features with the point cloud features to obtain the first features.

[0087] The feature extraction unit sends the first feature and the RGB feature to the point cloud completion unit, and the point cloud completion unit then completes the first point cloud according to the first feature and the RGB feature to obtain the completed point cloud of the first target.

[0088] In some possible application scenarios, the above-mentioned point cloud feature extraction unit can also be called a point cloud encoder (such as a PCN ENC module), and the point cloud completion unit can also be called a point cloud decoder (such as a PCN DEC module).

[0089] In other embodiments, to improve the geometric integrity and pose accuracy of the completed point cloud, the electronic device may also iteratively optimize the point cloud completion network during training. For example, as shown in FIG9 , the electronic device may incorporate a key point detection unit. Iterative optimization of the point cloud completion network can also be viewed as a process of using the key point detection unit to update or optimize parameters of the feature extraction unit and the point cloud completion unit. The key point detection unit includes an MLP. The main function of the key point detection unit is: in each round of training, multiple key points of the first target are determined respectively, and the feature extraction unit and the point cloud completion unit are used to obtain the completed point cloud for the key points determined in this round, and the MLP is used to determine the key point error Lof and the point cloud error Lcd of each round before and after, and the parameters of each unit in the point cloud completion network (such as FFB unit, DenseFusion unit, point cloud feature extraction unit, point cloud completion unit, etc.) are adjusted based on Lof and Lcd to minimize Lof and Lcd as much as possible, thereby enhancing the point cloud completion network's utilization of the information of the input first point cloud, improving the pose accuracy of the output completed point cloud, and ensuring the stability of the completed point cloud.

[0090] In addition, the above-mentioned iterative optimization of the point cloud completion network is equivalent to utilizing other methods or units (such as the above-mentioned key point detection unit) in addition to the point cloud completion network, and does not use the point cloud completion network to iteratively optimize itself. Therefore, this optimization method has a high generalization ability and can be used for point cloud completion networks in a variety of scenarios, fully considering the characteristics of different scenarios.

[0091] S504: The electronic device determines the relative pose between the completed point cloud and the initial point cloud, and determines the target pose of the first target based on the relative pose and the initial point cloud.

[0092] In some embodiments, when determining the target posture of the first target, the electronic device can transform the initial point cloud according to the relative posture to obtain the transformed target point cloud of the first target, and determine the target posture of the first target based on the target point cloud.

[0093] For example, transforming the initial point cloud based on the relative pose can be performed by superimposing the relative pose on the initial pose corresponding to the initial point cloud, thereby achieving the purpose of transforming the initial point cloud. The purpose of transforming the initial point cloud is to further adjust the pose of the initial point cloud so that the pose of the target point cloud obtained after adjustment is closer to the actual pose of the first target.

[0094] In some embodiments, as shown in Figure 10, when the electronic device determines the relative pose between the completed point cloud and the initial point cloud, it can first obtain at least one first key point in the completed point cloud (such as the hollow point in Figure 10), and then use the initial point cloud determined in the above step S501 to align and optimize the at least one first key point, so as to obtain the relative pose between the completed point cloud and the initial point cloud.

[0095] In some possible application scenarios, when registering each first key point, the electronic device can obtain a second point cloud corresponding to the first key point within a range with a preset length as the radius and centered on the first key point, wherein the second point cloud includes RGB information and position information for each point in the second point cloud. It is understandable that each point in the second point cloud can find a corresponding pixel point in the RGB image, and the pixel point in the RGB image includes RGB information, that is, color information, so each point in the second point cloud also includes RGB information. In addition, each point in the second point cloud also has its own position information, such as three-dimensional spatial coordinates.

[0096] Afterwards, the electronic device determines, for each second point cloud, the third point cloud corresponding to the second point cloud in the initial point cloud, and determines the relative position between the second point cloud and the third point cloud, wherein the third point cloud includes RGB information and position information for each point in the third point cloud. In addition, each point in the third point cloud can find a corresponding pixel point in the RGB image. The pixel point in the RGB image includes RGB information, that is, color information, so each point in the third point cloud also includes RGB information. In addition, each point in the third point cloud also has its own position information, such as three-dimensional spatial coordinates.

[0097] The electronic device then determines a registration error for the second point cloud based on the RGB information and position information of each point in the second point cloud and the RGB information and position information of each point in the third point cloud. Furthermore, if the registration error for each second point cloud is less than or equal to the first predetermined error, the electronic device determines a relative pose between the completed point cloud and the initial point cloud based on the relative pose between each second point cloud and the third point cloud.

[0098] In some possible application scenarios, when obtaining a second point cloud corresponding to a first key point, the electronic device may first select at least one second key point within a range with a preset radius centered on the first key point, thereby generating a second point cloud using the at least one second key point. Specifically, the second point cloud includes the at least one second key point. Subsequently, the electronic device finds a third key point corresponding to each second key point in the initial point cloud, thereby obtaining at least one third key point. The electronic device then generates a third point cloud using the at least one third key point. Specifically, the third point cloud includes the at least one third key point.

[0099] Afterwards, the electronic device can further determine the registration error of the second point cloud based on the RGB information and position information of each point in the second point cloud (i.e., each second key point) and the RGB information and position information of each point in the third point cloud (i.e., the third key point).

[0100] Among them, for each second key point, the electronic device determines the distance difference between the second key point and the third key point based on the position information of the second key point and the position information of the corresponding third key point, for example, using the three-dimensional space coordinates of the second key point and the three-dimensional space coordinates of the third key point to make a difference, and then taking the absolute value; and the electronic device then determines the RGB difference between the second key point and the third key point based on the RGB information of the second key point and the RGB information of the corresponding third key point, wherein the RGB information includes color values ​​corresponding to the three colors red, green, and blue, respectively, which are usually values ​​between 0-255.

[0101] Furthermore, the electronic device can respectively determine the red, green, and blue color values ​​in the RGB information of the second key point, and the red difference ΔR, green difference ΔG, and blue difference ΔB between the corresponding red, green, and blue color values ​​in the RGB information of the third key point, and then obtain the sum ΔR of the red difference, green difference, and blue difference after square operation. 2 +ΔG 2 +ΔB 2 , then this sum value can represent the RGB difference between the second key point and the corresponding third key point.

[0102] When determining the registration error of the second point cloud, the electronic device may determine the registration error of the second key point based on the distance difference and RGB difference between the second key point and the third key point. Finally, the electronic device determines the registration error of the second point cloud based on the registration error of each second key point.

[0103] Exemplarily, the electronic device calculates an average value of the registration error of each second key point, thereby obtaining the registration error of the second point cloud.

[0104] In some possible application scenarios, if the registration error of each second point cloud is less than or equal to a first preset error, it can indicate that the completed point cloud and the initial point cloud are successfully registered. The electronic device can then determine the relative pose between the completed point cloud and the initial point cloud based on the relative pose between each second point cloud and the third point cloud. For example, the electronic device can use a least squares method, polynomial fitting, or other method to fit the relative pose between each second point cloud and the third point cloud to obtain the relative pose between the completed point cloud and the initial point cloud.

[0105] If there is a second point cloud with a registration error greater than the first preset error, it means that the second point cloud has not been successfully registered with the corresponding third point cloud, and the electronic device can adjust the position of the first key point corresponding to the second point cloud according to the registration error of the second point cloud. It can be understood that after the position of the first key point is adjusted, the position of the second key point in the second point cloud will also change, and then there will be a new relative posture between the second point cloud and the corresponding third point cloud. Afterwards, the electronic device can again obtain the registration error of the new second point cloud corresponding to the first key point after the adjustment according to the content of the aforementioned embodiment, until the registration error of the new second point cloud is less than or equal to the first preset error. Afterwards, when the registration error of each second point cloud is less than or equal to the first preset error, the electronic device can determine the relative posture between the completed point cloud and the initial point cloud according to the current relative posture corresponding to each second point cloud.

[0106] With reference to the above approach, the electronic device can adjust the position of each first key point, thereby optimizing the first key point. Furthermore, if the registration error of the second point cloud is less than or equal to the first predetermined error, it indicates that the first key point corresponding to the second point cloud is in the optimal position or has been optimized. In this case, the pose estimation or determination based on the completed point cloud including the first key point is more accurate.

[0107] Also, it can be understood that adjusting the position of the first key point may include: performing translation and / or rotation operations on the first key point, and the specific adjustment method may be determined according to the registration error of the second point cloud.

[0108] In the above process of determining the registration error of the second point cloud, in addition to comparing the positions of the key points, the comparison of RGB information is also introduced. By comparing the RGB information, the difference or error between the key points can be more accurately determined, thereby more accurately determining the point cloud registration error corresponding to the key points and the relative position between the completed point cloud and the initial point cloud.

[0109] In some embodiments, when the electronic device obtains at least one first key point in the completed point cloud, as shown in FIG11, first, for an image group, the electronic device obtains at least one two-dimensional feature point 1101 of the first target from the RGB image of the image group, and obtains the depth information corresponding to each two-dimensional feature point 1101 from the depth image, wherein the at least one two-dimensional feature point 1101 is a portion of all pixels in the RGB image. In addition, in some possible application scenarios, the electronic device can obtain the at least one two-dimensional feature point 1101 from the RGB image based on the SIFT algorithm, which is not specifically limited in the embodiments of the present application.

[0110] Then, as shown in FIG11 , for at least two image groups acquired by the electronic device, the electronic device maps at least one two-dimensional feature point 1101 to a three-dimensional space based on camera intrinsic parameters, at least one two-dimensional feature point 1101 in each RGB image, and depth information corresponding to each two-dimensional feature point 1101 in each RGB image, thereby obtaining at least one three-dimensional feature point 1102. The camera intrinsic parameters refer to the intrinsic parameters of the RGB camera that captures the RGB images, or the intrinsic parameters of the depth camera that captures the RGB images and the depth images.

[0111] The electronic device can match the two-dimensional feature points 1101 between each RGB image, determine multiple two-dimensional feature points 1101 belonging to the same target point on the first target, and based on the depth information of each two-dimensional feature point 1101 and the camera intrinsic parameters, map the multiple two-dimensional feature points belonging to the same target point to the three-dimensional space, so that the target point becomes a three-dimensional feature point 1102, and has three-dimensional space position information or three-dimensional space coordinates.

[0112] In addition, the camera coordinate system can be determined through the camera intrinsic parameters, and the depth information of the two-dimensional feature point 1101 can represent the distance between the two-dimensional feature point 1101 and the camera. Then the three-dimensional space coordinates or three-dimensional space position information of the three-dimensional feature point 1102 determined based on the two-dimensional feature point 1101 represents the coordinates or position information of the three-dimensional feature point 1102 in the camera coordinate system.

[0113] In the above manner, the electronic device can map all the matched two-dimensional feature points 1101 in the multiple RGB images to the three-dimensional space, thereby obtaining at least one three-dimensional feature point 1102 of the first object.

[0114] Finally, as shown in FIG11 , the electronic device obtains at least one first key point 1103 of the first target from the at least one three-dimensional feature point 1102. The at least one first key point 1103 may be randomly selected, and in some possible application scenarios, the electronic device may randomly select the at least one first key point 1103 from the at least one three-dimensional feature point 1102 based on an FPS algorithm, which is not specifically limited in the embodiments of the present application.

[0115] In addition, in some possible application scenarios, after the above step S502, the first point cloud is in the object coordinate system, and the initial point cloud is still in the camera coordinate system. Therefore, in step S504, in order to ensure the accuracy of the registration error and the relative pose determination, the electronic device can also transform the completed point cloud according to the first pose of the first target obtained previously. Since the first pose represents the estimated pose of the first target in the camera coordinate system, the completed point cloud can be transformed from the current object coordinate system back to the camera coordinate system after being transformed according to the first pose, and compared with the initial point cloud in the camera coordinate system to determine the relative pose.

[0116] Then, it can be understood that the target posture of the first target subsequently determined by the electronic device is also the target posture of the first target in the camera coordinates.

[0117] The pose determination method provided in the above embodiment, after obtaining the RGB image and the corresponding depth image and the first point cloud of the first target, can obtain the completed point cloud of the first target based on the RGB features of the RGB image and the point cloud features of the first point cloud, that is, the first point cloud is completed, improved, or the point cloud information is enhanced, thereby obtaining a completed point cloud. Compared with the first point cloud, the completed point cloud includes more three-dimensional points and can more accurately characterize the position and posture, as well as the geometric information, of the first target. Therefore, the relative pose between the completed point cloud and the initial point cloud is determined based on the completed point cloud, and based on the relative pose and the initial pose of the first target, the target pose of the first target can be more accurately determined.

[0118] In addition, the above method can also optimize and adjust the key points in the completed point cloud based on the RGB information and position information of the points in the point cloud. Due to the addition of RGB information, when determining the registration error of the point cloud, the influence of multiple aspects (position information, RGB information) can be combined, so as to more accurately determine the relative pose between the completed point cloud and the initial point cloud, and more accurately determine the target pose of the first target based on the relative pose, while also achieving the purpose of optimizing the pose of the completed point cloud (that is, the target pose of the first target).

[0119] In the aforementioned embodiments, the initial point cloud of the first target is used as a reference, and the initial point cloud is transformed according to the relative pose. In other embodiments, the completed point cloud of the first target can also be used as a reference, and the completed point cloud is transformed according to the relative pose to obtain the target point cloud of the first target. This can also achieve the purpose of more accurate pose determination.

[0120] In some schemes, multiple embodiments of the present application can be combined, and the combined scheme can be implemented. Optionally, some operations in the process of each method embodiment are optionally combined, and / or the order of some operations is optionally changed. In addition, the execution order between the steps of each process is only exemplary and does not constitute a restriction on the execution order between the steps. There can also be other execution orders between the steps. It is not intended to indicate that the execution order is the only order in which these operations can be performed. Those of ordinary skill in the art will think of many ways to reorder the operations described in the embodiments of the present application. In addition, it should be noted that the process details involved in a certain embodiment of the present application are also applicable to other embodiments in a similar manner, or different embodiments can be used in combination.

[0121] Furthermore, some steps in the method embodiments may be equivalently replaced with other possible steps. Alternatively, some steps in the method embodiments may be optional and may be deleted in certain usage scenarios. Alternatively, other possible steps may be added to the method embodiments.

[0122] Furthermore, the various method embodiments may be implemented separately or in combination.

[0123] It is understandable that in order to implement the above functions, the aforementioned electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of this application.

[0124] In this embodiment, the electronic device can be divided into functional modules according to the above-mentioned method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into a single processing module. The above-mentioned integrated modules can be implemented in the form of hardware. It should be noted that the module division in this embodiment is illustrative and is only a logical functional division. In actual implementation, other division methods may be used.

[0125] An embodiment of the present application also provides a posture determination device, as shown in Figure 12, which may include an information acquisition module 1201, a point cloud acquisition module 1202, a point cloud completion module 1203 and a posture acquisition module 1204.

[0126] The information acquisition module 1201 is configured to acquire an RGB image of the first target and a depth image corresponding to the RGB image, and to acquire an initial point cloud of the first target.

[0127] The point cloud acquisition module 1202 is configured to determine a first point cloud of a first target based on the RGB image and the depth image.

[0128] The point cloud completion module 1203 is used to obtain the RGB features of the RGB image and the point cloud features of the first point cloud, and obtain the completed point cloud of the first target based on the RGB features and the point cloud features.

[0129] The pose acquisition module 1204 is used to determine the relative pose between the completed point cloud and the initial point cloud, and determine the target pose of the first target based on the relative pose and the initial point cloud.

[0130] In some embodiments, the posture acquisition module 1204 is further configured to transform the initial point cloud according to the relative posture to obtain a transformed target point cloud of the first target; and determine the target posture of the first target according to the target point cloud.

[0131] In some embodiments, the information acquisition module 1201 is further used to acquire at least two image groups including the first target, one image group corresponding to a camera perspective, and one image group including an RGB image and a depth image corresponding to the RGB image; and acquire one image group from the at least two image groups.

[0132] In some embodiments, the information acquisition module 1201 is further configured to determine an initial point cloud of the first target based on the depth images in the at least two image groups.

[0133] In some embodiments, the point cloud acquisition module 1202 is further used to determine the position of the first target in the depth image based on the RGB image and the depth image; and determine the first point cloud of the first target based on the camera intrinsic parameters and the position of the first target in the depth image.

[0134] In some embodiments, the point cloud completion module 1203 is further configured to fuse the RGB features and the point cloud features to obtain a first feature; and obtain a completed point cloud of the first target based on the first feature and the RGB feature.

[0135] In some embodiments, the posture acquisition module 1204 is also used to obtain at least one first key point in the completed point cloud; for each first key point, a second point cloud corresponding to the first key point is obtained within a range with the first key point as the center and a preset length as the radius; the second point cloud includes the RGB information and position information of each point in the second point cloud; for each second point cloud, the third point cloud corresponding to the second point cloud in the initial point cloud is determined, and the relative posture between the second point cloud and the third point cloud is determined; the third point cloud includes the RGB information and position information of each point in the third point cloud; based on the RGB information and position information of each point in the second point cloud, and the RGB information and position information of each point in the third point cloud, the registration error of the second point cloud is determined; if the registration error of each second point cloud is less than or equal to the first preset error, the relative posture between the completed point cloud and the initial point cloud is determined based on the relative posture between each second point cloud and the third point cloud.

[0136] In some embodiments, the second point cloud includes at least one second key point; each second key point corresponds to a third key point in the third point cloud; the posture acquisition module 1204 is also used to determine, for each second key point, the distance difference between the second key point and the third key point based on the position information of the second key point and the position information of the corresponding third key point; determine the RGB difference between the second key point and the third key point based on the RGB information of the second key point and the RGB information of the corresponding third key point; determine the alignment error of the second key point based on the distance difference and the RGB difference; determine the alignment error of the second point cloud based on the alignment error of each second key point.

[0137] In some embodiments, the RGB information of each point in the second point cloud and the third point cloud includes color values ​​corresponding to the three colors red, green, and blue, respectively; the posture acquisition module 1204 is also used to respectively determine the red difference, green difference, and blue difference between the red, green, and blue color values ​​in the RGB information of the second key point and the corresponding red, green, and blue color values ​​in the RGB information of the third key point; obtain the sum of the red difference, green difference, and blue difference after square operation, and the sum represents the RGB difference between the second key point and the corresponding third key point.

[0138] In some embodiments, the posture acquisition module 1204 is also used to adjust the position of the first key point corresponding to the second point cloud according to the registration error if there is a second point cloud with a registration error greater than the first preset error; and again obtain the registration error of the new second point cloud corresponding to the first key point after the adjusted position, until the registration error of the new second point cloud is less than or equal to the first preset error.

[0139] An embodiment of the present application further provides an electronic device, as shown in FIG13 , which may include one or more processors 1301 , a memory 1302 , and a communication interface 1303 .

[0140] The memory 1302 and the communication interface 1303 are coupled to the processor 1301. For example, the memory 1302, the communication interface 1303 and the processor 1301 may be coupled together via a bus 1304.

[0141] The communication interface 1303 is used to transmit data with other devices. The memory 1302 stores computer program code. The computer program code includes computer instructions. When the computer instructions are executed by the processor 1301, the electronic device executes the posture determination method in the embodiment of the present application.

[0142] Among them, the processor 1301 can be a processor or a controller, for example, a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a transistor logic device, a hardware component or any combination thereof. It can implement or execute the various exemplary logic blocks, modules and circuits described in conjunction with the embodiments of the present application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0143] The bus 1304 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus 1304 may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, FIG13 shows only one thick line, but this does not imply that there is only one bus or only one type of bus.

[0144] An embodiment of the present application further provides a computer-readable storage medium, in which computer program code is stored. When the processor executes the computer program code, the electronic device executes the relevant method steps in the method embodiment.

[0145] An embodiment of the present application further provides a computer program product, which, when executed on a computer, enables the computer to execute the relevant method steps in the above method embodiment.

[0146] Among them, the posture determination device, electronic device, computer storage medium or computer program product provided in this application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0147] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0148] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0149] The units described as separate components may or may not be physically separate, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0150] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0151] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the contributing part or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0152] The above content is only a specific embodiment of this application, but the scope of protection of this application is not limited to this. Any changes or replacements within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for determining a posture, characterized in that: The method comprises: Acquire an RGB image of a first target and a depth image corresponding to the RGB image, and acquire an initial point cloud of the first target; Determine a first point cloud of the first target according to the RGB image and the depth image; Acquire RGB features of the RGB image and point cloud features of the first point cloud, and obtain a completed point cloud of the first target according to the RGB features and the point cloud features; A relative pose between the completed point cloud and the initial point cloud is determined, and a target pose of the first target is determined based on the relative pose and the initial point cloud.

2. The method according to claim 1, characterized in that The step of determining the target pose of the first target according to the relative pose and the initial point cloud comprises: According to the relative position and posture, transform the initial point cloud to obtain a transformed target point cloud of the first target; Determine a target pose of the first target according to the target point cloud.

3. The method according to claim 1 or 2, characterized in that: The acquiring includes an RGB image of the first target and a depth image corresponding to the RGB image, including: Acquire at least two image groups including a first target, wherein one of the image groups corresponds to a camera view, and one of the image groups includes an RGB image and a depth image corresponding to the RGB image; A group of images is obtained from the at least two groups of images.

4. The method according to claim 3, characterized in that The obtaining of the initial point cloud of the first target includes: An initial point cloud of the first target is determined according to the depth images in the at least two image groups.

5. The method according to any one of claims 1 to 4, characterized in that: The step of determining a first point cloud of the first target according to the RGB image and the depth image includes: Determine a position of the first target in the depth image according to the RGB image and the depth image; A first point cloud of the first target is determined according to a camera intrinsic parameter and a position of the first target in the depth image.

6. The method according to any one of claims 1 to 5, characterized in that: The obtaining, according to the RGB features and the point cloud features, a completed point cloud of the first target includes: The RGB feature and the point cloud feature are combined to obtain a first feature; A completed point cloud of the first target is obtained according to the first feature and the RGB feature.

7. The method according to any one of claims 1 to 6, characterized in that: The determining the relative pose between the completed point cloud and the initial point cloud comprises: Acquire at least one first key point in the completed point cloud; For each first key point, within a range with the first key point as the center and a preset length as the radius, a second point cloud corresponding to the first key point is obtained; the second point cloud includes RGB information and position information of each point in the second point cloud; For each of the second point clouds, determine a third point cloud corresponding to the second point cloud in the initial point cloud, and determine a relative pose between the second point cloud and the third point cloud; the third point cloud includes RGB information and position information of each point in the third point cloud; Determine a registration error of the second point cloud according to the RGB information and position information of each point in the second point cloud and the RGB information and position information of each point in the third point cloud; If the registration error of each of the second point clouds is less than or equal to the first preset error, the relative pose between the completed point cloud and the initial point cloud is determined according to the relative pose between each of the second point clouds and the third point cloud.

8. The method according to claim 7, characterized in that The second point cloud includes at least one second key point; each of the second key points corresponds to a third key point in the third point cloud; and determining the registration error of the second point cloud according to the RGB information and position information of each point in the second point cloud and the RGB information and position information of each point in the third point cloud includes: For each of the second key points, Determine a distance difference between the second key point and the third key point according to the position information of the second key point and the corresponding position information of the third key point; Determine an RGB difference between the second key point and the third key point according to the RGB information of the second key point and the corresponding RGB information of the third key point; Determining a registration error of the second key point according to the distance difference and the RGB difference; A registration error of the second point cloud is determined according to the registration error of each of the second key points.

9. The method according to claim 8, characterized in that The RGB information of each point in the second point cloud and the third point cloud includes color values ​​corresponding to red, green, and blue respectively; and determining the RGB difference between the second key point and the third key point according to the RGB information of the second key point and the corresponding RGB information of the third key point includes: Respectively determine the red difference, green difference, and blue difference between the red, green, and blue color values ​​in the RGB information of the second key point and the corresponding red, green, and blue color values ​​in the RGB information of the third key point; Obtain a sum of the red difference, the green difference, and the blue difference after square operations are respectively performed, wherein the sum represents an RGB difference between the second key point and the corresponding third key point.

10. The method according to any one of claims 7 to 9, characterized in that: The method further comprises: If there is a second point cloud with a registration error greater than the first preset error, adjusting the position of the first key point corresponding to the second point cloud according to the registration error; The registration error of the new second point cloud corresponding to the first key point after the position adjustment is obtained again, until the registration error of the new second point cloud is less than or equal to the first preset error.

11. A posture determination device, characterized in that: include: An information acquisition module, used to acquire an RGB image of a first target and a depth image corresponding to the RGB image, and to acquire an initial point cloud of the first target; A point cloud acquisition module, used to determine a first point cloud of the first target according to the RGB image and the depth image; A point cloud completion module, used to obtain RGB features of the RGB image and point cloud features of the first point cloud, and obtain a completed point cloud of the first target according to the RGB features and the point cloud features; The pose acquisition module is used to determine the relative pose between the completed point cloud and the initial point cloud, and determine the target pose of the first target according to the relative pose and the initial point cloud.

12. The device according to claim 11, characterized in that The posture acquisition module is further used to transform the initial point cloud according to the relative posture to obtain a transformed target point cloud of the first target; and determine the target posture of the first target according to the target point cloud.

13. The device according to claim 11 or 12, characterized in that The information acquisition module is further used to acquire at least two image groups including the first target, one of the image groups corresponds to a camera perspective, and one of the image groups includes an RGB image and a depth image corresponding to the RGB image; A group of images is obtained from the at least two groups of images.

14. The device according to claim 13, characterized in that The information acquisition module is further used to determine the initial point cloud of the first target based on the depth images in the at least two image groups.

15. The device according to any one of claims 11 to 14, characterized in that: The point cloud acquisition module is also used to determine the position of the first target in the depth image based on the RGB image and the depth image; and determine the first point cloud of the first target based on the camera intrinsic parameters and the position of the first target in the depth image.

16. The device according to any one of claims 11 to 15, characterized in that: The point cloud completion module is also used to fuse the RGB feature and the point cloud feature to obtain a first feature; and obtain a completed point cloud of the first target based on the first feature and the RGB feature.

17. The device according to any one of claims 11 to 16, characterized in that: The posture acquisition module is further used to acquire at least one first key point in the completed point cloud; for each first key point, within a range with the first key point as the center and a preset length as the radius, acquire a second point cloud corresponding to the first key point; the second point cloud includes RGB information and position information of each point in the second point cloud; for each second point cloud, determine a third point cloud corresponding to the second point cloud in the initial point cloud, and determine the relative posture between the second point cloud and the third point cloud; the third point cloud includes RGB information and position information of each point in the third point cloud; According to the second RGB information and position information of each point in the point cloud, and RGB information and position information of each point in the third point cloud, to determine the registration error of the second point cloud; If the registration error of each of the second point clouds is less than or equal to the first preset error, the relative pose between the completed point cloud and the initial point cloud is determined according to the relative pose between each of the second point clouds and the third point cloud.

18. The device according to claim 17, characterized in that The second point cloud includes at least one second key point; each of the second key points corresponds to a third key point in the third point cloud; The posture acquisition module is further used to determine, for each of the second key points, a distance difference between the second key point and the third key point according to the position information of the second key point and the position information of the corresponding third key point; Determine an RGB difference between the second key point and the third key point according to the RGB information of the second key point and the corresponding RGB information of the third key point; Determine the registration error of the second key point according to the distance difference and the RGB difference; and determine the registration error of the second point cloud according to the registration error of each of the second key points.

19. The device according to claim 18, characterized in that The RGB information of each point in the second point cloud and the third point cloud includes color values ​​corresponding to red, green and blue respectively; The posture acquisition module is further used to respectively determine the red difference, green difference, and blue difference between the red, green, and blue color values ​​in the RGB information of the second key point and the corresponding red, green, and blue color values ​​in the RGB information of the third key point; Obtain a sum of the red difference, the green difference, and the blue difference after square operations are respectively performed, wherein the sum represents an RGB difference between the second key point and the corresponding third key point.

20. The device according to any one of claims 17 to 19, characterized in that The posture acquisition module is also used to adjust the position of the first key point corresponding to the second point cloud according to the registration error if there is a second point cloud with a registration error greater than the first preset error; and again obtain the registration error of the new second point cloud corresponding to the first key point after the adjusted position until the registration error of the new second point cloud is less than or equal to the first preset error.

21. An electronic device, characterized in that: It includes a memory and one or more processors; the memory is coupled to the processor; the memory stores computer program code, and the computer program code includes computer instructions. When the computer instructions are executed by the processor, the electronic device executes the posture determination method as described in any one of claims 1-10.

Images