Pose detection method, pose detection device, electronic equipment and storage medium

Abstract

The invention relates to a pose detection method, a pose detection device, electronic equipment and a storage medium. The pose detection method comprises: in response to an acquired to-be-processed image and a depth image corresponding to the to-be-processed image, determining a target image corresponding to a target object in the to-be-processed image, and determining first depth point cloud data corresponding to the target object in the depth image, the first depth point cloud data comprising a plurality of first sampling points; sampling the first depth point cloud data to obtain second depth point cloud data, the second depth point cloud data comprising a plurality of first sampling point sets, and each depth point cloud data set in the plurality of first sampling point sets comprising a plurality of first sampling points; and determining a pose detection result of the target object according to the target image and the second depth point cloud data. According to the invention, the reasoning time and operation power consumption of the pose detection process are reduced, and the detection network used for pose detection stably operates at the mobile terminal.

Description

Technical Field

[0001] The present disclosure relates to the field of posture estimation of rigid bodies, and in particular to a posture detection method, a posture detection device, an electronic device, and a storage medium. Background Art

[0002] Object pose estimation is a key task in the field of computer vision. Its main goal is to accurately obtain a 6-degree-of-freedom representation of the object's pose (including 3-degree-of-freedom rotation and 3-degree-of-freedom translation) in real scenes to provide the object's precise spatial position. Therefore, it is becoming increasingly important in computer vision-related tasks such as virtual reality, augmented reality, autonomous driving, and robotic operation. Summary of the Invention

[0003] In order to overcome the problems existing in the related art, the present disclosure provides a posture detection method, a posture detection device, an electronic device and a storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, a posture detection method is provided, characterized in that it includes: in response to acquiring an image to be processed and a depth image corresponding to the image to be processed, determining a target image corresponding to a target object in the image to be processed, and determining first depth point cloud data corresponding to the target object in the depth image, wherein the first depth point cloud data includes a plurality of first sampling points, and the plurality of first sampling points correspond to depth values ​​and pixel coordinates of pixel points of the target object image in the depth image; sampling the first depth point cloud data to obtain second depth point cloud data, wherein the second depth point cloud data includes a plurality of first sampling point sets, and each depth point cloud data set in the plurality of first sampling point sets includes a plurality of first sampling points; and determining a posture detection result of the target object according to the target image and the second depth point cloud data.

[0005] In one embodiment, determining the pose detection result of the target object based on the target image and the second depth point cloud data includes: inputting the target image and the second depth point cloud data into a preset model to obtain first three-dimensional point cloud data corresponding to the target object, the preset model being used to output three-dimensional point cloud data corresponding to the target object based on the input target object image and the corresponding depth point cloud data; determining the pose detection result of the target object based on the first three-dimensional point cloud data and preset template data, the pose detection result being a rotation and translation RT matrix, the rotation and translation RT matrix representing the three-degree-of-freedom rotation data and three-degree-of-freedom translation data of the target object.

[0006] In one embodiment, after acquiring the target image, the method further includes: acquiring an image mask corresponding to the target image; determining the first depth point cloud data corresponding to the target object in the depth image includes: determining the first depth point cloud data corresponding to the target object in the depth image based on an overlapping relationship between the image mask and the depth image.

[0007] In one embodiment, the first three-dimensional point cloud data includes multiple groups of second sampling point sets, and each second sampling point set in the multiple second sampling point sets corresponds one-to-one to each first sampling point set in the multiple first sampling point sets; determining the pose detection result of the target object based on the first three-dimensional point cloud data and the preset template data includes: determining the confidence of each second sampling point set in the first three-dimensional point cloud data, wherein the confidence represents the degree of fit between the second sampling point set and the target object; and performing the second sampling point set in the first three-dimensional point cloud data whose confidence is greater than the confidence threshold. The three-dimensional point cloud data is combined and determined to be the second three-dimensional point cloud data; according to the second quantity, the second three-dimensional point cloud data is clustered to obtain a preset number of three-dimensional point cloud pairs; a preset registration method is adopted to align the preset number of three-dimensional point cloud pairs according to the template data to obtain a translation and rotation RT matrix representing the translation and rotation relationship between the template data and the preset number of three-dimensional point cloud pairs, and the translation and rotation RT matrix is ​​determined as the posture detection result, wherein the template data is pre-acquired data corresponding to the initial posture of the target object, and the template data includes a preset number of three-dimensional point cloud pairs.

[0008] In one embodiment, the preset model is trained based on pre-collected training data, and the training data includes data generated by a modeling tool based on a target object and simulation data generated by a simulation tool.

[0009] In one embodiment, the data generated by the modeling tool based on the target object includes training data corresponding to the symmetrical object, and the training data corresponding to the symmetrical object is obtained in the following manner: determining the symmetry angle range of the symmetrical object relative to the positive direction of the camera, and determining the initial posture of the symmetrical object; within the symmetry angle range, rotating the symmetrical object multiple times around the symmetry axis in the positive direction of the camera, and obtaining a depth image and a three-primary color RGB image of the symmetrical object during each rotation process, and determining the depth image and the three-primary color RGB image as the training data.

[0010] In one embodiment, determining the target image corresponding to the target object in the image to be processed includes: obtaining the target image corresponding to the target object in the image to be processed through a target segmentation network, and the target segmentation network is trained based on the three primary color RGB image and depth image of the target object at different angles.

[0011] In one embodiment, sampling the first depth point cloud data to obtain second depth point cloud data includes: sampling the first depth point cloud data according to a first quantity through a preset sampling method to obtain second depth point cloud data, wherein the second depth point cloud data includes multiple first sampling point sets, and each depth point cloud data set in the multiple first sampling point sets includes a first number of first sampling points.

[0012] According to a second aspect of an embodiment of the present disclosure, a posture detection device is provided, characterized in that it includes: a determination unit, configured to determine, in response to acquiring an image to be processed and a depth image corresponding to the image to be processed, a target image corresponding to a target object in the image to be processed, and determine first depth point cloud data corresponding to the target object in the depth image, wherein the first depth point cloud data includes a plurality of first sampling points, and the plurality of first sampling points correspond to depth values ​​and pixel coordinates of pixel points of the target object image in the depth image; a sampling unit, configured to sample the first depth point cloud data to obtain second depth point cloud data, wherein the second depth point cloud data includes a plurality of first sampling point sets, and each depth point cloud data set in the plurality of first sampling point sets includes a plurality of first sampling points; and a processing unit, configured to determine a posture detection result of the target object based on the target image and the second depth point cloud data.

[0013] In one embodiment, the processing unit determines the pose detection result of the target object according to the target image and the second depth point cloud data in the following manner:

[0014] The target image and the second depth point cloud data are input into a preset model to obtain first three-dimensional point cloud data corresponding to the target object. The preset model is used to output three-dimensional point cloud data corresponding to the target object based on the input target object image and the corresponding depth point cloud data. The posture detection result of the target object is determined based on the first three-dimensional point cloud data and the preset template data. The posture detection result is a rotation and translation RT matrix, and the rotation and translation RT matrix represents the three-degree-of-freedom rotation data and three-degree-of-freedom translation data of the target object.

[0015] In one embodiment, after acquiring the target image, the device further includes: acquiring an image mask corresponding to the target image; the determination unit determines the first depth point cloud data corresponding to the target object in the depth image in the following manner: determining the first depth point cloud data corresponding to the target object in the depth image based on the overlapping relationship between the image mask and the depth image.

[0016] In one embodiment, the first three-dimensional point cloud data includes a plurality of second sampling point sets, and each second sampling point set in the plurality of second sampling point sets corresponds one-to-one to each first sampling point set in the plurality of first sampling point sets; the processing unit determines the pose detection result of the target object according to the first three-dimensional point cloud data and the preset template data in the following manner: determining the confidence of each second sampling point set in the first three-dimensional point cloud data, wherein the confidence represents the degree of fit between the second sampling point set and the target object; and processing the second sampling point set in the first three-dimensional point cloud data having a confidence greater than a confidence threshold. A set of sampling points is determined as second three-dimensional point cloud data; according to a second quantity, the second three-dimensional point cloud data is clustered to obtain a preset number of three-dimensional point cloud pairs; a preset alignment method is adopted to align the preset number of three-dimensional point cloud pairs according to the template data to obtain a translation and rotation RT matrix representing the translation and rotation relationship between the template data and the preset number of three-dimensional point cloud pairs, and the translation and rotation RT matrix is ​​determined as the posture detection result, wherein the template data is pre-acquired data corresponding to the initial posture of the target object, and the template data includes a preset number of three-dimensional point cloud pairs.

[0017] In one embodiment, the preset model is trained by the processing unit based on pre-collected training data, and the training data includes data generated by a modeling tool based on a target object and simulation data generated by a simulation tool.

[0018] In one embodiment, the data generated by the modeling tool based on the target object includes training data corresponding to the symmetrical object, and the training data corresponding to the symmetrical object is obtained by the processing unit in the following manner: determining the symmetry angle range of the symmetrical object relative to the positive direction of the camera, and determining the initial posture of the symmetrical object; within the symmetry angle range, rotating the symmetrical object around the symmetry axis in the positive direction of the camera multiple times, and obtaining a depth image and a three-primary color RGB image of the symmetrical object during each rotation process, and determining the depth image and the three-primary color RGB image as the training data.

[0019] In one embodiment, the determination unit determines the target image corresponding to the target object in the image to be processed in the following manner, including: obtaining the target image corresponding to the target object in the image to be processed through a target segmentation network, wherein the target segmentation network is trained based on the three primary color RGB image and depth image of the target object at different angles.

[0020] In one embodiment, the sampling unit samples the first depth point cloud data in the following manner to obtain second depth point cloud data: using a preset sampling method, the first depth point cloud data is sampled according to a first quantity to obtain second depth point cloud data, wherein the second depth point cloud data includes multiple first sampling point sets, and each depth point cloud data set in the multiple first sampling point sets includes a first number of first sampling points.

[0021] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to: execute the posture detection method described in the first aspect or any one of the embodiments of the first aspect.

[0022] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided, in which instructions are stored. When the instructions in the storage medium are executed by a processor, the processor is enabled to execute the posture detection method described in the first aspect or any one of the embodiments of the first aspect.

[0023] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: after acquiring an image to be processed and a depth image corresponding to the image to be processed, a target image corresponding to the target object is determined in the image to be processed, and first depth point cloud data corresponding to the target object is determined in the depth image. The depth point cloud data is sampled according to a first quantity to obtain second depth point cloud data, and the second depth point cloud data includes a plurality of first sampling point sets. Based on the acquired target image and the second depth point cloud data, the pose detection result of the target object is determined. Through the present disclosure, the first depth point cloud data is sampled in advance, and the independently distributed sampling points in the first depth point cloud data are sampled into a first sampling point set containing a plurality of sampling points, and second sampling data containing a plurality of first sampling point sets is obtained, thereby reducing the amount of computational complexity of subsequent computational steps using the second sampling data, thereby reducing the inference time and operating power consumption of the pose detection process, and allowing the detection network for pose detection to operate stably on the mobile terminal.

[0024] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0026] Figure 1 It is a block diagram of a posture detection method according to an exemplary embodiment of the present disclosure.

[0027] Figure 2 It is a block diagram of a posture detection method according to an exemplary embodiment of the present disclosure.

[0028] Figure 3 It is a block diagram of a posture detection method according to an exemplary embodiment of the present disclosure.

[0029] Figure 4 It is a block diagram of a posture detection method according to an exemplary embodiment of the present disclosure.

[0030] Figure 5 It is a block diagram of a posture detection method according to an exemplary embodiment of the present disclosure.

[0031] Figure 6 It is a block diagram of a posture detection method according to an exemplary embodiment of the present disclosure.

[0032] Figure 7 It is a block diagram of a posture detection method according to an exemplary embodiment of the present disclosure.

[0033] Figure 8 The figure is a schematic diagram of an application scenario of a posture detection method according to an exemplary embodiment of the present disclosure.

[0034] Figure 9 The figure is a flowchart of a posture detection method according to an exemplary embodiment.

[0035] Figure 10 The present invention is a flowchart of a method for determining a pose detection result of a target object according to an exemplary embodiment.

[0036] Figure 11 It is a block diagram of a posture detection method according to an exemplary embodiment of the present disclosure.

[0037] Figure 12 The flowchart of a method for determining first depth point cloud data is shown according to an exemplary embodiment.

[0038] Figure 13 The present invention is a flowchart of a method for determining a pose detection result of a target object according to an exemplary embodiment.

[0039] Figure 14The figure is a flowchart of a posture detection method according to an exemplary embodiment of the present disclosure.

[0040] Figure 15 The figure is a flowchart showing a method for acquiring training data corresponding to a symmetrical object according to an exemplary embodiment.

[0041] Figure 16 It is a block diagram showing a method for obtaining training data according to an exemplary embodiment of the present disclosure.

[0042] Figure 17 The present invention is a flowchart showing a method for determining a target image corresponding to a target object in an image to be processed according to an exemplary embodiment.

[0043] Figure 18 The figure is a flowchart of a method for obtaining second depth point cloud data according to an exemplary embodiment.

[0044] Figure 19 The figure is a flowchart of a posture detection method according to an exemplary embodiment of the present disclosure.

[0045] Figure 20 It is a block diagram of a posture detection device according to an exemplary embodiment.

[0046] Figure 21 The figure is a block diagram of a device for posture detection according to an exemplary embodiment. DETAILED DESCRIPTION

[0047] Exemplary embodiments are described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numerals in different drawings represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present disclosure.

[0048] The posture detection method provided by the embodiments of the present disclosure is applied to scenarios where the posture information of a target object in an image to be processed is determined. The posture information of the target object is also referred to as the posture information of the target object. The posture information can be understood as the position and / or posture of the target object.

[0049] Object pose estimation is a key task in computer vision. Its primary goal is to accurately obtain a 6-DOF representation (including 3-DOF rotation and 3-DOF translation) that represents the pose of an object in a real-world scene. Object pose estimation provides precise spatial position and pose information, and is increasingly important in computer vision-related tasks such as virtual reality, augmented reality, autonomous driving, and robotic manipulation.

[0050] In the related art, methods for pose detection (or pose estimation methods) include deep learning-based methods and traditional methods not based on deep learning. Among them, traditional methods not based on deep learning directly extract the point and line features of the target object from the input three primary color RGB image or three-dimensional point cloud, match the extracted point and line features with the preset reference image or CAD three-dimensional model, and output the object pose by measuring the point and line coordinates in the two spatial coordinate systems as constraints. However, traditional methods rely on manually designed feature extraction, and their performance may be affected by image noise, data quality or other factors.

[0051] In related technologies, the development of deep learning has significantly improved pose estimation methods in terms of accuracy, robustness, and adaptability to different scenarios. Depending on whether a 3D CAD model of the target object is provided, deep learning-based pose detection methods (i.e., pose estimation algorithms) can be divided into two types: category-level and instance-level. Category-level pose detection methods do not require a 3D CAD model of the target object, while instance-level pose detection methods do require a 3D CAD model of the target object.

[0052] In related technologies, instance-level pose estimation methods can be divided into three categories: regression-based methods, template-based methods, and feature-based methods:

[0053] 1. Regression-based method. This method predicts the pose directly from the input image without the need for intermediate key point representation. Figure 1 As shown in the box for the pose detection method, the regression-based approach proceeds as follows: acquiring a target image, performing object detection on the target image, and obtaining a region of interest (ROI) containing the target object in the image. The ROI is processed by a pose estimation network, and target regression is performed to obtain a rotation and translation matrix representing the target object's pose. However, due to the limitations of the training data for the pose estimation network, regression-based pose estimation methods are typically only able to handle scenarios and conditions similar to those used in the training data. When encountering new, unseen scenarios or conditions, the model's performance may significantly degrade. This limits the versatility and flexibility of this approach in practical applications. Furthermore, regression-based pose estimation methods suffer from high computational complexity and place high demands on device performance. Furthermore, the training process of the pose estimation network used in regression-based methods does not include simulation of special environments (such as occlusion and reflections). Consequently, estimation results for targets in these environments may be inaccurate. Consequently, regression-based pose estimation methods suffer from limited versatility, high device performance requirements, and a lack of robustness.

[0054] 2. Template-based method. This method finds the most similar template of the target in the input image based on the provided real CAD model, and then performs 6D pose estimation. Figure 2As shown in the block diagram of the pose detection method, the template-based method process is as follows: obtain the image to be processed, and provide a template through a preset CAD model. The template is the CAD model of the target object. The template CAD model and the image to be processed are processed by the pose estimation network to obtain the rotation and translation matrix that characterizes the pose of the target object. The template-based pose estimation method usually requires a large number of templates to perform feature matching and comparison with the input image, and there are also problems of high computational complexity and high performance requirements for the equipment. In special scenarios (such as when the target object is occluded), the occlusion will cause some template features to be unable to match, thereby affecting the accuracy of the pose estimation. Therefore, the template-based pose estimation method has the problems of high performance requirements for equipment and lack of robustness.

[0055] 3. Feature-based method. This method extracts features from the input image (i.e., the image to be processed), matches it with the corresponding features in the existing 3DCAD model, and then uses the PnP (Perspective-n-Point) algorithm to establish a correspondence between three-dimensional (3D) and two-dimensional (2D) coordinates. Among them, the PnP algorithm is a method for solving three-dimensional to two-dimensional points, which is used to estimate the camera's pose when the 3D space point and its position are known. This feature-based method establishes a 2D-3D relationship, and compared with the above two methods, it can accurately estimate the 6-degree-of-freedom position of the target object. Figure 3 The block diagram of the pose detection method is shown in FIG. The process of the feature-based method is as follows: obtain the image to be processed. Process the image to be processed through the pose estimation network to obtain the image of the target point cloud, process the image containing the target point cloud, and obtain the processing result. The feature-based pose estimation method usually requires a large amount of feature extraction, matching and optimization calculations, which leads to a high computational complexity of the method and high requirements for device performance. Moreover, the feature-based pose estimation method is aimed at special scenes (there are changes in lighting or occlusions on the surface of the object), and the features of the target object in the image may also be affected, thereby reducing the accuracy of the pose estimation. Therefore, the feature-based pose estimation method has the problems of high requirements for device performance and lack of robustness.

[0056] In summary, instance-level pose estimation methods generally have the problems of high computational complexity and poor processing effect for special scenarios, which leads to poor robustness of instance-level pose estimation methods and inability to run stably in portable mobile devices.

[0057] In one example, the instance-level pose estimation method is as follows Figure 4As shown in the block diagram of the pose detection method, after obtaining the RGB image input (i.e., the image to be processed) and the depth point cloud input (i.e., the depth image), the image features of the RGB image input are extracted step by step through the 2D feature extraction network, and the depth point cloud is converted into a 3D point cloud. The 3D point cloud features are extracted step by step through the 3D point cloud network. While extracting the image features and 3D point cloud features step by step, the image features and 3D point cloud features are fused step by step, and the output result containing the RGB image features and the depth image features is input. This method uses the feature pre-fusion method to fuse the features of the target object in the image to be processed and the corresponding depth image. That is, while performing multi-level downsampling on the image to be processed, the depth image is also subjected to a process equivalent to "multi-level downsampling". While downsampling each layer, the corresponding features of the image to be processed and the corresponding features of the depth image are fused. The layer-by-layer fusion of features at multiple levels results in a large amount of processing similar to "downsampling" of the depth image based on the nearest neighbor classification algorithm (K-Nearest Neighbor, KNN) during the feature fusion process, which increases the system time consumption and required power consumption, resulting in the inability of this posture detection method to run stably on portable mobile terminals.

[0058] In another example, the instance-level pose estimation method uses specific tools (ObjectDatasetTools) to generate the pose of the target object to obtain training data. However, the specific tools (ObjectDatasetTools) cannot simulate situations such as occlusion and reflection. Therefore, the related technology lacks solutions for target objects with occlusion and reflection in the image to be processed, and the pose information of the target object cannot be determined.

[0059] Furthermore, some instance-level pose estimation methods need to obtain the 3D point cloud information corresponding to the current pose of the target object through the pose estimation network to extract key point information, and then post-process the extracted key point information through a preset algorithm to complete the pose estimation. However, due to the influence of the preset algorithm, the pose estimation result is inaccurate. In one example, Figure 5 The block diagram of the pose detection method is shown in FIG. This method obtains the image input to be processed (RGB input, i.e., the image to be processed) and the corresponding depth map input (Depth input, i.e., the depth image) from the sensor input, and samples the depth image. The depth image after sampling is input into the pose estimation network, and the processed image is input into the pose estimation network after pre-processing operations such as sampling to generate a 3D point cloud and obtain the predicted key point information. The key point information is post-processed using the least squares fitting method to obtain the rotation and translation RT matrix that represents the pose information of the target object. Given that the related art uses the least squares fitting method for key point post-processing, this method has the problem of inaccurate pose prediction and fluctuating distances.

[0060] In related technologies, the goal of category-level pose estimation methods is to estimate the pose of objects of the same category without obtaining a known CAD model in advance. Category-level pose estimation algorithms include regression-based methods and prior-based methods:

[0061] 1. Regression-based method. This method generally introduces the Normalized Object Coordinate Space (NOCS) to directly predict the object's position and posture. Figure 6 The block diagram of the pose detection method in

[15] is shown in the figure. The process of the regression-based method can be simplified as shown in the figure below: obtain the image to be processed and obtain the depth image corresponding to the image to be processed. The image to be processed and the depth image are processed by the pose estimation network, and the normalized object coordinate space is introduced for regression processing to obtain the rotation and translation RT matrix that represents the pose of the target object. This regression-based pose method introduces the normalized object coordinate space, which increases the complexity of the calculation. During the model training and inference process, the input data needs to be normalized and inversely transformed when necessary. This increases computing time and resource consumption, and places high demands on device performance. In addition, in special scenarios (with occlusion or complex scenes) where some features are lost, the normalized object coordinate space cannot achieve accurate pose estimation. Therefore, the regression-based pose estimation method has the problems of high requirements on device performance and lack of robustness.

[0062] 2. Prior-based method. This method can obtain more accurate estimates by combining the prior knowledge learned in the provided examples. Figure 7 Block diagram of the pose detection method in [1]. The prior-based method is as follows: obtain the image to be processed and a preset mask corresponding to the image to be processed. The image to be processed and the preset mask are processed by the pose estimation network. The preset point cloud data of the target object is introduced to perform pose estimation on the processing results of the pose estimation network, and a rotation and translation RT matrix representing the pose of the target object is obtained. Prior-based pose estimation methods cannot be effectively applied in unseen environments. Prior knowledge cannot cover all possible scenarios and situations, resulting in reduced pose estimation performance in new environments. Prior-based pose estimation methods require complex optimization algorithms or inference processes to utilize prior knowledge, have high computational complexity, and have high requirements for device performance. In addition, for special scenarios (where there is occlusion or light reflection), the effectiveness of the prior knowledge is reduced, affecting the accuracy and stability of pose estimation. Therefore, prior-based pose estimation methods have the problems of limited versatility, high requirements for device performance, and lack of robustness.

[0063] In summary, category-level pose estimation methods also generally have the problems of high computational complexity and poor processing effect for special scenarios, which leads to poor robustness of instance-level pose estimation methods and inability to run stably in portable mobile devices.

[0064] In summary, the pose detection methods based on deep learning in the related arts (including instance-level pose estimation methods and category-level pose estimation methods) generally have high requirements for device performance and lack of robustness, resulting in the inability of deep learning pose estimation methods to run stably in portable mobile terminals, and for special scenarios (where there is occlusion, light reflection, or the target object is a symmetrical object), the accuracy of pose estimation cannot be guaranteed. Furthermore, the pose detection methods based on deep learning in some related arts use the least squares fitting method for post-processing key points, which has the problem of inaccurate predicted poses and fluctuating distances. Furthermore, the pose detection network implemented by the pose detection methods based on deep learning in some related arts has the problem of a single source of training data, resulting in the poor performance of the related arts in cross-scene environments, making it difficult to generalize to different scenarios and having poor versatility.

[0065] In view of this, the present disclosure proposes a posture detection method, which, after obtaining an image to be processed containing an image of a target object and a depth image corresponding to the image to be processed, determines the target image corresponding to the target object in the image to be processed, and determines the depth point cloud data corresponding to the target object in the depth image. The depth point cloud data is sampled according to a preset number to obtain sampled depth point cloud data, which includes a plurality of depth sampling point sets. Based on the target image and the sampled depth point cloud data input, three-dimensional point cloud data corresponding to the target object is obtained. The posture detection result of the target object is determined based on the three-dimensional point cloud data and the preset template data. Through the present disclosure, the inference time and running power consumption of the posture detection process are reduced, so that the detection network for posture detection can run stably on the mobile terminal.

[0066] In an exemplary embodiment of the present disclosure, Figure 8 As shown in the schematic diagram of the application scenario of the pose detection method, the pose detection method disclosed herein can be used in the following scenarios: A camera is used to acquire data about an object to be estimated. The terminal processes the data collected by the camera, i.e., extracts the target object from the acquired data, and estimates the 6-degree-of-freedom (DoF) pose of the target object based on the extracted target data.

[0067] Figure 9 FIG. 1 is a flow chart of a posture detection method according to an exemplary embodiment. Figure 9 As shown, the method includes steps S101 to S103.

[0068] In step S101, in response to obtaining an image to be processed and a depth image corresponding to the image to be processed, a target image corresponding to a target object in the image to be processed is determined, and first depth point cloud data corresponding to the target object in the depth image is determined, the first depth point cloud data including a plurality of first sampling points, and the plurality of first sampling points correspond to depth values ​​and pixel coordinates of pixel points of the target object image in the depth image.

[0069] In step S102, the first depth point cloud data is sampled to obtain second depth point cloud data, where the second depth point cloud data includes a plurality of first sampling point sets, and each depth point cloud data set in the plurality of first sampling point sets includes a plurality of first sampling points.

[0070] In step S103 , a pose detection result of the target object is determined based on the target image and the second depth point cloud data.

[0071] In an embodiment of the present disclosure, an RGB image (image to be processed) and a corresponding depth image containing an image of a target object are synchronously collected by an image acquisition device, and a target image and first depth point cloud data of the target object are extracted from the image to be processed and the depth image respectively. The first depth point cloud data points to the pixel points corresponding to the target object in the depth image, and contains pixel coordinate data and depth data corresponding to the pixel points of the target object. The present disclosure performs sampling processing on the depth point cloud data based on a first quantity, that is, for all pixels in the depth image corresponding to the target object, a preset number of pixel points corresponding to the target object (corresponding to the first sampling point) are classified into a sampling point set (first sampling set), and all the first sampling point sets obtained are determined as second depth point cloud data. The process of obtaining the second depth point cloud data can be regarded as a "downsampling" process of the first depth point cloud data.

[0072] It is understandable that after obtaining the first depth point cloud data and the target image, the present disclosure needs to perform feature fusion of the first depth point cloud data and the target image in the subsequent stage. The present disclosure performs a process similar to downsampling on the first depth point cloud data before performing feature fusion to obtain the second depth point cloud data, so that in the subsequent feature fusion stage, the second depth point cloud data is directly subjected to feature fusion with the target image after the downsampling, thereby avoiding the large amount of computation caused by step-by-step feature fusion, and solving the problems of excessive redundancy of network structure and algorithm flow in related technologies, slow deployment speed on mobile terminals, and high power consumption. The amount of computation in the posture detection process is reduced, and the performance requirements of the posture detection network for implementing the posture detection method are reduced, so that the detection network for posture detection can operate stably on portable mobile terminals.

[0073] In one example, after obtaining first depth point cloud data, the present disclosure uses a K-Nearest Neighbor (KNN) algorithm to perform multi-layer sampling on the first depth point cloud data based on a first number N to obtain M layers of point cloud images [N1, N2, N3, ..., NM] (i.e., second depth point cloud data). Where M is the number of layers, N is the pre-defined number of sampling points in each layer (i.e., the first number), and N1, N2, N3, ..., NM correspond to multiple first sampling point sets in the second depth point cloud data.

[0074] In an embodiment of the present disclosure, a target image and a second depth point cloud data are processed by a preset model to obtain first three-dimensional point cloud data corresponding to the target object. It is understandable that the first three-dimensional point cloud data is a set of three-dimensional coordinates that fit the surface of the target object. By comparing the first three-dimensional point cloud data with the preset template data, the current posture (i.e., current position) of the target object can be obtained. The following embodiments of the present disclosure illustrate a method for determining the position detection result of the target object.

[0075] Figure 10 FIG. 1 is a flow chart showing a method for determining a pose detection result of a target object according to an exemplary embodiment. Figure 10 As shown, the method includes steps S201 to S202.

[0076] In step S201 , the target image and the second depth point cloud data are input into a preset model to obtain first three-dimensional point cloud data corresponding to the target object.

[0077] Among them, the preset model is used to output three-dimensional point cloud data corresponding to the target object based on the input target object image and corresponding depth point cloud data.

[0078] In step S202, a pose detection result of the target object is determined based on the first three-dimensional point cloud data and the preset template data.

[0079] Among them, the pose detection result is the rotation and translation RT matrix, which represents the three-degree-of-freedom rotation data and three-degree-of-freedom translation data of the target object.

[0080] In the disclosed embodiment, after acquiring first depth point cloud data and a target image, a preset model is used to fuse the features of the first depth point cloud data and the target image to obtain three-dimensional point cloud data (i.e., first three-dimensional point cloud data) that has a fitting relationship with the target object. The pose detection result of the target object is determined based on the first three-dimensional point cloud data combined with pre-set template data.

[0081] It can be understood that the first three-dimensional point cloud data obtained through the preset model represents the current posture state of the target object, while the preset template data represents the initial posture of the target object. According to the first three-dimensional point cloud data and the preset template data, the ratio of the current posture of the target object to the initial posture can be achieved, thereby determining the posture detection result of the target object.

[0082] In an exemplary embodiment of the present disclosure, Figure 11 As shown in the block diagram of the pose detection method, after obtaining an RGB image input (i.e., the image to be processed) and a point cloud input (i.e., the depth image corresponding to the image to be processed), the present disclosure uses a 2D feature extraction network to obtain image features of the RGB image, and uses a deep point cloud network to obtain image features of the depth image. The two extracted depth images are fused, and the output result containing the RGB image features and the depth image features is input.

[0083] It is understood that the depth image in this disclosure corresponds to the image to be processed, so the acquisition of the first depth point cloud data in the depth image can be achieved based on the mask corresponding to the target image in the image to be processed. The following embodiments of this disclosure illustrate the method for determining the first depth point cloud data.

[0084] Figure 12 FIG. 1 is a flow chart showing a method for determining first depth point cloud data according to an exemplary embodiment. Figure 12 As shown, the method includes steps S301 to S302.

[0085] In step S301 , in response to obtaining an image to be processed and a depth image corresponding to the image to be processed, a target image corresponding to a target object in the image to be processed is determined, and an image mask corresponding to the target image is obtained.

[0086] In step S302 , first depth point cloud data corresponding to the target object in the depth image is determined based on the overlapping relationship between the image mask and the depth image.

[0087] In the disclosed embodiment, when a target image corresponding to a target object is determined in an image to be processed, an image mask corresponding to the target image is also obtained. Based on the correspondence between the image to be processed and the depth image, the image mask also overlaps with the image region corresponding to the target object in the depth image. Based on the overlapping relationship between the image mask and the depth image, pixel data (including pixel coordinates and depth values) of the overlapping image region in the depth image that overlaps with the image mask is obtained, and the pixel data of the overlapping image region is then used to determine the first depth point cloud data.

[0088] The following embodiments of the present disclosure illustrate a method for determining a pose detection result of a target object.

[0089] Figure 13 FIG. 1 is a flow chart showing a method for determining a pose detection result of a target object according to an exemplary embodiment. Figure 13 As shown, the method includes steps S401 to S404.

[0090] In step S401 , the confidence level of each second sampling point set in the first three-dimensional point cloud data is determined, where the confidence level represents the degree of fit between the second sampling point set and the target object.

[0091] In step S402 , a second set of sampling points in the first three-dimensional point cloud data whose confidence level is greater than a confidence threshold is determined as second three-dimensional point cloud data.

[0092] In step S403 , clustering processing is performed on the second three-dimensional point cloud data according to the second quantity to obtain a preset number of three-dimensional point cloud pairs.

[0093] In step S404, a preset registration method is used to align a preset number of three-dimensional point cloud pairs according to the template data to obtain a translation and rotation RT matrix that represents the translation and rotation relationship between the template data and the preset number of three-dimensional point cloud pairs, and the translation and rotation RT matrix is ​​determined as the posture detection result.

[0094] The template data is pre-acquired data corresponding to the initial posture of the target object, and the template data includes a preset number of three-dimensional point cloud pairs.

[0095] In the embodiment of the present disclosure, the acquired three-dimensional point cloud data (first three-dimensional point cloud data) corresponds to the second depth point cloud data. Therefore, the first three-dimensional point cloud data also includes multiple sampling point sets (second sampling point sets), and the multiple second sampling point sets included in the first three-dimensional point cloud data correspond one-to-one with the multiple first sampling point sets included in the second depth point cloud data. It is understandable that each of the multiple second sampling point sets in the first three-dimensional point cloud data has a different degree of fit with the target object, and only the second sampling point sets with a sufficiently high degree of fit can best represent the current position of the target object and are used to obtain the pose detection result of the target object. Therefore, after obtaining the first three-dimensional point cloud data corresponding to the target object, the present disclosure obtains the confidence corresponding to each second sampling point set. Based on the confidence corresponding to the second sampling point set and a preset confidence threshold (thre1), the first three-dimensional point cloud data is screened, and the second sampling point sets with a confidence greater than the threshold and capable of representing the current position of the target object are used as the second three-dimensional point cloud data to participate in the subsequent pose detection process. The present disclosure further screens the acquired first three-dimensional point cloud data to obtain second three-dimensional point cloud data that can accurately represent the current position of the target object, thereby ensuring the accuracy of the subsequently acquired posture detection results.

[0096] In an embodiment of the present disclosure, after acquiring the second three-dimensional point cloud data, clustering processing is performed on the second three-dimensional point cloud data according to the second quantity, thereby obtaining less data representing the current position of the target object, reducing the computational complexity of the subsequent pose detection result acquisition process. While ensuring the accuracy of the pose detection results, the pose detection method reduces the device performance requirements, enabling the pose detection method of the present disclosure to operate stably in a portable mobile terminal. In one example, the present disclosure sets the second quantity to 9, performs meanshift clustering on the second three-dimensional point cloud data, and ultimately obtains 9 groups of three-dimensional point cloud pairs corresponding to the target object.

[0097] In an embodiment of the present disclosure, after obtaining a three-dimensional point cloud pair representing the current position of the target object, the pose detection result is determined based on the conversion relationship between the preset template three-dimensional point cloud pair (i.e., template data) representing the initial position of the object to be measured and the three-dimensional point cloud pair. That is, a preset registration method is used to align the template data and a preset number of three-dimensional point cloud pairs to obtain a translation and rotation RT matrix representing the translation and rotation relationship between the template data and the preset number of three-dimensional point cloud pairs, and the translation and rotation RT matrix is ​​determined as the pose detection result. In one example, the template data in the present disclosure is obtained in the following way: a preset number of three-dimensional point cloud pairs corresponding to the initial state of the target object are obtained using the farthest point sampling algorithm (FPS) through a CAD model, and the obtained preset number of three-dimensional point cloud pairs are determined as template data. In another example, the present disclosure uses the Umeyama algorithm to perform registration between the three-dimensional point cloud pair and the template data to obtain a rotation and translation RT matrix representing the pose detection result.

[0098] In an exemplary embodiment of the present disclosure, Figure 14 As shown in the flowchart of the posture detection method, the present disclosure adopts the following method to determine the posture detection result of the target object based on the acquisition of the first three-dimensional point cloud data. After acquiring the first three-dimensional point cloud data, the first three-dimensional point cloud data is input into the entry for performing threshold judgment, the confidence (segmentation confidence) of each second sampling point set in the first three-dimensional point cloud data is determined, and the confidence of each second sampling point set is threshold judged, that is, whether the segmentation confidence is greater than the threshold is judged. The second sampling point set with a confidence less than or equal to the threshold is deleted (that is, the point cloud is deleted). The second sampling point set with a confidence greater than the threshold is retained, and all second sampling point sets greater than the threshold are point cloud clustered to obtain a preset number of point cloud pairs. Alignment is performed through the Umeyama algorithm, that is, the Umeyama algorithm is used to perform alignment between a preset number of point cloud pairs and a preset number of template point cloud pairs, and the rotation and translation matrix obtained by alignment is used as the posture detection result and output (that is, output posture).

[0099] It is understood that the preset models in this disclosure are trained based on pre-generated and / or collected training data. To ensure that the pose detection method in this disclosure is universally applicable to various scenarios, the training data in this disclosure needs to be obtained through multiple channels. The following embodiments of this disclosure illustrate the preset models in this disclosure.

[0100] In one implementation of the disclosed embodiment, the preset model is trained based on pre-collected training data, and the training data includes data generated by a modeling tool based on a target object and simulation data generated by a simulation tool.

[0101] In an embodiment of the present disclosure, for various preset target objects, Neural Radiance Fields (Nerf) is used to generate a 3D CAD model, and the RGB image and depth image of the target object are obtained 360° around the target object, and training data for training the preset model is generated by projection, that is, the corresponding RGB image, depth image and corresponding three-dimensional point cloud data of the same object in different postures are used as training data. It can be understood that the preset model in the present disclosure uses the RGB image corresponding to the target object and the depth point cloud data of the sampled depth image (corresponding to the second depth point cloud data) as input in actual application. Therefore, in the process of training to obtain the preset model, the depth point cloud data of the depth image must be sampled and processed using a preset sampling method, and the depth point cloud data of the sampled depth image is input into the training model corresponding to the RGB image of the target object for model training. In one example, the present disclosure simulates object reflection scenes and occlusion scenes and generates training data.

[0102] It is understandable that since the generation of real data requires a lot of manpower and time costs, it is impossible to obtain training data of sufficient magnitude when using real data as training data, and it is impossible to meet the generalization requirements of the posture detection method. Therefore, the present disclosure adopts a simulation tool, and for different target objects, it does not rely directly on real objects, and only uses simulation tools to generate corresponding RGB images, depth images and corresponding three-dimensional point cloud data of the target objects in different postures (i.e., postures), and uses them as training data. In one example, the present disclosure adopts the simulation tool BlenderProc, and synthesizes posture and depth maps by spherical ring projection, and then increases the complexity of the data by adding RGB public data sets to improve the generalization ability of the model.

[0103] It is understandable that in order to ensure that the posture detection method in the present disclosure has good posture detection accuracy for some feature targets, the present disclosure needs to adopt a special training data acquisition method for special targets.

[0104] Figure 15FIG. 1 is a flow chart showing a method for obtaining training data corresponding to a symmetrical object according to an exemplary embodiment. Figure 15 As shown, the method includes steps S501 to S502.

[0105] In step S501 , the symmetric angle range of the symmetrical object relative to the positive direction of the camera is determined, and the initial position and posture of the symmetrical object is determined.

[0106] In step S502, within the symmetric angle range, the symmetrical object is rotated multiple times around the symmetry axis in the positive direction of the camera, and a depth image and a three-primary color RGB image of the symmetrical object are obtained during each rotation process, and the depth image and the three-primary color RGB image are determined as training data.

[0107] It can be understood that for symmetrical objects (for example, a symmetrically designed office mouse is generally 180° axially symmetrical), if the symmetrical object has a symmetry angle α around the axis of symmetry, and the rotation angle of the symmetrical object around the axis of symmetry is β, then when the rotation angle β of the symmetrical object around the axis of symmetry is within the scale angle range (0, α), the two-dimensional imaging of the symmetrical object and the actual posture of the symmetrical object will have a one-to-one correspondence. Therefore, the present disclosure sets the positive direction of the symmetrical object relative to the camera, denotes the symmetry angle of the symmetrical object as α, and makes the rotation of the positive direction of the object around the axis of symmetry, and determines the rotation angle of the positive direction of the symmetrical object around the axis of symmetry as β, converts the rotation angle β of the positive direction of the symmetrical object around the axis of symmetry to the symmetry angle range (0, α), that is, β∈(0, α), and obtains training data after completing the rotation angle conversion. In an example, the present disclosure converts β to (0, α) when β∈(0, 360°), and converts it through the following process,

[0108]

[0109] Among them, [R 3*3 |t 3*1 ] and [R 3*3 |t 3*1 ] are all RT matrices, [R 3*3 |t 3*1 ] represents the pose directly generated by the modeling tool for the symmetrical object, the RT matrix [R 、 |t 、 ] represents the posture after transformation, β is the actual rotation angle of the symmetrical object, and α is the angle after transformation.

[0110] In an exemplary embodiment of the present disclosure, Figure 16As shown in the block diagram of the method for obtaining training data, the present disclosure adopts the following methods to obtain training data during the model training process: generating real training data through the neural radiation field Nerf, generating simulated training data through the simulation tool Blender, generating symmetrical object training data for symmetrical objects, and using the training data obtained by the three methods as training data for training the preset model. In one example, Nerf is used to generate real data based on real objects, and simulated data is not generated based on real objects. 1w groups of real data are obtained to simulate the image data (RGB images, depth images) collected in different lighting scenes of the target object. And 5w+ levels of simulated data are obtained to simulate different shooting scenes and enhance the generalization ability of the model. At the same time, the data is blurred to adapt to the situation where the camera moves too fast and causes the captured object to be blurred, thereby increasing robustness.

[0111] This paper constructs a data factory with strong generalization and robustness by combining different data generation methods, which is conducive to the cross-scene use of the network and the pose estimation of symmetrical objects.

[0112] It is understandable that in the context of object pose detection, when using a preset model to acquire 3D point cloud data based on the input second depth point cloud data and three-primary color images, excessive SoftMax operations will not only fail to improve accuracy, but will actually slow down inference speed and increase power consumption. This disclosure removes the redundant SoftMax layers in the preset model, reducing the inference time and running power consumption of the preset model without reducing accuracy.

[0113] The following embodiments of the present disclosure illustrate a method for determining a target image corresponding to a target object in an image to be processed.

[0114] Figure 17 FIG. 1 is a flow chart showing a method for determining a target image corresponding to a target object in an image to be processed according to an exemplary embodiment. Figure 17 As shown, the method includes steps S601 to S602.

[0115] In step S601 , in response to acquiring an image to be processed and a depth image corresponding to the image to be processed.

[0116] In step S602, a target image corresponding to a target object in the image to be processed is obtained through a target segmentation network, where the target segmentation network is trained based on three primary color RGB images and depth images of the target object at different angles.

[0117] In an embodiment of the present disclosure, a target image corresponding to a target object in an image to be processed is obtained through a preset image segmentation network. The present disclosure presets an image segmentation network for various target objects with posture detection (i.e., position detection) requirements. For target objects with posture detection requirements, three-primary-color RGB images and depth images of the target object at different angles are obtained, and training data for training the target segmentation network (i.e., three-primary-color RGB images, depth images, and target object image masks corresponding to each other at different angles) are obtained based on the three-primary-color RGB images and depth images of the target object at different angles. The target segmentation network is obtained based on the data training. Thus, for various target objects with posture detection requirements, the target image corresponding to the target object in the image to be processed is obtained through the image segmentation network.

[0118] The following embodiment of the present disclosure describes a method for obtaining second depth point cloud data.

[0119] Figure 18 FIG. 1 is a flow chart showing a method for obtaining second depth point cloud data according to an exemplary embodiment. Figure 18 As shown, the method includes steps S701 to S702.

[0120] In step S701 , first depth point cloud data corresponding to a target object in a depth image is determined.

[0121] In step S702, the first depth point cloud data is sampled according to a first quantity using a preset sampling method to obtain second depth point cloud data, where the second depth point cloud data includes multiple first sampling point sets, and each depth point cloud data set in the multiple first sampling point sets includes a first number of first sampling points.

[0122] In the embodiment of the present disclosure, the preset sampling method is a sampling method for sampling multiple sampling points into a single sampling point set. The process of sampling the first depth point cloud data to obtain the second depth point cloud data by the preset sampling method can be regarded as a "downsampling" process. Before performing feature fusion, the present disclosure performs a process similar to downsampling on the first depth point cloud data according to the first quantity through the preset sampling method to obtain the second depth point cloud data containing multiple first sampling point sets (each sampling point set contains the first number of first sampling points), so that in the subsequent feature fusion stage, the second depth point cloud data is directly subjected to feature fusion with the target image after the downsampling, avoiding the large amount of computation caused by the step-by-step feature fusion, and solving the problems of the network structure and algorithm process in the related technology being too redundant, the deployment speed being too slow on the mobile terminal, and the high power consumption. The amount of computation in the posture detection process is reduced, and the performance requirements of the posture detection network for implementing the posture detection method on the device are reduced, so that the detection network for posture detection can run stably on the portable mobile terminal.

[0123] In an exemplary embodiment, the preset sampling method used in the present disclosure is a sampling method based on the nearest neighbor classification algorithm (K-Nearest Neighbor, KNN).

[0124] In an exemplary embodiment of the present disclosure, Figure 19 As shown in the flowchart of the pose detection method, the present disclosure performs pose detection of the target object in the following manner: in response to the input of the image to be processed, the image data is preprocessed, the target segmentation is performed based on the image segmentation network, the region of interest is selected, and the region of interest corresponding to the target object (i.e., the target image) is obtained. And the region of interest is selected for the depth image area based on the acquired target image to obtain the depth point cloud data corresponding to the target image. The depth point cloud data and the target image are input into the pose estimation network, and 3D point cloud generation is performed to obtain a 3D point cloud corresponding to the target object. The 3D point cloud corresponding to the target object is used as the key point and the key point post-processing is performed to obtain a 3D point cloud pair that represents the current position of the target object. The alignment between the 3D point cloud pair and the template point cloud pair is achieved based on the preset algorithm, and the rotation and translation RT matrix obtained by the alignment is used as the pose detection result and output.

[0125] In the embodiments of the present disclosure, by using data from multiple sources as training data for the pose detection model, the present disclosure can be generalized to different scenarios, and has better performance for surface reflective objects and symmetrical objects. The present disclosure reduces the amount of computation in the feature fusion stage, reduces the demand for device performance, and makes the present disclosure easy to deploy on mobile terminals. The present disclosure uses the Umeyama algorithm for point alignment to obtain an accurate translation and rotation RT matrix. It can adapt to different distances and solves the problem of inaccurate translation matrix in the alignment process.

[0126] Based on the same concept, an embodiment of the present disclosure also provides a posture detection device 100.

[0127] It is understandable that the posture detection device 100 provided in the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function in order to realize the above functions. In combination with the units and algorithm steps of each example disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.

[0128] Figure 20 FIG. 1 is a block diagram of a posture detection device 100 according to an exemplary embodiment. Figure 20The device includes a determining unit 101, a sampling unit 102 and a processing unit 103.

[0129] The determining unit 101 is configured to determine a target image corresponding to a target object in the image to be processed and determine first depth point cloud data corresponding to the target object in the depth image in response to acquiring the image to be processed and the depth image corresponding to the target object.

[0130] The first depth point cloud data includes a plurality of first sampling points, and the plurality of first sampling points correspond to depth values ​​and pixel coordinates of pixel points of the target object image in the depth image.

[0131] The sampling unit 102 is configured to sample the first depth point cloud data to obtain second depth point cloud data, where the second depth point cloud data includes a plurality of first sampling point sets, and each depth point cloud data set in the plurality of first sampling point sets includes a plurality of first sampling points.

[0132] The processing unit 103 is configured to determine a pose detection result of the target object based on the target image and the second depth point cloud data.

[0133] In one embodiment, the processing unit 103 determines the pose detection result of the target object based on the target image and the second depth point cloud data in the following manner: the target image and the second depth point cloud data are input into a preset model to obtain first three-dimensional point cloud data corresponding to the target object, and the preset model is used to output three-dimensional point cloud data corresponding to the target object based on the input target object image and the corresponding depth point cloud data; based on the first three-dimensional point cloud data and the preset template data, the pose detection result of the target object is determined, and the pose detection result is a rotation and translation RT matrix, which represents the three-degree-of-freedom rotation data and three-degree-of-freedom translation data of the target object.

[0134] In one embodiment, after acquiring the target image, the apparatus further comprises: acquiring an image mask corresponding to the target image. The determining unit 101 determines first depth point cloud data corresponding to the target object in the depth image by determining the first depth point cloud data corresponding to the target object in the depth image based on an overlapping relationship between the image mask and the depth image.

[0135] In one embodiment, the first three-dimensional point cloud data includes multiple sets of second sampling point sets, each second sampling point set in the multiple sets of second sampling point sets corresponding one-to-one to each first sampling point set in the multiple sets of first sampling point sets. The processing unit 103 determines a pose detection result of the target object based on the first three-dimensional point cloud data and preset template data in the following manner: A confidence level is determined for each second sampling point set in the first three-dimensional point cloud data, where the confidence level represents the degree of fit between the second sampling point set and the target object. Second sampling point sets in the first three-dimensional point cloud data whose confidence level exceeds a confidence threshold are determined as second three-dimensional point cloud data. Clustering is performed on the second three-dimensional point cloud data based on a second quantity to obtain a preset number of three-dimensional point cloud pairs. A preset registration method is used to register the preset number of three-dimensional point cloud pairs based on the template data to obtain a translation and rotation RT matrix representing the translation and rotation relationship between the template data and the preset number of three-dimensional point cloud pairs. The translation and rotation RT matrix is ​​determined as the pose detection result. The template data is pre-acquired data corresponding to the initial pose of the target object, and the template data includes the preset number of three-dimensional point cloud pairs.

[0136] In one embodiment, the preset model is trained by the processing unit 103 based on pre-collected training data, where the training data includes data generated by a modeling tool based on a target object and simulation data generated by a simulation tool.

[0137] In one embodiment, the data generated based on the target object by the modeling tool includes training data corresponding to the symmetrical object. The training data corresponding to the symmetrical object is obtained by the processing unit 103 in the following manner: the symmetrical angle range of the symmetrical object relative to the positive direction of the camera is determined, and the initial position of the symmetrical object is determined. Within the symmetrical angle range, the symmetrical object is rotated multiple times around the symmetry axis in the positive direction of the camera. During each rotation, a depth image and a three-primary color RGB image of the symmetrical object are obtained, and the depth image and the three-primary color RGB image are determined as the training data.

[0138] In one embodiment, the determination unit 101 determines the target image corresponding to the target object in the image to be processed in the following manner, including: obtaining the target image corresponding to the target object in the image to be processed through a target segmentation network, where the target segmentation network is trained based on the three primary color RGB image and depth image of the target object at different angles.

[0139] In one embodiment, the sampling unit 102 samples the first depth point cloud data in the following manner to obtain second depth point cloud data: the first depth point cloud data is sampled according to a first quantity using a preset sampling method to obtain second depth point cloud data, the second depth point cloud data including multiple first sampling point sets, and each depth point cloud data set in the multiple first sampling point sets includes a first number of first sampling points.

[0140] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0141] Figure 21 FIG2 is a block diagram illustrating an apparatus 200 for posture detection according to an exemplary embodiment. The apparatus 200 may be provided as a terminal. For example, the apparatus 200 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0142] Reference Figure 21 , apparatus 200 may include one or more of the following components: a processing component 202 , a memory 204 , a power component 206 , a multimedia component 208 , an audio component 210 , an input / output (I / O) interface 212 , a sensor component 214 , and a communication component 216 .

[0143] The processing component 202 generally controls the overall operation of the device 200, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 202 may include one or more modules to facilitate interaction between the processing component 202 and other components. For example, the processing component 202 may include a multimedia module to facilitate interaction between the multimedia component 208 and the processing component 202.

[0144] The memory 204 is configured to store various types of data to support operations on the device 200. Examples of such data include instructions for any application or method operating on the device 200, contact data, phone book data, messages, pictures, videos, etc. The memory 204 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0145] The power component 206 provides power to the various components of the device 200. The power component 206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device 200.

[0146] The multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensor can not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 208 includes a front camera and / or a rear camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0147] The audio component 210 is configured to output and / or input audio signals. For example, the audio component 210 includes a microphone (MIC) that is configured to receive external audio signals when the device 200 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may be further stored in the memory 204 or transmitted via the communication component 216. In some embodiments, the audio component 210 further includes a speaker for outputting audio signals.

[0148] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as a keyboard, click wheel, buttons, etc. These buttons may include but are not limited to: a home button, volume buttons, a start button, and a lock button.

[0149] The sensor assembly 214 includes one or more sensors for providing various aspects of the status assessment of the device 200. For example, the sensor assembly 214 can detect the open / closed state of the device 200, the relative positioning of components, such as the display and keypad of the device 200. The sensor assembly 214 can also detect changes in the position of the device 200 or a component of the device 200, the presence or absence of user contact with the device 200, the orientation or acceleration / deceleration of the device 200, and temperature changes of the device 200. The sensor assembly 214 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 214 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 214 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0150] The communication component 216 is configured to facilitate wired or wireless communication between the device 200 and other devices. The device 200 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 216 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 216 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0151] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described method.

[0152] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as the memory 204 including instructions, which can be executed by the processor 220 of the apparatus 200 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0153] It is understood that in this disclosure, "plurality" refers to two or more than two, and other quantifiers are similar. "And / or" describes the association relationship of related objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects before and after are in an "or" relationship. The singular forms "a", "the" and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0154] It will be further understood that the terms "first," "second," and the like are used to describe various types of information, but such information should not be limited to these terms. These terms are used solely to distinguish information of the same type from one another and do not indicate a particular order or level of importance. In fact, the terms "first," "second," and the like are fully interchangeable. For example, first information could be referred to as second information, and similarly, second information could be referred to as first information without departing from the scope of this disclosure.

[0155] It can be further understood that the terms "center", "longitudinal", "lateral", "front", "back", "up", "down", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing this embodiment and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation.

[0156] It is further understood that, unless otherwise specified, “connection” includes a direct connection where there are no other components between the two elements, and also includes an indirect connection where there are other elements between the two elements.

[0157] It is further understood that although operations are described in a particular order in the drawings in the embodiments of the present disclosure, this should not be construed as requiring that the operations be performed in the particular order shown or in a serial order, or that all of the operations shown be performed to obtain the desired results. In certain circumstances, multitasking and parallel processing may be advantageous.

[0158] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein.

[0159] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the scope of the appended claims.

Claims

1. A posture detection method, characterized in that: include: In response to acquiring an image to be processed and a depth image corresponding to the image to be processed, determining a target image corresponding to a target object in the image to be processed, and determining first depth point cloud data corresponding to the target object in the depth image, wherein the first depth point cloud data includes a plurality of first sampling points, and the plurality of first sampling points correspond to depth values ​​and pixel coordinates of pixel points of the target object image in the depth image; Sampling the first depth point cloud data to obtain second depth point cloud data, where the second depth point cloud data includes a plurality of first sampling point sets, and each depth point cloud data set in the plurality of first sampling point sets includes a plurality of first sampling points; Determine a pose detection result of the target object based on the target image and the second depth point cloud data.

2. The method according to claim 1, characterized in that Determining a pose detection result of the target object according to the target image and the second depth point cloud data includes: Inputting the target image and the second depth point cloud data into a preset model to obtain first three-dimensional point cloud data corresponding to the target object, wherein the preset model is used to output three-dimensional point cloud data corresponding to the target object based on the input target object image and the corresponding depth point cloud data; Based on the first three-dimensional point cloud data and the preset template data, a posture detection result of the target object is determined, wherein the posture detection result is a rotation and translation RT matrix, and the rotation and translation RT matrix represents the three-degree-of-freedom rotation data and three-degree-of-freedom translation data of the target object.

3. The method according to claim 2, characterized in that After acquiring the target image, the method further includes: acquiring an image mask corresponding to the target image; The determining first depth point cloud data corresponding to the target object in the depth image includes: First depth point cloud data corresponding to the target object in the depth image is determined according to an overlapping relationship between the image mask and the depth image.

4. The method according to claim 2, characterized in that The first three-dimensional point cloud data includes a plurality of second sampling point sets, each second sampling point set in the plurality of second sampling point sets corresponds one-to-one to each first sampling point set in the plurality of first sampling point sets; Determining the pose detection result of the target object based on the first three-dimensional point cloud data and preset template data includes: Determining a confidence level of each second sampling point set in the first three-dimensional point cloud data, wherein the confidence level represents a degree of fit between the second sampling point set and the target object; Determining a second set of sampling points in the first three-dimensional point cloud data, whose confidence level is greater than a confidence threshold, as second three-dimensional point cloud data; performing clustering processing on the second three-dimensional point cloud data according to the second quantity to obtain a preset number of three-dimensional point cloud pairs; Using a preset registration method, registering the preset number of three-dimensional point cloud pairs according to the template data, obtaining a translation and rotation RT matrix representing the translation and rotation relationship between the template data and the preset number of three-dimensional point cloud pairs, and determining the translation and rotation RT matrix as the pose detection result, The template data is pre-acquired data corresponding to the initial posture of the target object, and the template data includes a preset number of three-dimensional point cloud pairs.

5. The method according to claim 2, characterized in that The preset model is trained based on pre-collected training data, and the training data includes data generated by a modeling tool based on a target object and simulation data generated by a simulation tool.

6. The method according to claim 5, characterized in that The data generated based on the target object by the modeling tool includes training data corresponding to the symmetrical object, and the training data corresponding to the symmetrical object is obtained in the following manner: Determine the symmetry angle range of the symmetrical object relative to the positive direction of the camera, and determine the initial pose of the symmetrical object; Within the symmetric angle range, the symmetrical object is rotated multiple times around the symmetry axis in the positive direction of the camera, and a depth image and a three-primary color RGB image of the symmetrical object are obtained during each rotation process, and the depth image and the three-primary color RGB image are determined as the training data.

7. The method according to claim 1, characterized in that Determining a target image corresponding to a target object in the image to be processed includes: The target image corresponding to the target object in the image to be processed is obtained through a target segmentation network, and the target segmentation network is trained based on the three primary color RGB image and depth image of the target object at different angles.

8. The method according to claim 1, characterized in that The sampling of the first depth point cloud data to obtain second depth point cloud data includes: The first depth point cloud data is sampled according to a first quantity using a preset sampling method to obtain second depth point cloud data. The second depth point cloud data includes a plurality of first sampling point sets, and each depth point cloud data set in the plurality of first sampling point sets includes a first number of first sampling points.

9. A posture detection device, characterized in that: include: a determining unit, configured to, in response to acquiring an image to be processed and a depth image corresponding to the image to be processed, determine a target image corresponding to a target object in the image to be processed, and determine first depth point cloud data corresponding to the target object in the depth image, wherein the first depth point cloud data includes a plurality of first sampling points, and the plurality of first sampling points correspond to depth values ​​and pixel coordinates of pixel points of the target object image in the depth image; a sampling unit, configured to sample the first depth point cloud data to obtain second depth point cloud data, wherein the second depth point cloud data includes a plurality of first sampling point sets, and each depth point cloud data set in the plurality of first sampling point sets includes a plurality of first sampling points; A processing unit is used to determine a pose detection result of the target object based on the target image and the second depth point cloud data.

10. The device according to claim 9, characterized in that The processing unit determines a pose detection result of the target object according to the target image and the second depth point cloud data in the following manner: The target image and the second depth point cloud data are input into a preset model to obtain first three-dimensional point cloud data corresponding to the target object. The preset model is used to output three-dimensional point cloud data corresponding to the target object based on the input target object image and the corresponding depth point cloud data. The posture detection result of the target object is determined based on the first three-dimensional point cloud data and the preset template data. The posture detection result is a rotation and translation RT matrix, and the rotation and translation RT matrix represents the three-degree-of-freedom rotation data and three-degree-of-freedom translation data of the target object.

11. The device according to claim 9, characterized in that After acquiring the target image, the apparatus further comprises: acquiring an image mask corresponding to the target image; The determining unit determines the first depth point cloud data corresponding to the target object in the depth image in the following manner: First depth point cloud data corresponding to the target object in the depth image is determined according to an overlapping relationship between the image mask and the depth image.

12. The device according to claim 10, characterized in that The first three-dimensional point cloud data includes a plurality of second sampling point sets, each second sampling point set in the plurality of second sampling point sets corresponds one-to-one to each first sampling point set in the plurality of first sampling point sets; The processing unit determines the pose detection result of the target object according to the first three-dimensional point cloud data and the preset template data in the following manner: Determining a confidence level of each second sampling point set in the first three-dimensional point cloud data, wherein the confidence level represents a degree of fit between the second sampling point set and the target object; Determining a second set of sampling points in the first three-dimensional point cloud data, whose confidence level is greater than a confidence threshold, as second three-dimensional point cloud data; performing clustering processing on the second three-dimensional point cloud data according to the second quantity to obtain a preset number of three-dimensional point cloud pairs; Using a preset registration method, registering the preset number of three-dimensional point cloud pairs according to the template data, obtaining a translation and rotation RT matrix representing the translation and rotation relationship between the template data and the preset number of three-dimensional point cloud pairs, and determining the translation and rotation RT matrix as the pose detection result, The template data is pre-acquired data corresponding to the initial posture of the target object, and the template data includes a preset number of three-dimensional point cloud pairs.

13. The device according to claim 10, characterized in that The preset model is trained by the processing unit based on pre-collected training data, and the training data includes data generated by a modeling tool based on a target object and simulation data generated by a simulation tool.

14. The device according to claim 13, characterized in that The data generated based on the target object by the modeling tool includes training data corresponding to the symmetrical object, and the training data corresponding to the symmetrical object is obtained by the processing unit in the following manner: Determine the symmetry angle range of the symmetrical object relative to the positive direction of the camera, and determine the initial pose of the symmetrical object; Within the symmetric angle range, the symmetrical object is rotated multiple times around the symmetry axis in the positive direction of the camera, and a depth image and a three-primary color RGB image of the symmetrical object are obtained during each rotation process, and the depth image and the three-primary color RGB image are determined as the training data.

15. The device according to claim 9, characterized in that The determining unit determines the target image corresponding to the target object in the image to be processed in the following manner, including: The target image corresponding to the target object in the image to be processed is obtained through a target segmentation network, and the target segmentation network is trained based on the three primary color RGB image and depth image of the target object at different angles.

16. The device according to claim 9, characterized in that The sampling unit samples the first depth point cloud data in the following manner to obtain second depth point cloud data: The first depth point cloud data is sampled according to a first quantity in a preset sampling method to obtain second depth point cloud data. The second depth point cloud data includes a plurality of first sampling point sets, and each depth point cloud data set in the plurality of first sampling point sets includes a first number of first sampling points.

17. An electronic device, characterized in that: include: processor: a memory for storing processor-executable instructions; The processor is configured to execute the posture detection method according to any one of claims 1 to 8.

18. A storage medium, characterized in that The storage medium stores instructions, and when the instructions in the storage medium are executed by the processor, the processor is enabled to execute the posture detection method according to any one of claims 1 to 8.

Images