Object posture estimation method, device, electronic device and storage medium
By combining color images and depth images, adaptively selecting three-dimensional or two-dimensional key points for object pose estimation, the accuracy problem under the influence of depth image quality in the prior art is solved, and higher pose estimation accuracy and robustness are achieved sex.
Patent Information
- Application Number
- CN202011050187.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-09-29
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2040-09-29
AI Technical Summary
The existing pose estimation methods for object are limited in complex application scenarios and cannot meet the real-time and accuracy requirements. Especially when the depth image quality is poor, the pose estimation results are inaccurate.
By combining color images and depth images, the reliability of the depth image is determined. When reliable, the object pose estimation is performed based on three-dimensional key points, and when unreliable, the estimation is performed based on two-dimensional key points, which reduces the dependence on the quality of the depth image and improves the pose estimation. robustness and accuracy.
It improves the accuracy and robustness of object attitude estimation, and can obtain accurate results in the absence of depth data, noise or occlusion, meeting the needs of real-time and efficient object attitude detection.
Smart Images

Figure CN114332214B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence and augmented reality technology. Specifically, the present application relates to an object posture estimation method, device, electronic device and storage medium. Background Art
[0002] With the development of information technology and artificial intelligence, society has increasingly strong demands for automation and intelligence, and technologies such as virtual reality, autonomous driving and robotics are becoming more and more important. Among them, object posture estimation technology can solve the posture information of the camera relative to the object, and based on this posture information, the spatial shape around the object can be constructed. Therefore, it plays an important role in technologies such as virtual reality, autonomous driving, and augmented reality.
[0003] In the prior art, object posture estimation is often performed based on color images. However, the accuracy of the results that can be achieved by this method in complex application situations is limited and cannot achieve the accuracy required for practical applications. Summary of the invention
[0004] The purpose of this application is to provide a method for object posture estimation to improve the accuracy of object posture estimation results. The solution provided in the embodiment of this application is as follows:
[0005] On the one hand, the present application provides a method for estimating an object posture, comprising:
[0006] Determining the reliability of the depth image based on the color image and the depth image of the object;
[0007] When the depth image is reliable, the pose of the object is estimated based on the 3D key points;
[0008] When the depth image is unreliable, the pose of the object is estimated based on 2D key points.
[0009] On the other hand, the present application provides an object posture estimation device, comprising:
[0010] An image reliability determination module, used to determine the reliability of the depth image based on the color image and the depth image of the object;
[0011] The posture estimation module is used to estimate the posture of the object based on the three-dimensional key points when the depth image is reliable; when the depth image is unreliable, the posture estimation of the object is based on the two-dimensional key points.
[0012] On the other hand, the present application provides an electronic device, the electronic device comprising a memory and a processor;
[0013] A computer program is stored in the memory;
[0014] The processor is used to execute the object posture estimation method provided in the embodiment of the present application when running a computer program.
[0015] On the other hand, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the object posture estimation method provided in the embodiment of the present application is executed.
[0016] The beneficial effects brought about by the technical solution provided in the embodiments of the present application will be described in detail in conjunction with various optional embodiments in the description of the specific implementation method section below, and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in describing the embodiments of the present application are briefly introduced below.
[0018] Figure 1 A flowchart of an object posture estimation method provided by an embodiment of the present application is shown;
[0019] Figure 2 This is a flow chart of object posture estimation based on a single feature extraction in an example of this application;
[0020] Figure 3 This is a flow chart of object posture estimation based on two feature extractions in an example of this application;
[0021] Figure 4 This is a flow chart of performing posture estimation of an object based on a depth image in an example of the present application;
[0022] Figure 5 A flowchart of fusing image features and point cloud features to obtain fused features provided in an example of the present application;
[0023] Figure 6 A flowchart of the present application is shown, which is based on a primary feature extraction and uses a second appearance feature of the target object to perform object posture estimation;
[0024] Figure 7 A flowchart of object posture estimation based on two feature extractions and using the second appearance feature of the target object provided by the present application is shown;
[0025] Figure 8 A flowchart of an object posture estimation method provided by another embodiment of the present application is shown;
[0026] Fig. 9 A flowchart of object posture estimation based on color image and depth image provided by an embodiment of the present application is shown;
[0027] Fig.10 A flowchart of an embodiment of the present invention for estimating an object posture based on an initial frame in a video is shown.
[0028] Fig.11 A schematic diagram of three initial frame selection methods provided by an embodiment of the present application is shown;
[0029] Fig.12 A flowchart of determining whether a video frame is an initial frame based on object detection provided by an embodiment of the present application is shown;
[0030] Fig.13 A flowchart of confirming a new object or a new posture provided by an embodiment of the present application is shown;
[0031] Fig.14 A schematic diagram showing the structure of an object posture estimation device provided by an embodiment of the present application is shown;
[0032] Fig.15 A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION
[0033] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present invention.
[0034] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.
[0035] In order to better understand and illustrate the solutions provided by the embodiments of the present application, the relevant technologies involved in the present application are first described below.
[0036] Object: An object with six degrees of freedom in space, also called a 6-degree-of-freedom object, such as a cup, a monitor, a book, etc.
[0037] Augmented reality technology provides users with a real information experience by adding virtual content to the display scene in front of the user. In three-dimensional space, the augmented reality system needs to have high-precision real-time processing and understanding of the three-dimensional state of surrounding objects in order to present a high-quality virtual-real fusion effect in front of the user.
[0038] Posture estimation uses geometric models or structures to represent the structure and shape of an object, and by extracting the features of the object, a correspondence is established between the model and the image, and then the spatial posture of the object is estimated through geometry or other methods. The model used here can be a simple geometric shape, such as a plane or cylinder, or a certain geometric structure, or a three-dimensional model obtained through laser scanning or other methods.
[0039] 6D pose estimation of images: For a given single image containing color and depth information, estimate the 6D pose of the target object in the image. The 6D pose is also called the 6-degree-of-freedom pose, which includes 3D position and 3D spatial orientation.
[0040] MLP (Multi-Layer Perceptron) network: It is a forward-structured artificial neural network, a multi-layer neural network.
[0041] Semantic segmentation: Separate the target object from the background.
[0042] Object detection: determining the location of objects in an image.
[0043] Instance segmentation: Determine which pixels belong to each target object.
[0044] During the research process, the inventors found that the existing object posture estimation method can use depth images and color images to estimate the object posture, but this method relies on the quality of the depth image. Once the depth data is missing, there are errors and noise, the accuracy of the obtained object posture is not high.
[0045] Furthermore, when estimating the 6D pose of an object based on color images and depth images, heterogeneous reception is used to process the color image and depth image separately, and a fusion network is used to extract features, and pose estimation is performed based on the extracted features. However, this method only uses the features of a single object, and when the object has image overlap or shadows, accurate object pose estimation cannot be achieved.
[0046] In addition, when fusing color image features and depth features for object pose estimation, a large amount of memory and resources are consumed, the efficiency of accurate pose estimation is low and cannot meet the real-time requirements of object pose estimation.
[0047] In order to solve at least one of the above problems, the present application provides an object posture estimation method, which can effectively improve the accuracy of object posture estimation when estimating the object posture based on color images and depth images, and does not rely on the reliable quality of depth images.
[0048] In order to make the purpose, technical solutions and advantages of the present application clearer, the following will be combined with specific embodiments and drawings to describe in detail how the various optional implementation methods of the present application and the technical solutions of the embodiments of the present application solve the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the drawings. Figure 1 A method for estimating an object posture provided by an embodiment of the present application is shown in FIG. 1 , and the method includes:
[0049] Step S110: determining the reliability of the depth image according to the color image and the depth image of the object;
[0050] Step S120, when the depth image is reliable, performing posture estimation of the object based on the three-dimensional key points;
[0051] Step S130: when the depth image is unreliable, the posture of the object is estimated based on the two-dimensional key points.
[0052] Among them, the color image and the depth image are images corresponding to the same scene and containing the same object. The type and quantity of the objects are not limited here. The method of acquiring the color image and the corresponding depth image is not limited in the embodiment of the present application. For example, it can be a color image and a depth image simultaneously acquired by an ordinary image acquisition device and a depth image acquisition device, or it can be acquired by an image acquisition device or video acquisition device that has both depth image and color image acquisition functions. For example, an RGBD image (Red+Green+Blue+depth, an image containing the three primary colors of red, green and blue and depth information) is acquired by an image acquisition device, and the depth image and the color image can be obtained based on the RGBD image.
[0053] Among them, for the above color image, whether to use a grayscale image or a color image can be configured and selected according to actual application requirements. For example, in order to obtain a better posture estimation effect, a color image can be used. If the efficiency of posture estimation is required to be higher, a grayscale image can be used. For the convenience of description, in the following example description, a color image will be used as an example for description.
[0054] In practical applications, continuous and real-time object posture estimation is required in many application scenarios. At this time, optionally, the RGBD video of the object can be collected by a video acquisition device. Each frame of the video is an RGBD image. The color image and depth image of the object can be extracted from the same video frame. For each video frame, the corresponding color image and depth image can be obtained. The color image corresponding to the same video frame is consistent with the object in the depth image, and the object can be one or more. For the color image and depth image corresponding to each video frame collected, the real-time estimation of the object posture can be achieved based on the solution provided in the embodiment of the present application.
[0055] Among them, image feature extraction and processing can be performed based on the color image and the depth image, and the processed image features can be used to determine the reliability of the depth image. Specifically, the reliability determination of the depth image provided in the present application can be a reliability determination of the depth data in the depth image. Usually, the depth image includes multiple data such as depth data and contour data. Determining the reliability of the depth image can specifically be determining whether the depth data of the depth image is reliable.
[0056] Among them, the reliability judgment can be obtained by setting a reliability threshold or setting a ratio threshold of reliable pixels for statistics. When the reliability corresponding to the depth image exceeds the set reliability threshold or the pixel ratio of statistically reliable pixels exceeds the set ratio threshold, it indicates that the depth image is reliable.
[0057] If the depth image is reliable, the object pose estimation is performed based on the 3D key points; if it is unreliable, the object pose estimation is performed based on the 2D key points.
[0058] Among them, the two-dimensional key points can be obtained only by using the color image. In this case, in the network structure described in this application, the MLP network whose input is in the point cloud feature format of the feature extraction module for extracting the two-dimensional key points is replaced by a CNN network whose input is in the image feature format. In addition, the two-dimensional key points and the three-dimensional key points can also be obtained based on the depth image and the color image.
[0059] If the depth image is reliable, the pose estimation of the object is performed based on the three-dimensional key points. Optionally, the three-dimensional key points can be obtained based on the depth image and the color image, and the pose estimation of the object is performed using the obtained three-dimensional key points. The data of the three-dimensional key points includes the depth data provided by the depth image. Compared with the method of estimating the pose of an object using only the color image, the present application estimates the pose of an object based on the color image and the depth image, which increases the dimension of the basic data and helps to improve the accuracy and precision of the pose estimation of the object. Moreover, it is possible to obtain accurate poses of objects even when the object is blocked, the sensor is noisy, or the lighting conditions are insufficient.
[0060] If the depth image is unreliable, the pose estimation of the object is performed based on the two-dimensional key points. Optionally, the two-dimensional key points of the image can be obtained based on the contour data of the color image and the depth image, and the pose estimation is performed using the obtained two-dimensional key points. Since the depth data in the depth image is unreliable, in this case, if the pose estimation of the object is performed based on the depth data, the pose of the object may be incorrect. Therefore, in this case, the pose estimation can be performed based on the color image and the object contour data in the depth image, which does not rely on the integrity of the depth data in the depth image. In the case of missing depth data, errors and noise, the pose estimation is performed based on the color image and the object contour data. Compared with the solution of estimating the pose of the object only through the color image, the accuracy of the estimated pose of the object can also be improved.
[0061] The present application uses depth images and color images to estimate the pose of an object. First, the reliability of the depth image is determined based on the color image and the depth image. When the depth image is reliable, the pose of the object is estimated based on three-dimensional key points. When the depth image is unreliable, that is, when the depth data is missing, has errors and noise, the pose of the object is estimated based on two-dimensional key points, reducing the dependence on the complete depth image to improve the accuracy of the object pose estimation. Moreover, the solution provided by the present application adaptively selects whether to perform the pose estimation of the object based on the depth image or the color image according to the reliability judgment result of the depth image, which is conducive to improving the robustness of the object pose estimation.
[0062] The following is a detailed description of the various optional embodiments provided in this application.
[0063] In an optional embodiment of the present application, the reliability of the depth image is determined according to the color image and the depth image of the object, which can be obtained by at least one of the schemes A and B:
[0064] Option A may include the following steps:
[0065] A1: Extract image features based on color image, or color image and depth image;
[0066] A2, extract point cloud features based on depth images;
[0067] A3, fuse the image features and point cloud features to obtain fused features;
[0068] A4, determines the reliability of the depth image based on the fused features.
[0069] It is understandable that some of the above steps may not have a certain order, for example, between step A1 and step A2.
[0070] The solution provided in this embodiment can extract image features based on color images, or can extract image features based on color images and depth images, wherein the image features can be extracted through image feature extraction networks, image feature extraction algorithms, etc. Since the subsequent steps in the solution provided in this embodiment will fuse image features and point cloud features, when only extracting image features based on color images, the fused features also include data such as depth data and contour data of the depth image.
[0071] An optional embodiment of the present application provides a solution as follows Figure 2 As shown, in this scheme, feature extraction is performed based on the color image and the depth image, and image feature extraction is performed through an image feature extraction network, that is, the color image and the depth image are input into the image feature extraction network. When the color image is a color image, the input of the image feature extraction network is a 4-channel image of H*W obtained by concatenating the color image and the depth image pixel by pixel, wherein H is the image height, W is the image width, and the 4 channels are three channels corresponding to the RGB data of the color image and a channel corresponding to the depth data of the depth image. The output of the image feature extraction network includes an image feature vector for each pixel.
[0072] Optionally, to extract point cloud features based on the depth image, the depth image can be first converted into a point cloud to obtain point cloud data corresponding to the depth image, and then point cloud features can be extracted based on the point cloud data to obtain point cloud features corresponding to the depth image. Point cloud feature extraction can be performed through a point cloud feature extraction network, the input of which is point cloud data, and the output of which includes a point cloud feature vector for each three-dimensional point, thereby obtaining a point cloud feature vector for each pixel, wherein the point cloud feature vector can be represented as Figure 2 The geometric eigenvectors shown in .
[0073] The obtained image features are fused with the point cloud features to obtain fused features. Specifically, for each pixel contained in the image, the image feature vector of each pixel obtained by image feature extraction and the point cloud feature vector of each pixel obtained by point cloud feature extraction are fused pixel by pixel to obtain fused features.
[0074] Optionally, the fusion operation may be a dense fusion, and the fusion operation may be to concatenate the image feature vector and the point cloud feature vector, or to project the image feature vector and / or the point cloud feature vector to another feature space, and use the feature vector in the feature space as the fusion feature of the pixel. Among them, by fusion mode of projecting the image feature vector and / or the point cloud feature vector to another feature space, it is possible to filter out the interference information in the image and reduce the information redundancy of the fusion feature.
[0075] Compared with the method of using color images alone to make reliability judgments, the reliability judgment of depth images based on fusion features is more accurate and is conducive to improving the efficiency of judging the reliability of depth images.
[0076] Optionally, the process of judging the reliability of a depth image based on the fusion feature may be as follows: obtaining a depth reliability image composed of the depth data of each pixel based on the fusion feature, and determining whether the depth image is reliable according to the depth data of each pixel in the depth reliability image and a preset reliability threshold or ratio threshold.
[0077] Option B may include the following steps:
[0078] B1, extract image features based on color image and depth image;
[0079] B2, determine the reliability of the depth image based on the image features.
[0080] An optional embodiment of the present application provides a solution as follows Figure 3 As shown, the image feature extraction network is used to extract features from the color image and the depth image to obtain image features. When there are at least two target objects in the image, the target objects are regionalized, the image features can be regionalized, and the image area of the suspected target object is segmented to obtain the image area (ROI, Region of Interest) of each target object. Pooling is performed on the image area to obtain the image features of the image area, and the depth reliability image corresponding to the target object is obtained based on the pooled image features. The depth reliability image corresponding to the target object is compared with a preset reliability threshold or ratio threshold to determine the reliability of the image data.
[0081] In the optional embodiment provided in the present application, the reliability of the depth image may be determined based on the fusion feature in the following manners, including:
[0082] C1, based on the fusion features, obtain the object instance segmentation image and the depth reliability image;
[0083] C2, based on the object instance segmentation image and the depth reliability image, determines the reliability of the depth image corresponding to each target object in the color image.
[0084] Combination Figure 2As shown in the figure, after obtaining the fusion features of the image features and the point cloud features, the reliability of the depth image can be predicted based on the fusion features, the depth reliability image corresponding to the image can be determined, and the object instance segmentation can be performed based on the semantic segmentation and center offset estimation of the fusion features to obtain the object instance segmentation image. Then, the reliability of the depth image corresponding to each target object in the color image is determined based on the object instance segmentation image and the depth reliability image.
[0085] Specifically, pixel points in the image area corresponding to each target object can be obtained according to the object instance segmentation image, and the depth data of each pixel point in the image area can be determined according to the depth reliability image, that is, the depth image corresponding to each target object is obtained, and the reliability of the depth image corresponding to each target object is determined according to a preset reliability threshold or a ratio threshold. Specifically, if the depth data of the depth image of a target object exceeds the preset threshold, or the ratio of statistically reliable pixels is greater than the preset ratio threshold, then the depth image of the target object is a reliable image; otherwise, the depth image of the target object is unreliable.
[0086] Since there may be more than one target object in an image, when some target objects are reliable and some are unreliable, the reliability results of each target object cannot be known based on the reliability judgment result of the depth image corresponding to the entire image. To address this problem, in the solution provided in this embodiment, fusion features are used to obtain an object instance segmentation image corresponding to each target object, and then a depth image corresponding to each target object is obtained. The reliability of the depth image corresponding to each target object is determined in combination with a preset reliability threshold, so as to determine the reliability of each target object in the image.
[0087] As another optional method, the reliability of the depth image can be judged without making separate judgments on the objects, but can be a holistic judgment, that is, no matter whether there is one or more target objects in the image, the image as a whole can be judged to obtain a judgment result. If the judgment result is reliable, for each target object, the posture estimation method based on three-dimensional key points can be used to estimate the object posture; if the judgment result is unreliable, for each target object, the posture estimation method based on two-dimensional key points can be used to estimate the object posture. This method is conducive to reducing the amount of data processing in the reliability judgment process and improving the judgment efficiency.
[0088] In an optional embodiment provided by the present application, determining the reliability of the depth image according to the image features may include:
[0089] D1, obtain object instance segmentation image and depth reliability image based on image features;
[0090] D2, based on the object instance segmentation image and the depth reliability image, determines the reliability of the depth image corresponding to each target object in the color image.
[0091] Combination Figure 3 As shown, image features are obtained based on the color image and the depth image, and the image features are regionalized. The regionalization can be performed through a region proposal network to extract image regions that may belong to the target object, and an object instance segmentation image and a depth reliability image corresponding to the target object are obtained based on the image region of each target object. The depth image corresponding to the target object is determined according to the object instance segmentation image and the depth reliability image, and the reliability of the depth image corresponding to the target object is determined according to a comparison result of the depth data of each pixel in the depth image with a preset threshold.
[0092] The solution provided in this embodiment determines the reliability of the depth image corresponding to the target object based on image features, segments the target object based on the image features, and obtains the depth image corresponding to each target object, thereby reducing the amount of data processing for obtaining the reliability judgment of the depth image.
[0093] In an optional embodiment of the present application, obtaining an object instance segmentation image and a depth reliability image according to image features can be achieved by the following methods, including:
[0094] E1, obtaining the regional image features of the image regions corresponding to each target object according to the image features;
[0095] E2, for each target object, based on the regional image features corresponding to the target object, determine the depth reliability image of the target object; based on the regional image features corresponding to each target object, obtain the object instance segmentation image.
[0096] Combination Figure 3 As shown, image features are determined based on color images and depth images, and the image features are regionalized. The target object can be divided into regions through a region proposal network or a region segmentation algorithm to obtain the image region corresponding to each target object. Then, feature extraction is performed based on the image region corresponding to each target object to obtain regional image features, wherein the regional image features can be obtained by pooling the object image region (ROI pooling) to achieve image division at the target object level.
[0097] For each target object, based on the regional image features corresponding to the target object, the regional image features can be processed by the fully connected layer of the neural network to determine the depth reliability image corresponding to the target object. Subsequently, the depth reliability image can be judged whether it is reliable in combination with the preset threshold of the depth data corresponding to the target object.
[0098] For each target object, based on the regional image features corresponding to the target object, the image can be segmented into object instances through a neural network (such as a convolutional neural network) to obtain an object instance segmentation image. The contour information of the target object and the object detail information within the contour can be obtained through the object instance segmentation image.
[0099] On this basis, the depth image corresponding to each target object is determined according to the object instance segmentation image and the corresponding depth reliability image, and the reliability of the depth image of the target object is determined according to the depth image and the corresponding preset threshold.
[0100] The solution provided in this embodiment performs regional division at the target object level based on image features, and on this basis, obtains the contour and detail information of the target object through neural network processing to obtain accurate segmentation of the target object, which is beneficial to improving the judgment accuracy of the reliability results of the depth image corresponding to the target object.
[0101] In this embodiment, further, based on the object instance segmentation image and the regional image features corresponding to each target object, the object posture estimation based on the depth image or the object posture estimation based on the color image can be realized, as follows:
[0102] E3, obtaining the image features of each target object according to the object instance segmentation image and the regional image features corresponding to each target object;
[0103] E4: For each target object, the pose of the object is estimated based on the image features of the target object.
[0104] Combination Figure 3 As shown, the object instance segmentation image and the corresponding regional image features are processed, such as multiplying them, to obtain the image features corresponding to the target object.
[0105] For each target object, the image features of the target object can be used as basic data for object posture estimation, such as basic data for object posture estimation based on two-dimensional key points, or basic data for object posture estimation based on three-dimensional key points. Acquisition of a two-dimensional key point offset image or a three-dimensional key point offset image of the estimated object based on the image features of the target object is beneficial to improving the accuracy of the two-dimensional key point offset image and the three-dimensional key point offset image.
[0106] In an optional implementation, step S120 estimates the posture of the object based on the three-dimensional key points, which can be performed in the following manner, as shown in the flowchart. Figure 4 As shown, including:
[0107] S1301, extracting image features based on the color image, or based on the color image and the depth image;
[0108] S1302, extracting point cloud features according to the depth image;
[0109] S1303, fusing the image features and the point cloud features to obtain fused features;
[0110] S1304: Estimate the posture of the object based on the fused features.
[0111] The solution provided in this embodiment can extract image features based on color images, and can also extract image features based on color images and depth images, wherein the extraction of image features can be performed through image feature extraction networks, image feature extraction algorithms, etc. Since image features and point cloud features will be fused subsequently, when image features are extracted only based on color images, the fused features also include data such as depth data and contour data of the depth image.
[0112] An optional embodiment of the present application provides a solution as follows Figure 2 As shown, in this scheme, feature extraction is performed based on color images and depth images, and image feature extraction is performed through an image feature extraction network, that is, the color image and the depth image are input into the image feature extraction network, and the output of the feature extraction network includes an image feature vector for each pixel.
[0113] Optionally, point cloud features are extracted based on the depth image, firstly, the depth image is converted into a point cloud to obtain point cloud data corresponding to the depth image, and point cloud features are extracted based on the point cloud data to obtain point cloud features corresponding to the depth image. Point cloud feature extraction can be performed by a point cloud feature extraction network, the input of which is point cloud data, and the output includes a point cloud feature vector for each three-dimensional point.
[0114] The obtained image features are fused with the point cloud features to obtain fused features. Specifically, for each pixel point contained in the image, for each pixel point, the image feature vector and the point cloud feature vector of the pixel point are fused to obtain the fused features. The fusion operation may be dense fusion.
[0115] Compared with the method of using color images alone to make reliability judgments, the reliability judgment of depth images based on fusion features is more accurate and is conducive to improving the efficiency of obtaining the reliability of depth images.
[0116] Based on the fusion features, the two-dimensional key point offset image and the three-dimensional key point offset image are obtained, and the two-dimensional key point offset image and the three-dimensional key point offset image are obtained through deep learning. Then, the posture of the object is estimated according to the reliability judgment result of the depth image. If the depth image is reliable, the posture of the object is estimated according to the three-dimensional key point offset image; if the depth image is unreliable, the posture of the object is estimated according to the two-dimensional key point offset image.
[0117] Combination Figure 3 As shown, the target object is regionalized using image features to obtain instance segmentation image features of the target object, point cloud features are extracted from the depth image to obtain instance segmentation point cloud features, the segmented image features and point cloud features are fused, the 3D key point offset image is estimated based on the fused features, and then the object posture is estimated based on the 3D key point offset image.
[0118] Among them, the three channels in the three-dimensional key point offset image represent the deviation vector from the three-dimensional coordinates of the object point corresponding to the pixel point to the three-dimensional coordinates of a preset reference key point on the object. Specifically, the pose estimation of the object based on the three-dimensional key point offset image can be performed in the following manner: a straight line is determined from any three-dimensional pixel point of the target object to pass through the preset reference key point of the object, and for all N pixels in the pixel area of the target object, according to their corresponding three-dimensional key point offset images, N straight lines passing through the preset reference key point can be obtained, and then the three-dimensional coordinates of the preset reference key point of the target object can be determined by voting. After the three-dimensional coordinates of M key points on the target object are obtained in this way, the three-dimensional rotation and three-dimensional translation transformations between the two groups of three-dimensional points can be obtained by least squares fitting of M corresponding points on the three-dimensional model of the object to obtain the 6D pose of the object.
[0119] Combination Figure 3 As shown, image features are extracted based on the color image and the depth image, and the reliability of the depth image is judged based on the image features to obtain the reliability judgment result of the depth image. If the depth image is reliable, the image features are fused with the point cloud features extracted based on the depth image, and the three-dimensional key points are acquired based on the fused features.
[0120] If there are multiple target objects, the image features and point cloud features corresponding to each target object are fused, and the three-dimensional key points are acquired based on the fused features.
[0121] Specifically, the fused features are obtained by fusing the image features with the point cloud features, and the process of estimating the pose of the object based on the fused image is as follows: use a tool for drawing three-dimensional images, such as the MPL tool, to estimate the three-dimensional key point offset image according to the fused features of the target object, and estimate the pose of the object based on the three-dimensional key point offset image.
[0122] Accordingly, the posture estimation of the object based on the two-dimensional key points provided in step S130 can be performed in the following manner, including:
[0123] F1, extract image features based on color image and depth image;
[0124] F2, estimate the object’s posture based on image features.
[0125] Since this scheme is a scheme for estimating the posture of an object based on a color image when the depth image is unreliable, this scheme obtains the contour data of the object in the depth image based on the depth image, and estimates the posture of the object based on the contour data of the object in the color image and the depth image. Since the basis for object posture estimation does not include the depth data of the object, the posture of the object is estimated based on image features, which is also called object posture estimation based on two-dimensional key points.
[0126] Optionally, performing pose estimation of an object based on image features may be performed in the following manner, including: using a convolutional neural network to estimate a two-dimensional key point offset image of the target object based on the image features of the target object, and performing pose estimation of the object based on the two-dimensional key point offset image.
[0127] Optionally, the target object is regionalized according to the image features to obtain regional image features, instance segmentation image features are obtained based on the regional image features, and then the instance segmentation image features are used as input of a convolutional neural network to estimate a two-dimensional key point offset image, and the pose of the object is estimated based on the two-dimensional key point offset image.
[0128] Among them, the two channels of each pixel in the two-dimensional key point offset image represent the deviation vector from the two-dimensional coordinates of the object point corresponding to the pixel to the two-dimensional coordinates of the preset reference point on the target object. Specifically, the pose estimation of the object based on the two-dimensional key point offset image can be performed in the following manner: a straight line is determined from any pixel point of the target object to pass through the preset reference point, and for all N pixels in the pixel area of the target object, according to their corresponding two-dimensional key point offset images, N straight lines passing through the preset reference point can be obtained, and then the two-dimensional coordinates of the preset reference point on the target object can be determined by voting. After obtaining the two-dimensional coordinates of M preset reference points on the target object in this way, the corresponding points of these M key points on the three-dimensional model can be used to calculate the three-dimensional rotation and three-dimensional translation transformation between the camera coordinate system and the object coordinate system according to the PnP (Perspective-n-Point) algorithm to obtain the 6D pose of the object.
[0129] In an optional embodiment, the fusion of the image features and the point cloud features provided in S1303 to obtain the fusion features can be achieved in the following manner, as shown in the flowchart. Figure 5 As shown, including:
[0130] S13031, obtaining an object instance segmentation image according to the image features;
[0131] S13032, segmenting the image according to the object instance to obtain image features and point cloud features of each target object;
[0132] S13033, fusing the image features and point cloud features of the same target object to obtain fused features of each target object.
[0133] Combination Figure 3 As shown, regional processing is performed based on image features to obtain regional image features, and object instance segmentation is performed based on the regional image features to obtain object instance segmentation images. Then, instance segmentation image features, i.e., image features of the target object, are obtained based on the regional image features and the object instance segmentation image. Instance segmentation geometric features, i.e., point cloud features of the target object, are obtained based on the instance segmentation image and the point cloud features.
[0134] Then the image features and point cloud features corresponding to the same target object are fused to obtain the fused features of each target object.
[0135] This embodiment provides another solution for obtaining fusion features of the target object. This solution can be applicable to the situation where there are one or more target objects in the color image. Compared with the solution of directly extracting features from the color image and the depth image and performing feature fusion, the solution provided in this embodiment is processed by regionalization and instance segmentation to refine the image features and point cloud features of each target object. After two feature extractions, it is conducive to obtaining more accurate fusion features.
[0136] On this basis, the pose estimation of the object is performed according to the fusion features, including:
[0137] According to the fused features of each target object, the posture of each target object is estimated.
[0138] Among them, the posture of each target object is estimated according to the fusion features of the target object. For each target object, the following operations are performed: the three-dimensional key point offset image is estimated based on the fusion features of the target object, and then the posture of the target object is estimated based on the three-dimensional key point offset image, and the posture of each target object is obtained in turn.
[0139] When there are multiple target objects, regionalization processing and object instance segmentation are performed on each target object to obtain accurate fusion features of each target object, and then the posture of each target object is estimated based on the fusion features, which is conducive to obtaining the accurate posture of all target objects in the color image.
[0140] In the case where there are multiple target objects in the image, the present application also provides an optional embodiment, which performs posture estimation based on the object relationship of the target objects in the following manner, including:
[0141] H1, based on the color image and the depth image, obtain the first appearance feature of each target object and the geometric relationship feature between the target objects;
[0142] H2, for each target object, determine the second appearance feature of the target object based on the first appearance feature of the target object, the first appearance features of other target objects other than the target object, and the geometric relationship features between the target object and other target objects other than the target object.
[0143] Combination Figure 6 As shown, feature extraction and fusion are performed based on the color image and the depth image, and the first appearance feature of each target object and the geometric relationship feature between each target object are obtained based on the fusion feature. Among them, the object instance segmentation image is obtained based on the dense feature, and the geometric relationship feature between each target object is obtained based on the object instance segmentation image.
[0144] Among them, obtaining the first appearance features of each target object based on the fused features can be achieved in the following ways, including: based on the fused features, using a voting method to predict the center point coordinates of the target object to which each pixel point in the input data belongs and the appearance features of the target object; for the predicted center point coordinates, using a clustering method to obtain a number of target objects and the pixels of each target object; for each target object, by fusing the features of the pixel points belonging to the target object, the first appearance features of each target object are obtained.
[0145] Combination Figure 7 As shown, image feature extraction is performed based on the color image and the depth image, regional processing is performed based on the image features to obtain regional image features, and the first appearance features of each target object and the geometric relationship features between each target object are obtained based on the regional image features. Among them, object detection is performed based on the regional image features, and the geometric relationship features between each target object are obtained based on the object detection results.
[0146] For each target object, a second appearance feature of the target object is determined based on the first appearance feature of the target object, the first appearance features of other target objects other than the target object, and the geometric relationship features between the target object and other target objects other than the target object, so as to estimate the posture of the object based on the second appearance feature.
[0147] Optionally, the pose estimation of the object can be performed based on the fusion features, which can be obtained by:
[0148] The posture of each target object is estimated according to the fused feature and the second appearance feature of each target object.
[0149] Combination Figure 6 As shown, a two-dimensional key point offset image and a three-dimensional key point offset image are estimated based on the relational feature fusion of the fusion feature of the target object and the second appearance feature corresponding to the target object. If the depth image is unreliable, the object posture is estimated based on the two-dimensional key point offset image; if the depth image is reliable, the object posture is estimated based on the three-dimensional key point offset image.
[0150] The embodiment of the present application also provides another method for estimating the posture of an object based on fusion features. Figure 7 As shown, regionalization processing is performed based on image features to obtain regional image features, object instance segmentation is performed based on the regional image features to obtain an object instance segmentation image, segmented image features are obtained based on the regional image features and the object instance segmentation image, feature fusion is performed based on the segmented image features and the segmented point cloud features, and the second appearance features of the target object, three-dimensional key point offset image is estimated based on the fused features, and object posture estimation is performed based on the estimated three-dimensional key point offset image.
[0151] Optionally, the pose estimation of the object based on the image features can be obtained by the following methods, including:
[0152] The posture of each target object is estimated according to the image features and the second appearance features of each target object.
[0153] Specifically, image features are extracted based on the color image and the depth image, regional image features are obtained based on the image features, segmented image features are obtained based on the regional image features, two-dimensional key point offset image estimation is performed based on the image features of the target object and the corresponding second appearance features, and the posture of the target object is estimated based on the estimated two-dimensional key point offset image.
[0154] The solution provided in this embodiment introduces the relationship features of the target object, determines the second appearance features of the target object by extracting and fusing the geometric relationship information between multiple objects and adding the first appearance features of different target objects themselves, thereby improving the accuracy and robustness of the object posture estimation method in the case of object occlusion, field of view truncation, and small objects.
[0155] In an optional embodiment of the present application, based on the color image and the depth image, the first appearance feature of each target object and the geometric relationship feature between the target objects can be obtained by at least one of the following schemes (Scheme 1: steps P1-P5, Scheme 2: steps Q1-Q4):
[0156] P1, extracting image features based on color image, or based on color image and depth image;
[0157] P2, extract point cloud data based on the depth image;
[0158] P3, fuses the image features and point cloud features to obtain fused features;
[0159] P4, based on the fusion features, obtain the first appearance features of each target object and the object instance segmentation image;
[0160] P5, segment the image based on object instances to obtain the geometric relationship features between target objects.
[0161] This method determines the appearance features of each target object and the object instance segmentation image based on the fusion features, and obtains the geometric relationship features between the target objects based on the object instance segmentation image. This scheme determines the appearance features of the target objects and the geometric relationship features between the target objects based on a one-time feature extraction, thereby realizing the rapid acquisition of the appearance features of the target objects and the geometric relationship features between the target objects.
[0162] Q1, extract image features based on color image and depth image;
[0163] Q2, obtaining the regional image features corresponding to the image regions of each target object according to the image features;
[0164] Q3, obtaining a first appearance feature of each target object and a corresponding object detection result based on the regional image features corresponding to the image regions of each target object;
[0165] Q4, based on the object detection results of each target object, obtain the geometric relationship features of each target object.
[0166] The solution provided in this embodiment is based on two feature extractions, one for image feature extraction and one for regional image feature extraction, to obtain the first appearance features of the target object and the geometric relationship features of each target object. The two feature extractions are beneficial to improving the accuracy of the first appearance features and the geometric relationship features of each target object.
[0167] The aforementioned color image and depth image are obtained based on a frame of color depth image in the video. In an optional embodiment, the reliability of the depth image is determined based on the color image and depth image of the object, which can be achieved in the following manner:
[0168] If the video frame corresponding to the color image and the depth image is an initial frame of the video, the reliability of the depth image is determined based on the color image and the depth image.
[0169] The color image and the depth image are extracted from the video frame including the color information and the depth information. If the video frame corresponding to the color image and the depth image is the initial frame of the video, that is, before this frame, there is no reference information about the posture of the video frame, then the color image and the depth image are used to determine the reliability of the depth image. Optionally, the scheme for determining the reliability of the depth image provided in any of the above embodiments can be used to perform reliability judgment, and then the posture estimation scheme for the object provided in any of the above embodiments can be used to perform object posture estimation to obtain the object posture estimation result.
[0170] The object posture estimation method provided in the embodiment of the present application has a flow chart as follows: Figure 8 As shown, after obtaining the object's posture estimation result, it also includes:
[0171] S140, storing the object posture estimation result into a posture result list; the posture result includes each target object and the object posture corresponding to each target object.
[0172] The obtained object posture estimation results are stored in a pre-constructed posture result list so that posture estimation can be performed later based on the already obtained object posture. That is, the object posture in the posture result list is used as a reference to simplify the subsequent acquisition of the object posture, improve the real-time acquisition of the object posture, and reduce the amount of data processing to obtain the object posture.
[0173] On this basis, if the video frame corresponding to the color image and the depth image is not an initial frame, an object posture estimation method provided by an optional embodiment of the present application further includes:
[0174] S150, obtaining motion parameters of a video acquisition device corresponding to a video frame, and determining a posture result corresponding to the video frame based on the motion parameters and an object posture result of an initial frame corresponding to the video frame;
[0175] S160: According to the gesture result corresponding to the video frame, the gesture result of the initial frame corresponding to the video frame in the gesture result list is updated.
[0176] In the embodiment of the present application, a video may have multiple video frame sequences, each of which has only one initial frame, and the other video frames are non-initial frames. If the video frame corresponding to the color image and the depth image is not the initial frame, that is, at least one target object and the corresponding object posture are stored in the posture result list, that is, at least the posture result list corresponding to the initial frame is stored in the posture result list, and the motion parameters of the video acquisition device of the video frame of the non-initial frame corresponding to the color image and the depth image are obtained. The motion parameter of the video acquisition device can be a relative motion parameter, and the relative motion parameter is the motion parameter of the video frame of the non-initial frame relative to the acquisition device of the initial frame. Based on the relative motion parameter and the object posture result of the initial frame of the sequence where the video frame is located, the object posture result of the video frame is determined. According to this method, all target objects and corresponding object posture results in the video frame are obtained, and the posture result composed of the target object of the video frame and its corresponding object posture is stored in the posture result list, and the posture result corresponding to the video frame is used as the posture result of the initial frame corresponding to the video frame, and the posture result list is updated.
[0177] The solution provided in this embodiment uses the corresponding posture result in the initial frame and the motion parameters of the video acquisition device to determine the object posture estimation of the non-initial frame in the video frame sequence where the initial frame is located, and uses the difference between the previous and next video frames to estimate the object posture, which is conducive to improving the efficiency of obtaining the object posture and improving the real-time performance of obtaining the object posture. Moreover, since the data difference between adjacent video frames is small, the posture result list is updated using the posture result corresponding to the video frame of the non-initial frame, which is conducive to reducing the data processing amount of the next video frame.
[0178] In an optional embodiment of the present application, the following methods are also provided to determine whether a video frame in a video is an initial frame, including:
[0179] For any video frame, if the video frame is the first frame of the video, then the video frame is the initial frame of the video;
[0180] For any video frame, object detection is performed on the video frame to confirm the target object and the object posture in the video frame; it is detected whether the target object or the object posture in the video frame appears for the first time, and whether the video frame is an initial frame; when the target object or the object posture in the video frame appears for the first time, the video frame is the initial frame of the video.
[0181] Combination Fig.10 As shown, Fig.10 This is a flowchart for object pose estimation based on the initial frame in the video. Fig.10 It is only an optional method and does not constitute a limitation on the embodiment of the present application. A video frame is read. The video frame is an RGBD image, that is, the image includes color information and depth information. The color image and depth image of the object can be obtained based on the RGBD image. It is determined whether the video frame is an initial frame. If so, object instance segmentation is performed based on a convolutional neural network, and the 6-DOF posture estimation of the object is performed based on the object instance segmentation image (corresponding to Fig.10 ), obtain the target object and the corresponding object posture, and update the posture result list based on the target object and the corresponding object posture (corresponding to Fig.10 ). If the video frame is not the initial frame, based on the 6-DOF camera motion parameters of the current video frame, for each object instance in the pose result list, the motion parameters of the camera in the current video frame are used to transform the pose of the object instance in the initial frame to the current video frame, and obtain the object pose of the non-initial frame.
[0182] For the selection of the initial frame, you can combine Fig.11 As shown, Fig.11 A schematic diagram showing three initial frame selection methods provided by an embodiment is shown. For any video frame, if the video frame is the first frame of the video, then the video frame is the initial frame of the video. If a video has only one initial frame, then the initial frame is the first frame of the video, corresponding to Fig.11 The initial frame selection method in (1).
[0183] For a video, the video can also be divided into multiple video frame sequences according to a preset period, and the first frame in each video frame sequence is the initial frame of the video frame sequence, corresponding to the initial frame selection method (2).
[0184] The present application also provides an initial frame selection method (3), which divides the video into at least one video frame sequence, such as Fig.11 The initial frame selection controller disclosed in the initial frame selection method (3) determines that the video frame where the new object and the new posture are located is the initial frame of the video frame sequence through monitoring of the new object and the new posture.
[0185] Specifically, object detection is performed on each video frame in the video, and the target object and object posture in the video frame are determined based on the detection result, by detecting whether the target object or object posture in the video frame appears for the first time. For example, the target object and object posture corresponding to the current video frame can be compared with the target object and object posture in the posture result list to determine whether the target object and object posture corresponding to the current video frame appear for the first time. When at least one of the target object or object posture in the video frame appears for the first time, it indicates that the video frame is the initial frame of the video or video clip.
[0186] The embodiment of the present application determines whether a video frame is an initial frame by the above method, and performs targeted processing according to the judgment result. If the video frame is not an initial frame, the object posture of the initial frame and the motion parameters corresponding to the video frame are used to estimate the object posture, so as to improve the real-time performance of the object posture estimation. The present embodiment provides a method for determining whether a video frame is an initial frame, and determines whether a video frame is an initial frame according to whether a new object or a new posture appears in the video frame. The judgment method and the judgment result are more suitable for the object posture estimation scheme provided by the present application. Compared with the scheme of setting only the first frame of a video as the initial frame, the scheme of performing the initial frame according to the new object or the new posture is conducive to reducing the amount of data processing for object posture estimation of non-initial frames of video frames according to the initial frame. Fig.12 A flowchart for determining whether a video frame is an initial frame based on object detection is provided in an optional embodiment of the present application, combined with Fig.12 As shown, for the acquired new RGBD video frame, the target object and object posture in the video frame are obtained through object detection (corresponding to Fig.12 ), combined with the pose result list (corresponding to Fig.12 The current object instance and its 6-DOF posture list in the table) determines whether there is a new target object based on the comparison result ( Fig.12 If the object instance in the video is an object instance in the video, or a new posture, if so, set the current video frame as the initial frame, if not, obtain a new video frame.
[0187] After determining whether the video frame is an initial frame by any of the above methods, if it is an initial frame, the depth data reliability of the depth image is judged, and the object posture estimation is performed according to the judgment result. If it is not an initial frame, the object posture estimation is performed using the scheme provided in steps S160 to S170.
[0188] In an optional embodiment, for a video frame, detecting whether a target object or an object posture in a video frame appears for the first time may also be implemented in the following manner, which may include:
[0189] K1, obtain the image bounding box of each target object in the video frame;
[0190] K2, matching the image bounding box of each target object with the image bounding box corresponding to each target object in the pose result list;
[0191] K3, when there is a matching target object in the posture result list, compare the first point cloud data of the image bounding box corresponding to each target object in the video frame with the second point cloud data of each target object in the previous video frame of the video frame to determine whether there is a difference between the first point cloud data and the second point cloud data; if so, determine that the object posture corresponding to the target object appears for the first time;
[0192] K4, when there is no matching target object in the object posture result list, determine that the target object appears for the first time.
[0193] The present embodiment provides a method for determining whether a target object or an object posture appears for the first time, by comparing an image bounding box of the target object in a video frame with an image bounding box of the target object in a posture result list. If there is no image bounding box in the posture result list that matches the image bounding box of the target object in the video frame, it indicates that the target object in the video frame appears for the first time; if there is an image bounding box in the posture result list that matches the image bounding box corresponding to the video frame, it indicates that the target object in the video frame does not appear for the first time.
[0194] For video frames in which the target object does not appear for the first time, the object posture of the target object is further judged, and the first point cloud data in the image bounding box corresponding to the target object in the video frame is compared with the second point cloud data corresponding to the target object in the previous video frame of the video frame. If the first point cloud data is the same as the second point cloud data, it indicates that the posture of the target object in the video frame is not the first time it appears; if the first point cloud data is different from the second point cloud data, that is, there is a data difference, it indicates that the object posture of the target object appears for the first time.
[0195] Combination Fig.13 As shown, Fig.13 A flowchart for confirming a new object or a new posture provided by an embodiment of the present application, for each object image bounding box obtained by object detection, a posture result list (corresponding to Fig.13For each object in the current object instance list in the current video frame, compare whether the image bounding boxes of the two objects match and overlap, and whether a matching object can be found in the image bounding boxes of the two objects. If not, it indicates that a new target object is detected. If not, continue to detect whether the point cloud in the bounding box has changed compared with the previous frame of the current video frame. If so, it indicates that a new object posture is detected. If not, it indicates that no new object and new posture are detected.
[0196] The present application provides a scheme for determining whether a target object or an object posture in a video frame appears for the first time. If any of the target object or the object posture is determined to appear for the first time, the video frame is determined to be an initial frame, and the object posture is estimated based on the scheme provided in steps S110 to S140, wherein the target object is determined based on an image bounding box, without the need for more detailed data. It is possible to quickly and accurately determine whether the target object is a new object, and by comparing whether the point cloud data in the image bounding box is consistent, it is possible to accurately determine whether the object posture appears for the first time.
[0197] In order to better understand and illustrate the solutions provided by the embodiments of the present application, the following further describes the various optional solutions provided by the embodiments of the present application in combination with several specific examples. Figure 3 , Figure 4 , Figure 6 , Figure 7 and Fig. 9 The module box corresponding to the sharp corner represents the operation steps, and the module box corresponding to the rounded corner represents the processing results.
[0198] Example 1
[0199] Combination Fig. 9 As shown, Fig. 9 A flowchart for object posture estimation based on color image and depth image is provided for an embodiment. Specifically, a color image and a depth image of the same object are obtained, and image features are extracted from the color image and the depth image using an image convolution network to obtain image features. Point cloud features are extracted from the point cloud data of the depth image using a point cloud feature extraction network to obtain point cloud features, and then the image features and the point cloud features are color-deep fused to obtain fused features. The following two branches are operated based on the fused features: Branch 1, depth reliability prediction is performed based on the fused features to obtain a depth reliability image; semantic segmentation is performed on the object based on the fused features to obtain an object semantic segmentation image, and object center offset estimation is performed to obtain an object center image; in this branch, instance segmentation is performed on the image based on the object semantic segmentation image and the object center image to obtain an object instance segmentation image; and then the depth data reliability is judged based on the object instance segmentation image and the depth reliability image. Branch 2, relational feature fusion is performed on the object instance segmentation image and the fused features to obtain the fused features.
[0200] Determine whether to use two-dimensional key points or three-dimensional key points for object pose estimation based on the judgment result of the reliability of the depth data. Specifically, if the depth data is unreliable, the two-dimensional key point offset estimation is performed based on the fused features to obtain a two-dimensional key point offset image, and two-dimensional key point voting is performed based on the two-dimensional key point offset image to obtain the two-dimensional coordinates of the preset reference key points, and 6D pose estimation is performed based on the three-dimensional coordinates of the preset reference key points of the ePnP algorithm; if the depth data is reliable, three-dimensional key point voting is performed based on the three-dimensional key point offset image, and 6D pose estimation is performed through least squares fitting.
[0201] Example 2
[0202] Combination Figure 2 As shown, a color image and a depth image of the same object are obtained, and an image feature extraction network is used to extract image features from the color image and the depth image to obtain image features. A point cloud feature extraction network is used to extract point cloud features from the point cloud data of the depth image to obtain point cloud features. The obtained image features and point cloud features are fused. The fusion operation here can be dense fusion. The following three branch operations are performed based on the fused features. First, a depth reliability prediction is performed on the image based on the fused features to obtain a depth reliability image. Second, semantic segmentation is performed on the object based on the fused features to obtain an object semantic segmentation image, and object center offset estimation is performed to obtain an object center image. Third, a two-dimensional key point offset image and a three-dimensional key point offset image are obtained.
[0203] Then, object instance segmentation is performed based on the object semantic segmentation image and the object center image to obtain the object instance segmentation image, and the reliability of the depth data in the depth image is judged based on the object instance segmentation image and the depth reliability image. According to the judgment result, it is determined whether to use two-dimensional key points or three-dimensional key points to estimate the object posture. Specifically, if the depth data is unreliable, two-dimensional key point voting is performed based on the two-dimensional key point offset image to obtain the two-dimensional coordinates of the preset reference key points, and 6D posture estimation is performed based on the three-dimensional coordinates of the preset reference key points according to the PnP algorithm; if the depth data is reliable, three-dimensional key point voting is performed based on the three-dimensional key point offset image to obtain the three-dimensional coordinates of the reference key points, and 6D posture estimation is performed by least squares fitting.
[0204] Example 3
[0205] Combination Figure 3As shown in the figure, a color image and a depth image of the same object are obtained, and the image feature extraction network is used to extract image features from the color image and the depth image to obtain image features. The image features are input into the region proposal network for regional processing, and then after ROI pooling, the regional image features are obtained. The following processing is performed based on the regional image features: first, the feature processing of the fully connected layer is performed to obtain a depth reliability image; second, the convolutional (CNN) segmentation network is used to perform object instance segmentation to obtain an object instance segmentation image.
[0206] The reliability of the depth data is judged based on the object instance segmentation image and the depth reliability image, and then the instance segmentation image features are obtained based on the object instance segmentation image and the regional image features. When the depth data is unreliable, the instance segmentation image features are used as the input for object pose estimation using two-dimensional key points. Specifically, the instance segmentation image features are used as the input of the convolutional neural network, and the object two-dimensional key point offset image is output, and then the object pose is estimated based on the two-dimensional key point offset image.
[0207] When the depth data corresponding to the depth image is reliable, the instance segmentation image features are fused with the instance segmentation geometric features obtained by the power feature extraction network, and the fused features are used to perform: object pose estimation based on 3D key points. Specifically, the 3D drawing tool MPL is used to estimate the 3D key point offset image of the object based on the fused features, and then the object pose estimation is performed based on the 3D key point offset image. Figure 2 compared to, Figure 3 The scheme shown uses a two-level feature extraction network for feature extraction, and the extracted features are more refined, which is conducive to obtaining accurate object posture estimation results.
[0208] Example 4
[0209] Combination Figure 6 As shown, an image feature extraction network is used to extract image features from color images and depth images to obtain image features, a point cloud feature extraction network is used to extract point cloud features from point cloud data converted from the depth image to obtain point cloud features, the image features are fused with the point cloud features to obtain fused features, object semantic segmentation and object instance segmentation are performed based on the fused features to obtain an object instance segmentation image.
[0210] and Figure 2 Compared with the scheme shown, Figure 6 The scheme shown in the figure adds an object relationship branch. The weighted appearance features obtained by the object relationship branch will affect the object pose estimation based on 3D key points, but have no effect on the object pose estimation based on 2D key points. Figure 2The scheme shown is the same, so the following is an example of object posture estimation based on three-dimensional key points. In the object relationship branch, the appearance features of a single target object and the appearance features of multiple target objects are determined based on the fusion features, that is, the first appearance features of each target object are obtained, and the geometric relationship features between the multiple target objects are obtained based on the object instance segmentation image, and the first appearance features of each target object and the geometric relationship features between the target objects are added to obtain the relationship features of the target object, and the obtained relationship features of the target object are added pixel by pixel with the first appearance features corresponding to itself to obtain the weighted appearance features of the target object, that is, the second appearance features of the target object.
[0211] Performing posture regression processing based on the second appearance feature, such as using a regression network to process the second appearance feature to obtain the object posture predicted by the object relationship branch.
[0212] Based on the second appearance feature and the fusion feature, relational feature fusion is performed to estimate the 3D key point offset to obtain a 3D key point offset image. When the depth image is reliable, the 3D key point offset image is used to estimate the object pose to obtain the object pose based on the 3D key points.
[0213] After obtaining the object posture predicted by the object relationship branch and the object posture based on the three-dimensional key points, object posture selection is performed, and the selection criteria are adjusted according to the actual situation to realize the 6-degree-of-freedom posture estimation of the object and obtain the 6D posture of the object.
[0214] This solution incorporates object relationship analysis and combines the relationship features between different objects to perform object posture estimation, which is beneficial to improving the accuracy of the image area corresponding to each target object. Accurate image area division is beneficial to improving the accuracy of object posture estimation.
[0215] Example 5
[0216] Combination Figure 7 As shown in the figure, the image feature extraction network is used to extract image features from color images and depth images to obtain image features. The image features are input into the region proposal network for regional processing, and then after ROI pooling, the regional image features are obtained.
[0217] and Figure 3 Compared with the scheme shown, Figure 7The scheme shown adds an object relationship branch, that is, the first appearance features of each target object and the geometric relationship features between each target object are performed based on the regional image features, wherein the first appearance features of each target object correspond to the image appearance features of a single target object and the image appearance features of multiple target objects. The geometric relationship features between the target objects in the object relationship branch are obtained based on the object detection results, and the object detection results are obtained by inputting the obtained regional image features into the segmentation convolutional network for processing. Then, the image appearance features of a single target object, the image appearance features of multiple target objects, and the geometric relationship features between multiple target objects are added with attention to obtain the relationship features of the target objects. Then, the relationship features and appearance features of the target objects are added pixel by pixel to obtain the weighted appearance features, that is, the second appearance features of the target objects, and the object posture is estimated based on the second appearance features to obtain the object posture predicted by the object relationship branch.
[0218] and Figure 3 Compared with the scheme shown, Figure 7 The scheme shown in the figure adds an object relationship branch. The weighted appearance features obtained by the object relationship branch will affect the object pose estimation based on 3D key points, but have no effect on the object pose estimation based on 2D key points. Figure 3 The schemes shown are the same, so the following is explained by taking the object posture estimation based on three-dimensional key points as an example.
[0219] After ROI pooling, the regional image features are obtained, and the regional image features are input into the segmentation convolutional network to obtain the object instance segmentation image. Then, the segmentation image features, i.e., the instance segmentation image features, are obtained based on the regional image features and the object instance segmentation image. Point cloud data conversion and point cloud feature extraction are performed on the depth image to obtain the segmentation geometric features, i.e., the instance segmentation geometric features. The obtained segmentation image features, segmentation geometric features, and weighted appearance features are fused to perform 3D key point offset estimation to obtain a 3D key point offset image. The object posture is estimated based on the 3D key point offset image to obtain the object posture based on the 3D key point.
[0220] The posture selection is performed based on the object posture predicted by the object relationship branch and the object posture based on the three-dimensional key points, and the 6-degree-of-freedom posture of the object, that is, the 6D posture, is determined according to the set selection conditions.
[0221] This solution adds object relationship analysis on the basis of secondary feature extraction. Based on more refined features, it is helpful to improve the precise division of each object area, and then combine the relationship features between different objects to estimate the object posture, further improving the accuracy of the image area corresponding to each target object and the accuracy of object posture estimation.
[0222] Corresponding to the object posture estimation method provided in the present application, the present application embodiment also provides an object posture estimation device 100, whose structural schematic diagram is as follows: Fig.14 As shown in , the object posture estimation device 100 includes: an image reliability determination module 110, a first and a second posture estimation module 120, which are specifically used for:
[0223] An image reliability determination module 110, configured to determine the reliability of the depth image based on the color image and the depth image of the object;
[0224] A posture estimation module 120, for performing posture estimation of the object based on three-dimensional key points when the depth image is reliable;
[0225] The posture estimation module 120 is also used to estimate the posture of the object based on the two-dimensional key points when the depth image is unreliable.
[0226] Optionally, the image reliability determination module 110 includes at least one of the following:
[0227] A reliability determination unit based on fusion features is used to extract image features based on a color image, or based on a color image and a depth image; extract point cloud features based on the depth image; fuse the image features and the point cloud features to obtain fusion features; and determine the reliability of the depth image based on the fusion features;
[0228] The image feature determination reliability unit is used to extract image features based on the color image and the depth image; and determine the reliability of the depth image according to the image features.
[0229] Optionally, the fusion feature determines the reliability unit, specifically for:
[0230] Based on the fusion features, obtaining an object instance segmentation image and a depth reliability image;
[0231] The reliability of the depth image corresponding to each target object in the color image is determined according to the object instance segmentation image and the depth reliability image.
[0232] Optionally, the image feature determination reliability unit is specifically used to:
[0233] According to the image features, an object instance segmentation image and a depth reliability image are obtained;
[0234] The reliability of the depth image corresponding to each target object in the color image is determined according to the object instance segmentation image and the depth reliability image.
[0235] Optionally, the image feature determination reliability unit is further configured to:
[0236] According to the image features, regional image features of the image regions corresponding to each target object are obtained;
[0237] For each target object, a depth reliability image of the target object is determined based on the regional image features corresponding to the target object; and an object instance segmentation image is obtained based on the regional image features corresponding to each target object.
[0238] Optionally, the object posture estimation device 100 further includes:
[0239] A geometric relationship feature acquisition module is used to obtain a first appearance feature of each target object and a geometric relationship feature between each target object based on the color image and the depth image;
[0240] A second appearance feature determination module is used to determine, for each target object, a second appearance feature of the target object based on the first appearance feature of the target object, the first appearance features of other target objects other than the target object, and the geometric relationship features between the target object and the other target objects other than the target object;
[0241] The first posture estimation unit is further used to: perform posture estimation of each target object according to the fusion feature and the second appearance feature of each target object;
[0242] The second posture estimation unit is further used to estimate the posture of each target object according to the image feature and the second appearance feature of each target object.
[0243] Optionally, obtaining a geometric relationship feature module includes at least one of the following:
[0244] Extract image features based on the color image, or based on the color image and the depth image; extract point cloud features based on the depth image; fuse the image features and the point cloud features to obtain fused features; obtain the first appearance features of each target object and the object instance segmentation image based on the fused features; obtain the geometric relationship features between the target objects based on the object instance segmentation image;
[0245] Based on the color image and the depth image, image features are extracted; based on the image features, regional image features corresponding to the image area of each target object are obtained; based on the regional image features corresponding to the image area of each target object, first appearance features of each target object and corresponding object detection results are obtained; based on the object detection results of each target object, geometric relationship features of each target object are obtained.
[0246] Optionally, the object posture estimation device further includes: an initial frame module;
[0247] The initial frame module is used to determine whether a video frame is an initial frame by detecting whether a target object or an object posture in the video frame appears for the first time.
[0248] Optionally, the initial frame module is further used to:
[0249] Obtain the image bounding box of each target object in the video frame;
[0250] Match the image bounding box of each target object with the image bounding box corresponding to each target object in the pose result list;
[0251] When there is a matching target object in the posture result list, the first point cloud data of the image bounding box corresponding to each target object in the video frame and the second point cloud data corresponding to each target object in the previous video frame of the video frame are compared to determine whether there is a difference between the first point cloud data and the second point cloud data; if so, determine that the object posture corresponding to the target object appears for the first time;
[0252] When there is no matching target object in the object pose result list, it is determined that the target object appears for the first time.
[0253] The present application also provides an electronic device, which includes a memory and a processor; wherein a computer program is stored in the memory; and the processor is used to execute the method provided in any optional embodiment of the present application when running the computer program.
[0254] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the method provided in any optional embodiment of the present application is executed.
[0255] As an alternative, Fig.15 A schematic diagram of the structure of an electronic device applicable to the embodiment of the present application is shown. Fig.15 As shown, the electronic device 4000 may include a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, such as through a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.
[0256] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.
[0257] The bus 4002 may include a path to transmit information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.15 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0258] The memory 4003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.
[0259] The memory 4003 is used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the application code (computer program) stored in the memory 4003 to implement the content shown in any of the above method embodiments.
[0260] In the embodiments provided in the present application, the above-mentioned object posture estimation method performed by the electronic device can be performed using an artificial intelligence model.
[0261] According to an embodiment of the present application, the method performed in an electronic device can obtain output data for identifying an image or image features in an image by using image data or video data as input data of an artificial intelligence model. The artificial intelligence model can be obtained by training. Here, "obtained by training" means obtaining a predefined operation rule or artificial intelligence model configured to perform a desired feature (or purpose) by training a basic artificial intelligence model with multiple training data through a training algorithm. The artificial intelligence model may include multiple neural network layers. Each of the multiple neural network layers includes multiple weight values, and the neural network calculation is performed by calculating between the calculation result of the previous layer and the multiple weight values.
[0262] Visual understanding is a technology for recognizing and processing things like human vision, and includes, for example, object recognition, object tracking, image retrieval, human recognition, scene recognition, 3D reconstruction / localization, or image enhancement.
[0263] The object posture estimation device provided in the present application can implement at least one module among multiple modules through an AI model. Functions associated with AI can be executed through a non-volatile memory, a volatile memory, and a processor.
[0264] The processor may include one or more processors. In this case, the one or more processors may be general-purpose processors (e.g., central processing unit (CPU), application processor (AP), etc.), or pure graphics processing units (e.g., graphics processing unit (GPU), visual processing unit (VPU), and / or AI-specific processors (e.g., neural processing unit (NPU)).
[0265] The one or more processors control the processing of input data according to predefined operating rules or artificial intelligence (AI) models stored in non-volatile memory and volatile memory. The predefined operating rules or artificial intelligence models are provided by training or learning.
[0266] Here, providing by learning means obtaining a predefined operating rule or an AI model with desired characteristics by applying a learning algorithm to a plurality of learning data. The learning can be performed in the device itself in which the AI according to the embodiment is executed, and / or can be implemented by a separate server / system.
[0267] The AI model can be composed of multiple neural network layers. Each layer has multiple weight values, and the calculation of one layer is performed by the calculation result of the previous layer and the multiple weights of the current layer. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q networks.
[0268] A learning algorithm is a method of using a plurality of learning data to train a predetermined target device (e.g., a robot) to enable, allow, or control the target device to make a determination or prediction. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.
[0269] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.
[0270] The above descriptions are only some embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A method performed by an electronic device, It is characterized in that include: Determining the reliability of the depth image based on the color image and the depth image of the object; When the depth image is reliable, performing posture estimation of the object based on three-dimensional key points; When the depth image is unreliable, the posture of the object is estimated based on two-dimensional key points.
2. The method according to claim 1, It is characterized in that Determining the reliability of the depth image according to the color image and the depth image of the object includes at least one of the following: Extracting image features based on the color image, or based on the color image and the depth image; extracting point cloud features based on the depth image; fusing the image features and the point cloud features to obtain fused features; and determining the reliability of the depth image based on the fused features; Based on the color image and the depth image, image features are extracted; and the reliability of the depth image is determined according to the image features.
3. The method according to claim 2, It is characterized in that The determining the reliability of the depth image based on the fusion feature includes: Based on the fusion features, obtaining an object instance segmentation image and a depth reliability image; The reliability of the depth image corresponding to each target object in the color image is determined according to the object instance segmentation image and the depth reliability image.
4. The method according to claim 2, It is characterized in that The determining the reliability of the depth image according to the image features comprises: According to the image features, obtaining an object instance segmentation image and a depth reliability image; The reliability of the depth image corresponding to each target object in the color image is determined according to the object instance segmentation image and the depth reliability image.
5. The method according to claim 4, It is characterized in that The step of obtaining an object instance segmentation image and a depth reliability image according to the image features includes: According to the image features, obtaining regional image features of the image regions corresponding to each target object; For each target object, a depth reliability image of the target object is determined based on the regional image features corresponding to the target object; and an object instance segmentation image is obtained based on the regional image features corresponding to each target object.
6. The method according to claim 1, It is characterized in that Also includes: Based on the color image and the depth image, obtaining a first appearance feature of each target object and a geometric relationship feature between the target objects; For each target object, a second appearance feature of the target object is determined based on the first appearance feature of the target object, the first appearance features of other target objects other than the target object, and the geometric relationship features between the target object and other target objects other than the target object.
7. The method according to claim 6, It is characterized in that Estimating the posture of the object based on the three-dimensional key points includes: The posture of each target object is estimated according to the fusion feature and the second appearance feature of each target object.
8. The method according to claim 6, It is characterized in that Estimating the posture of the object based on the three-dimensional key points includes: The posture of each target object is estimated according to the image feature and the second appearance feature of each target object.
9. The method according to claim 6, It is characterized in that The obtaining, based on the color image and the depth image, of the first appearance feature of each target object and the geometric relationship feature between the target objects includes at least one of the following: Extracting image features based on the color image, or based on the color image and the depth image; extracting point cloud features based on the depth image; fusing the image features and the point cloud features to obtain fused features; Based on the fusion features, the first appearance features of each target object and the object instance segmentation image are obtained; based on the object instance segmentation image, the geometric relationship features between the target objects are obtained; Extracting image features based on the color image and the depth image; and obtaining regional image features corresponding to the image regions of each target object according to the image features; Based on the regional image features corresponding to the image regions of the target objects, obtaining the first appearance features of the target objects and the corresponding object detection results; Based on the object detection results of each target object, the geometric relationship features of each target object are obtained.
10. The method according to claim 1, It is characterized in that Also includes: Whether the video frame is an initial frame is determined by detecting whether the target object or the object posture in the video frame appears for the first time.
11. The method according to claim 10, It is characterized in that The detecting whether the target object or the object posture in the video frame appears for the first time includes: Obtain the image bounding box of each target object in the video frame; Match the image bounding box of each target object with the image bounding box corresponding to each target object in the pose result list; When there is a matching target object in the posture result list, compare the first point cloud data of the image bounding box corresponding to each target object in the video frame with the second point cloud data of each target object in the previous video frame of the video frame, and if there is a difference between the first point cloud data and the second point cloud data, determine that the object posture corresponding to the target object appears for the first time; or When there is no matching target object in the gesture result list, it is determined that the target object appears for the first time.
12. The method according to claim 10, It is characterized in that Also includes: In response to the video frame being a non-initial frame, acquiring motion parameters corresponding to the video frame, and determining a posture result corresponding to the video frame according to the motion parameters and an object posture result of an initial frame corresponding to the video frame; and According to the posture result corresponding to the video frame, the object posture result of the initial frame corresponding to the video frame in the posture result list is updated.
13. An electronic device, It is characterized in that including memory and processor; The memory stores a computer program; The processor is configured to execute the method according to any one of claims 1 to 12 when running the computer program.
14. A computer-readable storage medium, It is characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 12 is executed.
