A real-time 6D pose estimation method and computer-readable storage medium
By using point-by-point feature splicing of preset key points and attention mechanisms in 6D pose estimation, combined with the global-local conversion module and point-by-point attention voting network, the problems of median pose estimation in the existing technology are solved. , realizes high-precision and efficient pose estimation.
Patent Information
- Application Number
- CN202111669299.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-31
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-12-31
AI Technical Summary
The existing 6D pose estimation method is not very accurate when dealing with low-texture objects or scenes with large illumination changes. The RGB-D image-based method is difficult to achieve real-time performance due to the complexity of information processing, which limits its practical application. deployment.
By selecting preset key points from the three-dimensional model of the target object, using RGB-D images to extract texture features and geometric features, combining attention mechanism and point-by-point feature stitching, the coordinates of preset key points in the camera coordinate system are predicted. Perform pose fitting calculations, use global-local conversion module to reduce background interference, and design a point-by-point attention voting network to integrate irregular point cloud information.
It realizes high-precision real-time 6D pose estimation without the need for complex feature extraction networks, improves the real-time and pose estimation accuracy of the network, reduces the introduction of external interference information, and is suitable for practical applications.
Smart Images

Figure CN114359377B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of posture estimation, and in particular to a real-time 6D posture estimation method and a computer-readable storage medium. Background Art
[0002] With the rapid development of robotics, the application of dexterous robot manipulation in industrial automation and people's daily life is becoming more and more extensive. Real-time 6D pose estimation of the target object is the primary prerequisite for the robot to achieve dexterous manipulation, which directly affects the stability and consistency of the robot's manipulation of the object.
[0003] The 6D pose of the target object represents the 6 degrees of freedom of the object in Cartesian space, including 3 degrees of freedom of displacement (translation) and 3 degrees of freedom of spatial rotation. Pose is a relative concept, which is used to represent the relative rigid transformation relationship between two coordinate systems. Since the relative relationship between the camera and the robot is known in practical applications, the 6D pose estimation in the present invention refers to the transformation relationship between the coordinate system of the object to be estimated and the camera coordinate system.
[0004] Common object 6D pose estimation methods can be divided into three types: based on RGB images, based on point clouds, and based on RGB-D images. The pose estimation algorithm based on RGB images mainly relies on the use of artificially designed feature extraction operators or convolutional neural networks to extract the texture features of the object from the image, and estimates the 6D pose information of the object to be measured based on these features. However, this method is highly dependent on the texture information of the object in the image, and it is difficult to ensure good accuracy when facing low-texture objects or scenes with large lighting changes. The pose estimation method based on point clouds mainly relies on laser radar to obtain point cloud information, but laser radar is expensive and has harsh installation conditions, making it difficult to deploy on a robotic arm; at the same time, because the point cloud information of the radar is relatively sparse, it is not conducive to the pose estimation of small objects. With the development of technology in recent years, the price of depth cameras has gradually decreased, which has also made the pose estimation method based on RGB-D gain attention. RBG-D images contain both texture information in RGB images and geometric information in depth images, which is conducive to further improving the accuracy of pose estimation algorithms, but how to quickly process this large amount of information has also become an issue that cannot be ignored. In the past, pose estimation algorithms based on RGB-D images (Frustum-P, PVN3D, FFB6D, etc.) were directly used for pose estimation without distinction after extracting features. Since there is no reasonable classification and utilization of features, such networks often require very complex feature extraction networks to ensure the speed of pose estimation, which also greatly limits their real-time performance. Therefore, such methods are difficult to be deployed in practical applications.
[0005] The disclosure of the above background technology content is only used to assist in understanding the concept and technical solution of the present invention. It does not necessarily belong to the prior art of this patent application. In the absence of clear evidence that the above content has been disclosed on the filing date of this patent application, the above background technology should not be used to evaluate the novelty and creativity of the present application. Summary of the invention
[0006] In order to solve the existing problems, the present invention provides a real-time 6D pose estimation method and a computer-readable storage medium.
[0007] In order to solve the above problems, the technical solution adopted by the present invention is as follows:
[0008] A real-time 6D pose estimation method, characterized in that it includes the following steps: S1: selecting preset key points from a three-dimensional model of a target object and obtaining the coordinates of the preset key points in the three-dimensional model based on the coordinate system of the three-dimensional model origin; S2: predicting the coordinates of the preset key points of the target object in the actual scene based on the camera coordinate system through a network; S3: fitting the coordinates of the preset key points in the three-dimensional model origin coordinate system and the coordinates in the camera coordinate system to obtain the current pose of the target object.
[0009] Preferably, predicting the coordinates of the preset key points in the actual scene based on the camera coordinate system through the network includes the following steps: S21: acquiring an RGB-D image of the target object and converting the global information in the RGB-D image into local information, wherein the local information is the local RGB image of the target object and the local point cloud information of the target object; S22: performing texture feature extraction on the local RGB image of the target object, performing geometric feature extraction on the local point cloud information of the target object, performing point-by-point feature splicing using the point-by-point dense correspondence obtained by dense fusion, and identifying the point cloud belonging to the target object based on the point-by-point features; S23: predicting the offset vector between each visible point and the preset key point using the point-by-point features of the target object; evaluating the mapping relationship between each visible point and each preset key point using the point-by-point features and the offset vector, and screening and fusing the predicted information of each visible point according to the evaluation results to obtain the coordinates of the preset key point in the camera coordinate system.
[0010] Preferably, the RGB-D image of the target object is intercepted according to the predicted bounding box to obtain the local information; the center point of the local point cloud of the target object after interception is set as the origin, a point cloud center point coordinate system is established with the same direction as the camera coordinate system, and the coordinates of the target object in the three-dimensional model origin coordinate system are transferred to the point cloud center point coordinate system, and the offset from the camera to the point cloud center point in the camera coordinate system is eliminated to highlight the local information.
[0011] Preferably, a method combining ResNet and upsampling is used to extract texture features from the local RGB image of the target object; and PointNet is used to extract geometric features from the local point cloud information of the target object.
[0012] Preferably, a maximum pooling method is used to obtain global features and the global features and the point-by-point features of each visible point are used as input to predict the semantic label scores of all the visible points and identify the point cloud belonging to the target object; softmax cross entropy is used as the loss function during training:
[0013]
[0014] Where N is the number of points, c n It represents the prediction confidence of each category after the nth visible point is processed by softmax, l n is with c n The corresponding true value, n is the number of visible points, and the category with the highest score of each label is selected as the prediction result.
[0015] Preferably, a multilayer perceptron and ResNet are combined to design an offset vector prediction network to predict the point-by-point offset vector between each of the visible points and each of the predicted key points;
[0016] For a set of visible points belonging to the target object and a set of said preset key points The offset vector prediction network is based on the point-by-point feature f n As input output the offset vector
[0017] During the training phase, L vec Supervision Prediction:
[0018]
[0019] Among them, M is the number of preset key points, is the true value of the offset vector;
[0020] Each of the visual points predicts the preset key point coordinates The coordinates of the visible point and the offset vector are obtained by the following formula:
[0021]
[0022] Preferably, the mapping relationship between each visual point and each preset key point is evaluated by using the point-by-point feature and the offset vector, and each visual point is screened by point-by-point attention voting according to the evaluation result; L is added during the training process. infl As the loss function for influence prediction, its expression is as follows:
[0023]
[0024] The point-by-point predictions are classified according to the distance between the predictions and the ground truth in the geometric space; p is considered as a positive term, and is higher than the threshold T n is regarded as a negative term, and the rest are ignored; the ignored visible points do not participate in the loss calculation; when the point is a positive term, ζ p =1, when it is a negative term n =1, in other cases ζ p and n is 0.
[0025] Preferably, a predetermined proportion of the visual points are selected and voting weights are allocated according to the influence of the visual points. The voting process can be expressed by the following formula:
[0026]
[0027] in, Indicates the visible point vpc n For the preset key point kpc m The influence of , δ(x) represents a weight distribution function, when When it is less than the predetermined ratio, δ(x)=0; when When the value is greater than or equal to the predetermined ratio, the output result of δ(x) is the output after the influence of the predetermined ratio is normalized using the softmax function;
[0028] The preset key point voting network is supervised and trained using the following loss function:
[0029]
[0030] in, Indicates the real value of the corresponding preset key point coordinates.
[0031] Preferably, the predicted second coordinate of the preset key point is matched with the first coordinate in the existing model using the Kabsch algorithm to calculate the posture of the 6D object composed of rotation R and translation T.
[0032] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above methods are implemented.
[0033] The beneficial effects of the present invention are: providing a real-time 6D pose estimation method and a computer-readable storage medium. By introducing the idea of an attention mechanism, before performing the final pose estimation, the mapping relationship between each visible point and each preset key point is evaluated based on the point-by-point features extracted by the front network, and each visible point is allowed to vote on the coordinates of each key point based on the evaluation result, so as to effectively integrate all visible point features. The present invention does not require a complex feature extraction network to achieve accurate pose estimation, so that the network has excellent real-time performance and achieves the highest pose estimation accuracy at the same speed.
[0034] Furthermore, the present invention alleviates the interference of background and global bias on pose estimation by means of a global-local conversion module, and further strengthens the local information of the object itself.
[0035] Furthermore, the present invention proposes a new method for integrating irregular point cloud information. With the help of the method of the present invention, the rational integration of irregular point cloud information can be achieved, which is conducive to strengthening the network to screen out more excellent features and reduce the introduction of external interference information. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 It is a schematic diagram of a real-time 6D pose estimation method in an embodiment of the present invention.
[0037] Figure 2 It is a schematic diagram of selecting preset key points in a three-dimensional model in an embodiment of the present invention.
[0038] Figure 3 It is a schematic diagram of a method for predicting the coordinates of preset key points in an actual scene based on a camera coordinate system through a network in an embodiment of the present invention.
[0039] Figure 4 It is a schematic diagram of a process of predicting the coordinates of the preset key points in the actual scene based on the camera coordinate system through a network in an embodiment of the present invention.
[0040] Figure 5 It is a semantic segmentation network structure diagram in an embodiment of the present invention.
[0041] Figure 6Schematic diagram of the structure of the point-by-point attention voting module in an embodiment of the present invention. DETAILED DESCRIPTION
[0042] In order to make the technical problems, technical solutions and beneficial effects to be solved by the embodiments of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0043] It should be noted that when a component is referred to as being "fixed to" or "disposed on" another component, it can be directly on the other component or indirectly on the other component. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element. In addition, connection can be used for both fixing and circuit connection.
[0044] It should be understood that the orientation or position relationship indicated by terms such as "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside" and "outside" are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.
[0045] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.
[0046] like Figure 1 As shown, the present invention provides a real-time 6D pose estimation method, comprising the following steps:
[0047] S1: Selecting a preset key point from a three-dimensional model of a target object and obtaining the coordinates of the preset key point in the three-dimensional model based on a three-dimensional model origin coordinate system;
[0048] S2: predicting the coordinates of the preset key points of the target object in the actual scene based on the camera coordinate system through the network;
[0049] S3: Fitting and calculating the coordinates of the preset key points in the three-dimensional model origin coordinate system and the camera coordinate system to obtain the current posture of the target object.
[0050] In one embodiment of the present invention, the three-dimensional model is generally obtained by laser scanning the object or building the three-dimensional model of the object by oneself.
[0051] like Figure 2 , which is a schematic diagram of selecting preset key points in a three-dimensional model in an embodiment of the present invention, wherein the center point and eight corner points of a rectangular box in an existing three-dimensional model are selected as preset key points.
[0052] like Figure 3 As shown, predicting the coordinates of the preset key points in the actual scene based on the camera coordinate system through the network includes the following steps:
[0053] S21: Acquire an RGB-D image of the target object and convert global information in the RGB-D image into local information, where the local information is a local RGB image of the target object and local point cloud information of the target object;
[0054] S22: extracting texture features from the local RGB image of the target object, extracting geometric features from the local point cloud information of the target object, performing point-by-point feature splicing using the point-by-point dense correspondence obtained by dense fusion, and identifying the point cloud belonging to the target object based on the point-by-point features;
[0055] S23: Predict the offset vector between each visible point and the preset key point using the point-by-point features of the target object; evaluate the mapping relationship between each visible point and each preset key point using the point-by-point features and the offset vector, and filter and fuse the prediction information of each visible point according to the evaluation results to obtain the coordinates of the preset key point in the camera coordinate system.
[0056] like Figure 4 , is a schematic diagram of a process of predicting the coordinates of the preset key points in the actual scene based on the camera coordinate system through a network in an embodiment of the present invention. The network used in the present invention mainly includes a global-local conversion module, a point-by-point feature extraction module and a key point voting module.
[0057] Most of the pose estimation methods in the prior art directly input global information for pose estimation, but there is a large amount of background and global bias information in the global image, and the information of the target object and the relative image relationship of the internal structure are extracted in detail. The present invention adopts a global to local conversion module to eliminate background errors and strengthen the relative relationship inside the target object, thereby realizing the local information enhancement of the target object.
[0058] For irregular point cloud features, the existing method is often to directly use a MaxPool to merge them indiscriminately. However, due to reasons such as lighting, occlusion, sensor noise, etc., the feature expression capabilities of each point in the point cloud are not the same. Direct unified processing is likely to introduce external noise and cause the network to ignore influential features. The present invention proposes a point-by-point attention voting network, which first uses an attention mechanism to evaluate point-by-point features, and then fuses point-by-point prediction information by voting based on the evaluation results. The present invention extends the idea of the attention mechanism to the voting process, thereby improving the network's ability to integrate irregular point cloud features, thereby achieving reasonable integration of point cloud information.
[0059] In order to ensure the accuracy of pose estimation, the network needs to use a complex feature extraction network to obtain the deep features of the object. Therefore, it is very difficult to ensure the real-time performance and high precision of the network at the same time. The present invention uses a point-by-point attention voting network to improve the feature utilization rate of the overall network, so that the network can achieve high-precision pose estimation without an overly complex feature extraction network, which greatly improves the real-time performance of the network.
[0060] In the present invention, the global information in the input RGB-D image is first converted into local information through the target detection network and point cloud coordinate transformation to reduce background interference and computational waste.
[0061] In the present invention, only the RGB image is used as input, and the estimated target is roughly located using the target detection network. The main purpose of this step is to reduce the interference of the cluttered environment on the estimation and avoid unnecessary background calculations. This process does not need to excessively pursue the prediction accuracy of the detection frame. Therefore, compared to networks such as R-FCN, Faster R-CNN, and Mask R-CNN, which have high prediction accuracy but long time consumption, the method of the present invention uses YOLOv3 with good real-time performance to handle this task to avoid excessive computational overhead.
[0062] After that, the present invention performs image interception on the RGB image and the depth image according to the predicted bounding box to obtain local information. It can be understood that the bounding box is predicted and output by the target inspection network in the global-local conversion module. The target detection network is an existing mature algorithm. It can predict the approximate position of the object (i.e., generate a two-dimensional detection box) using only the RGB image, and convert the depth image into point cloud information at the same time. In order to avoid the weakening of local features by excessive global offset in the point cloud, the present invention sets the center point of the local point cloud of the target object after interception as the origin, establishes a point cloud center point coordinate system in the same direction as the camera coordinate system, and transfers the coordinates of the point cloud collected by the depth camera of the target object from the camera coordinate system to the point cloud center point coordinate system, eliminating the offset from the camera to the point cloud center point in the original camera coordinate system to highlight the local information. Since the conversion relationship between the point cloud center coordinate system and the camera coordinate system can be obtained by the detection box coordinates and the camera internal parameters, the predicted object's pose in the camera coordinate system can be converted into the predicted object's pose in the point cloud center point coordinate system.
[0063] This processing step can reduce the interference of the background and global bias information in the original information on the next step of extracting the local features of the object.
[0064] The intercepted RGB image and the converted point cloud information are then input into the point-by-point feature extraction module. Since the RGB image is two-dimensional information and the point cloud is three-dimensional information, and the spatial dimensions are different, the present invention adopts a method of extracting features first and then fusing them. The present invention uses a method combining ResNet with upsampling to extract texture features from the local RGB image of the target object, which not only expands the receptive domain, but also facilitates the subsequent splicing of point pixel correspondences; at the same time, PointNet is used to extract geometric features from the local point cloud information of the target object; and then the point-by-point dense correspondence obtained by dense fusion is used to perform point-by-point feature splicing.
[0065] In order to further reduce the interference of background on posture estimation, the present invention also performs semantic segmentation on all extracted visible point information.
[0066] like Figure 5 , which is a semantic segmentation network structure diagram in an embodiment of the present invention. After feature extraction, a feature vector with a length of 512 is extracted from each point in the point cloud and each pixel in the RGB image.
[0067] Each point cloud corresponds to a pixel in the original depth image. The same position in the depth image and RGB image represents the color and distance of the same point respectively. Based on this correspondence, the two vectors of the same point are concatenated together to form a feature vector with a length of 1024.
[0068] In the semantic segmentation task, the present invention uses the maximum pooling method to obtain global features and uses the global features and the point-by-point features of each visible point as input to predict the semantic label scores of all the visible points and identify the point cloud belonging to the target object;
[0069] During training, softmax cross entropy is used as the loss function:
[0070]
[0071] Where N is the number of points, c n It represents the prediction confidence of each category after the nth visible point is processed by softmax, l n is with c n The corresponding true value, n is the number of visible points, and the category with the highest score of each label is selected as the prediction result.
[0072] Since each part of an object is affected by different external factors such as occlusion, illumination change, and camera error in practical applications, not all visible points can be used to predict key points. To solve this problem, the present invention designs the point-by-point attention voting module.
[0073] like Figure 6 As shown, it is a schematic diagram of the structure of the point-by-point attention voting module in an embodiment of the present invention. The present invention uses the point-by-point features after segmentation to realize voting on the coordinates of key points, which consists of three parts: vector prediction, impact evaluation and voting.
[0074] Firstly, a bias vector prediction network is designed by combining a multi-layer perceptron and ResNet to predict the point-by-point bias vector between each of the visible points and each of the predicted key points;
[0075] For a set of visible points belonging to the target object and a set of said preset key points The offset vector prediction network is based on the point-by-point feature f n As input, output is the offset vector
[0076] During the training phase, L vec Supervision Prediction:
[0077]
[0078] Among them, M is the number of preset key points, is the true value of the offset vector;
[0079] Each of the visual points predicts the preset key point coordinates The coordinates of the visible point and the offset vector are obtained by the following formula:
[0080]
[0081] In one embodiment of the present invention, in order to reasonably integrate the prediction results of each visible point, the mapping relationship between each visible point and each preset key point is evaluated using the point-by-point features and the offset vector, and each visible point is screened by point-by-point attention voting based on the evaluation results; based on the point-by-point features after segmentation, the present invention adds the visible point coordinates and the predicted offset vector as input, and outputs a three-dimensional vector to represent the influence of each visible point feature.
[0082] It can be understood that the mapping relationship is the potential association between the features of each visual point and the key point. The stronger the mapping relationship, the greater the probability that the result predicted by the point is relatively accurate.
[0083] In order to subjectively guide the network to select a point set with good key point prediction ability, L is added during the training process. infl As the loss function for influence prediction, its expression is as follows:
[0084]
[0085] The point-by-point predictions are classified according to the distance between the predictions and the ground truth in the geometric space; p is considered as a positive term, and is higher than the threshold T n is regarded as a negative term, and the rest are ignored; the ignored visible points do not participate in the loss calculation; when the point is a positive term, ζ p =1, when it is a negative term n =1, in other cases ζ p and n is 0.
[0086] A predetermined proportion of the visual points are selected and voting weights are allocated according to the influence of the visual points. The voting process can be expressed as follows:
[0087]
[0088] in, Indicates the visible point vpc n For the preset key point kpc m The influence of , δ(x) represents a weight distribution function, when When it is less than the predetermined ratio, δ(x)=0; when When the value is greater than or equal to the predetermined ratio, the output result of δ(x) is the output after the influence of the predetermined ratio is normalized using the softmax function.
[0089] In one embodiment of the present invention, a large number of experiments have shown that selecting the top one-third of influential points is conducive to achieving a more robust prediction, which can reduce the interference of points with poor mapping relationships on the prediction without excessively omitting points with good mapping relationships.
[0090] The preset key point voting network is supervised and trained using the following loss function:
[0091]
[0092] in, Indicates the real value of the corresponding preset key point coordinates.
[0093] Finally, the Kabsch algorithm is used to match the predicted second coordinates of the preset key points with the first coordinates in the existing model, and the posture of the 6D object composed of rotation R and translation T is calculated. Kabsch is an existing algorithm, and its main function is to estimate the current posture of the object by fitting the key point coordinates predicted by the network and the key points preset in the model.
[0094] The present invention successfully achieves the highest pose estimation accuracy at the same speed through reasonable network design.
[0095] Furthermore, the present invention alleviates the interference of background and global bias on pose estimation by means of a global-local conversion module, and further strengthens the local information of the object itself.
[0096] Furthermore, the present invention proposes a new method for integrating irregular point cloud information. With the help of the method of the present invention, the rational integration of irregular point cloud information can be achieved, which is conducive to strengthening the network to screen out more excellent features and reduce the introduction of external interference information.
[0097] In a specific embodiment, the detection accuracy comparison of the present invention and several other methods under the Linemod dataset is shown in Table 1. PVNet, which only uses RGB images for posture estimation, has an advantage in speed, but it is difficult to achieve sufficient accuracy. Although PVN3D has higher prediction accuracy than our method on many objects, the method of the present invention is more than 4 times faster than it and can be more flexibly deployed in many practical applications. Compared with other methods, this method has higher accuracy in predicting objects with sparse textures and small volumes (such as Ape and Duck). It can be seen that this is inseparable from the fact that point-by-point attention voting can better highlight the geometric features of target objects in the network.
[0098] Table 1 Pose estimation accuracy of the present invention and other methods on Linemod dataset
[0099]
[0100] An embodiment of the present application also provides a control device, including a processor and a storage medium for storing a computer program; wherein the processor is used to execute at least the method described above when executing the computer program.
[0101] An embodiment of the present application also provides a storage medium for storing a computer program, which at least performs the method described above when executed.
[0102] An embodiment of the present application further provides a processor, which executes a computer program and at least performs the method described above.
[0103] The storage medium may be implemented by any type of volatile or non-volatile storage device, or a combination thereof. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory may be a disk memory or a tape memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAMEnhanced Synchronous Dynamic Random Access Memory), synchronous link dynamic random access memory (SLDRAM), direct memory bus random access memory (DRRAM). The storage media described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memory.
[0104] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.
[0105] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0106] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.
[0107] A person skilled in the art can understand that: all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.
[0108] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.
[0109] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.
[0110] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.
[0111] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.
[0112] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.
Claims
1. A real-time 6D pose estimation method, It is characterized in that The steps include: S1: Selecting a preset key point from a three-dimensional model of a target object and obtaining the coordinates of the preset key point in the three-dimensional model based on a three-dimensional model origin coordinate system; S2: predicting the coordinates of the preset key points of the target object in the actual scene based on the camera coordinate system through the network; S3: fitting and calculating the coordinates of the preset key point in the three-dimensional model origin coordinate system and the coordinates of the preset key point in the camera coordinate system to obtain the current position and posture of the target object; Predicting the coordinates of the preset key points in the actual scene based on the camera coordinate system through the network includes the following steps: S21: Acquire an RGB-D image of the target object and convert global information in the RGB-D image into local information, where the local information is a local RGB image of the target object and local point cloud information of the target object; S22: extracting texture features from the local RGB image of the target object, extracting geometric features from the local point cloud information of the target object, performing point-by-point feature splicing using the point-by-point dense correspondence obtained by dense fusion, and identifying the point cloud belonging to the target object based on the point-by-point features; S23: Predict the offset vector between each visible point and the preset key point using the point-by-point features of the target object; evaluate the mapping relationship between each visible point and each preset key point using the point-by-point features and the offset vector, and filter and fuse the prediction information of each visible point according to the evaluation results to obtain the coordinates of the preset key point in the camera coordinate system.
2. The real-time 6D pose estimation method according to claim 1, It is characterized in that Intercepting the RGB-D image of the target object according to the predicted bounding box to obtain the local information; The center point of the local point cloud of the target object after interception is set as the origin, a point cloud center point coordinate system is established with the same direction as the camera coordinate system, and the coordinates of the target object in the three-dimensional model origin coordinate system are transferred to the point cloud center point coordinate system, eliminating the offset from the camera to the point cloud center point in the camera coordinate system to highlight the local information.
3. The real-time 6D pose estimation method according to claim 2, It is characterized in that Using a method combining ResNet and upsampling to extract texture features from a local RGB image of the target object; PointNet is used to extract geometric features from the local point cloud information of the target object.
4. The real-time 6D pose estimation method according to claim 3, It is characterized in that Using a maximum pooling method to obtain global features and taking the global features and the point-by-point features of each visible point as input, predicting the semantic label scores of all the visible points, and identifying the point cloud belonging to the target object; During training, softmax cross entropy is used as the loss function: Where N is the number of points, c n It represents the prediction confidence of each category after the nth visible point is processed by softmax, l n is with c n The corresponding true value, n is the number of visible points, and the category with the highest score of each label is selected as the prediction result.
5. The real-time 6D pose estimation method according to claim 4, It is characterized in that Combining a multi-layer perceptron and ResNet to design an offset vector prediction network to predict the point-by-point offset vector between each of the visible points and each of the predicted key points; For a set of visible points belonging to the target object and a set of said preset key points The offset vector prediction network is based on the point-by-point feature f n As input output the offset vector During the training phase, L vec Supervision Prediction: Among them, M is the number of preset key points, is the true value of the offset vector; Each of the visual points predicts the preset key point coordinates The coordinates of the visible point and the offset vector are obtained by the following formula:
6. The real-time 6D pose estimation method according to claim 5, It is characterized in that Using the point-by-point features and the offset vectors, the mapping relationship between each of the visual points and each of the preset key points is evaluated, and point-by-point attention voting is performed on each of the visual points for screening according to the evaluation results; Add L during training infl As the loss function for influence prediction, its expression is as follows: The point-by-point predictions are classified according to the distance between the predictions and the ground truth in the geometric space; p is considered as a positive term, and is higher than the threshold T n are regarded as negative items, and the rest are ignored items; the ignored visual points do not participate in the loss calculation; When the point is positive p =1, when it is a negative term n =1, in other cases ζ p and n is 0.
7. The real-time 6D pose estimation method according to claim 6, It is characterized in that A predetermined proportion of the visual points are selected and voting weights are allocated according to the influence of the visual points. The voting process can be expressed as follows: in, Indicates the visible point vpc n For the preset key point kpc m The influence of , δ(x) represents a weight distribution function, when When it is less than the predetermined ratio, δ(x)=0; when When the value is greater than or equal to the predetermined ratio, the output result of δ(x) is the output after the influence of the predetermined ratio is normalized using the softmax function; The preset key point voting network is supervised and trained using the following loss function: in, Indicates the real value of the corresponding preset key point coordinates.
8. The real-time 6D pose estimation method according to claim 7, It is characterized in that The predicted second coordinates of the preset key points are matched with the first coordinates in the existing model using the Kabsch algorithm, and the posture of the 6D object consisting of rotation R and translation T is calculated.
9. A computer-readable storage medium storing a computer program. It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.
