Pose prediction method and device, nonvolatile storage medium and electronic equipment

Abstract

The invention discloses a pose prediction method and device, a nonvolatile storage medium and electronic equipment. The method comprises the following steps: acquiring a target image, target point cloud data corresponding to the target image and shape prior data corresponding to the target image; determining a first global feature according to the target image and the target point cloud data, the first global feature including a feature of the target image and a feature of the target point cloud data; second global features are determined according to the shape prior data, and the second global features comprise geometric features of the shape prior data; and determining a target deformation field and a target matching matrix of the target object according to the first global feature and the second global feature, and determining a pose prediction result of the target object according to the target deformation field and the target matching matrix. The technical problem that the motion state of an object cannot be accurately predicted due to the fact that a pose prediction method in the related technology cannot guarantee the accuracy of a prediction result in a complex environment is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of electric digital data processing, in particular to a pose prediction method and device, a nonvolatile storage medium and an electronic device. BACKGROUND

[0002] In the related art, when estimating the pose of an object, the required computing resources are too high, and the time consumption is too long. Moreover, in an environment with visual occlusion, light changes or complex background, it is difficult to ensure the accuracy of the prediction result.

[0003] To address the above problems, no effective solutions have been proposed so far. SUMMARY

[0004] Embodiments of the present application provide a pose prediction method and device, a nonvolatile storage medium and an electronic device to at least solve the technical problem that the pose prediction method in the related art cannot accurately predict the motion state of an object due to the inability to ensure the accuracy of the prediction result in a complex environment.

[0005] According to an aspect of an embodiment of the present application, a pose prediction method is provided, comprising: obtaining a target image, target point cloud data corresponding to the target image, and shape prior data corresponding to the target image, wherein the target image is an image of a target object collected, the target point cloud data is point cloud data of the target object, and the shape prior data includes shape information of the target object; determining a first global feature according to the target image and the target point cloud data, wherein the first global feature includes features of the target image and features of the target point cloud data; determining a second global feature according to the shape prior data, wherein the second global feature includes geometric features of the shape prior data; determining a target deformation field and a target matching matrix of the target object according to the first global feature and the second global feature, and determining a pose prediction result of the target object according to the target deformation field and the target matching matrix, wherein the target deformation field is used to adjust the shape of the target object, the target matching matrix is used to determine the position and direction of the target object, and the pose prediction result includes pose information and size information of the target object.

[0006] Optionally, the feature extraction network comprises a first feature extraction branch, a second feature extraction branch and a feature processing branch, the first feature extraction branch is configured to extract image features of the target image, the second feature extraction branch is configured to extract point cloud features of the target point cloud data, and the feature processing branch is configured to process intermediate processing results of the first feature extraction branch and the second feature extraction branch; the first global feature is determined according to the target image and the target point cloud data, comprising: processing the target image and the target point cloud data by the feature extraction network to obtain a first output result of the first feature extraction branch, a second output result of the second feature extraction branch and a third output result of the feature processing branch; and the first global feature is obtained by fusing the first output result, the second output result and the third output result.

[0007] Optionally, the first feature extraction branch and the feature processing branch, and the second feature extraction branch and the feature processing branch are connected through an attention mechanism module, wherein the intermediate processing result of the first feature extraction branch and the intermediate processing result of the second feature extraction branch are input into corresponding convolution layers of the feature processing branch after being processed by the attention mechanism module.

[0008] Optionally, the target deformation field and the target matching matrix of the target object are determined according to the first global feature and the second global feature, comprising: processing the first global feature and the second global feature by a feature exchange network to obtain an initial first fusion feature and an initial second fusion feature; processing the first fusion feature and the second fusion feature by a model recursive recovery network to obtain an initial deformation field and an initial matching matrix; determining a recursive feature according to the initial deformation field and shape prior data, and iteratively processing the recursive feature, the first global feature and the second global feature to obtain the target deformation field and the target matching matrix, wherein in each iteration, the first global feature, the second global feature and the recursive feature are processed by the feature exchange network to obtain the first fusion feature and the second fusion feature, and the deformation field and the matching matrix are determined according to the first fusion feature and the second fusion feature, and the recursive feature in the next iteration is determined according to the deformation field and the shape prior data in the case where the iteration termination condition is not reached.

[0009] Optionally, determining the recursive features based on the initial deformation field and shape prior data, and iteratively processing the recursive features, the first global features and the second global features includes: a first step, processing the recursive features, the first global features and the second global features through a feature exchange network to obtain the first fusion features and the second fusion features; a second step, determining the deformation field and the matching matrix based on the first fusion features and the second fusion features; a third step, determining whether the iteration termination condition is met, and jumping to the fifth step if it is determined that the iteration termination condition is met, otherwise jumping to the fourth step, wherein the iteration termination condition includes that the number of iterations reaches a preset number threshold, or the deformation field residual corresponding to the deformation field and the matching matrix residual corresponding to the matching matrix meet preset requirements; a fourth step, determining the recursive features again based on the deformation field and the shape prior data, and jumping to the first step; a fifth step, determining the deformation field obtained in this round of iteration as the target deformation field, and determining the matching matrix obtained in this round of iteration as the target matching matrix.

[0010] Optionally, determining the pose prediction result of the target object based on the target deformation field and the target matching matrix includes: establishing a normalized object coordinate space model based on the target deformation field and shape prior data; and determining the pose prediction result based on the normalized object coordinate space model and the target point cloud data.

[0011] Optionally, determining the pose prediction result of the target object based on the target deformation field and the target matching matrix includes: determining the coordinate values ​​of each point in the target point cloud data in the normalized object space coordinates based on the target point cloud data and the target matching matrix; determining the similarity transformation parameters based on the coordinate values ​​of each point in the point cloud data and the normalized object coordinate space model, and determining the pose prediction result based on the similarity transformation parameters.

[0012] According to another aspect of an embodiment of the present application, a posture prediction device is also provided, including: a first processing module for acquiring a target image, target point cloud data corresponding to the target image, and shape prior data corresponding to the target image, wherein the target image is a captured image of the target object, the target point cloud data is point cloud data of the target object, and the shape prior data includes shape information of the target object; a second processing module for determining a first global feature based on the target image and the target point cloud data, wherein the first global feature includes features of the target image and features of the target point cloud data; a third processing module for determining a second global feature based on the shape prior data, wherein the second global feature includes geometric features of the shape prior data; a fourth processing module for determining a target deformation field and a target matching matrix of the target object based on the first global feature and the second global feature, and determining a posture prediction result of the target object based on the target deformation field and the target matching matrix, wherein the target deformation field is used to adjust the shape of the target object, the target matching matrix is ​​used to determine the position and orientation of the target object, and the posture prediction result includes posture information and size information of the target object.

[0013] According to another aspect of the embodiments of the present application, a nonvolatile storage medium is also provided, the nonvolatile storage medium storing a program, wherein the program, when executed, controls a device in which the nonvolatile storage medium is located to perform the pose prediction method.

[0014] According to another aspect of the embodiments of the present application, an electronic device is also provided, comprising a memory and a processor, the processor being configured to execute a program stored in the memory, wherein the program, when executed, performs the pose prediction method.

[0015] According to another aspect of the embodiments of the present application, a computer program product is also provided, comprising a computer program, the computer program, when executed by a processor, implementing the pose prediction method.

[0016] In the embodiments of the present application, the target image, the target point cloud data corresponding to the target image, and the shape prior data corresponding to the target image are acquired, wherein the target image is an image of a target object collected, the target point cloud data is point cloud data of the target object, and the shape prior data comprises shape information of the target object; the first global feature is determined according to the target image and the target point cloud data, wherein the first global feature comprises features of the target image and features of the target point cloud data; the second global feature is determined according to the shape prior data, wherein the second global feature comprises geometric features of the shape prior data; the target deformation field and the target matching matrix of the target object are determined according to the first global feature and the second global feature, and the pose prediction result of the target object is determined according to the target deformation field and the target matching matrix, wherein the target deformation field is used to adjust the shape of the target object, the target matching matrix is used to determine the position and direction of the target object, and the pose prediction result comprises the pose information and the size information of the target object. By introducing the shape prior data, the image data and the point cloud data to jointly determine the deformation field and the matching matrix, and determining the pose prediction result according to the deformation field and the matching matrix, the purpose of avoiding the interference of the external environment on the prediction result is achieved, thereby realizing the technical effect of improving the accuracy of the pose prediction result, and further solving the technical problem that the pose prediction method in the related art cannot accurately predict the motion state of the object due to the inability to guarantee the accuracy of the prediction result in a complex environment. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings, which are included to provide a further understanding of the present application and are incorporated in and constitute a part of this application, illustrate embodiments of the present application and serve to explain the present application. In the drawings:

[0018] Figure 1 is a structural schematic diagram of a computer terminal (mobile terminal) provided according to an embodiment of the present application;

[0019] Figure 2 is a flowchart of a pose prediction method according to an embodiment of the present application;

[0020] Figure 3 is a structural diagram of a pose prediction model according to an embodiment of the present application;

[0021] Figure 4 is a flowchart of a pose prediction flow according to an embodiment of the present application;

[0022] Figure 5 is a structural diagram of a pose prediction apparatus according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make the personnel in the art better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0025] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

[0026] 6DOF (Six Degrees of Freedom): refers to the ability of an object to move freely in three-dimensional space in terms of position and orientation. Specifically, it includes movement along the X, Y, and Z axes (translational freedom) and rotation around these three axes (rotational freedom). 6DOF technology is widely used and is particularly important in fields such as robotics, virtual reality (VR), augmented reality (AR), and aerospace. It allows devices or objects to accurately control and simulate all possible movements and positions in physical space, making user interaction and system control more precise and natural.

[0027] RGBD: is an image data format that contains both color (RGB) and depth information. In RGBD data, the RGB part provides color information for each pixel, while the D part provides distance information from the camera, i.e., the depth of each pixel. This data format allows simultaneous acquisition and processing of the visual appearance and spatial position of objects, and is widely used in computer vision and machine learning fields such as 3D reconstruction, scene understanding, and robot navigation.

[0028] MLP (Multi-Layer Perceptron): is a type of feedforward artificial neural network composed of multiple layers, including an input layer, one or more hidden layers, and an output layer. Each layer is composed of multiple neurons, which are connected to neurons in the previous and subsequent layers through weights. MLP uses nonlinear activation functions such as Sigmoid, ReLU, etc., allowing the network to handle nonlinear problems.

[0029] ResNet (Residual Network): is a deep convolutional neural network architecture proposed by researchers at Microsoft Research in 2015 to address the problem of gradient vanishing or explosion when increasing the depth of deep learning models. The core idea of ResNet is to introduce a residual learning framework, which directly passes input information across layers to output, and the network only needs to learn the residual part between input and output.

[0030] With the rapid development of artificial intelligence and computer vision technology, automated 6D pose estimation has become a key technology in the fields of industrial automation, augmented reality, and robot navigation. Among them, the class-level 6D pose estimation technology, which predicts the three-dimensional position and orientation of unseen objects, is one of the most complex and challenging problems. Currently, most pose estimation methods rely on high-precision sensors and complex algorithms to handle single-instance recognition and localization, but these methods cannot be directly applied to unseen object classes, limiting their generalization and practicality.

[0031] Specifically, the 6D pose estimation method in the related art has the following problems:

[0032] First, the robustness is insufficient: the traditional pose estimation method often relies on specific markers or simple geometric models, which makes it difficult to maintain high accuracy and stability in environments with visual occlusion, light changes or complex backgrounds.

[0033] Second, poor generalization ability: these methods usually need to be adjusted or trained for each new scene or object, and have poor adaptability to new or unseen objects. This limits their application in dynamic environments or variable tasks.

[0034] Third, high demand for computing resources: many traditional methods, such as iterative closest point algorithm (ICP) or complex three-dimensional model matching, are computationally expensive, requiring a large amount of processing time and high-performance computing resources, which is not practical in real-time or resource-constrained applications.

[0035] Fourth, lack of flexibility in scale and angle: traditional methods are usually sensitive to the size and direction of objects, and their performance will be greatly reduced if the preset model deviates from the actual object in size or angle.

[0036] Fifth, poor handling of asymmetric and complex shapes: when the shape of the object is complex or asymmetric, traditional pose estimation methods often have difficulty accurately recognizing and positioning because they usually assume that the object has a certain symmetry or regular shape.

[0037] To solve the above problems, the related solutions are provided in the embodiments of the present application, which are described in detail below.

[0038] According to the embodiments of the present application, a method embodiment of a pose prediction method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0039] The method embodiment provided by the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing a pose prediction method is shown. As Figure 1As shown, the computer terminal 10 (or mobile device 10) may include one or more (illustrated as 102a, 102b, ..., 102n) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0040] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0041] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the posture prediction method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned posture prediction method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0042] The transmission device 106 is configured to receive or send data via a network. The network can include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network interface controller (NIC) that can connect to other network devices through a base station to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module that is configured to communicate with the Internet wirelessly.

[0043] The display can be a liquid crystal display (LCD) that is touch screen type, for example, which can enable a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0044] In the above operating environment, the embodiments of the present application provide a pose prediction method, as shown in the following formula (1): Figure 2 The method includes the following steps:

[0045] In step S202, a target image, target point cloud data corresponding to the target image, and shape prior data corresponding to the target image are obtained, wherein the target image is an image of a target object collected, the target point cloud data is point cloud data of the target object, and the shape prior data includes shape information of the target object.

[0046] In some embodiments of the present application, the target image can be a calibrated RGB-D image, and the target of the pose estimation is to estimate the 6D pose and 3D size of an object of interest (i.e., the target object). The 6D pose of the target object is represented as {R, T} ∈ SE(3), wherein R ∈ SO(3) represents a 3D rotation, and T ∈ R3 represents a 3D translation. The 3D size is represented as S ∈ R3. For each instance of the target object, an instance observation is represented as (I, V), wherein I ∈ R H×W×3 is an image block, is a corresponding point cloud block from a depth image, i.e., the target point cloud data, and (H, W) represents the size of the image block. N v is the number of points in the point cloud V, is the corresponding shape prior, N p is the number of points in the shape prior V c .

[0047] In step S204, a first global feature is determined according to the target image and the target point cloud data, wherein the first global feature includes features of the target image and features of the target point cloud data.

[0048] In the technical solution provided in step S204, as shown in the following formula (2): Figure 3As shown, the feature extraction network includes a first feature extraction branch, a second feature extraction branch and a feature processing branch. The first feature extraction branch is used to extract the image features of the target image, the second feature extraction branch is used to extract the point cloud features of the target point cloud data, and the feature processing branch is used to process the intermediate processing results of the first feature extraction branch and the second feature extraction branch. Determining the first global feature based on the target image and the target point cloud data includes: processing the target image and the target point cloud data through the feature extraction network to obtain the first output result of the first feature extraction branch, the second output result of the second feature extraction branch and the third output result of the feature processing branch; fusing the first output result, the second output result and the third output result to obtain the first global feature. The three branches are Figure 3 Different colors are used to mark them, where blue represents the first feature extraction branch, green represents the second feature extraction branch, and red represents the feature processing branch.

[0049] As an optional implementation, the first feature extraction branch and the feature processing branch, as well as the second feature extraction branch and the feature processing branch are connected through an attention mechanism module, wherein the intermediate processing results of the first feature extraction branch and the intermediate processing results of the second feature extraction branch are processed by the attention mechanism module and then input into the corresponding convolution layer of the feature processing branch.

[0050] In some embodiments of the present application, each branch of the feature extraction network can be a ResNet network. ResNet is a classic and widely verified deep network that is widely used for feature extraction. Figure 3 As shown in , two ResNet networks can be used to extract features from RGB images and depth data (that is, point cloud data) respectively, and another ResNet network can be used to fuse the extracted features. Figure 3 As shown, the method provided in this application also integrates an attention-assisted module into the ResNet network. Taking advantage of the ability to enhance key features and improve multi-scale data processing, an attention-assisted module is designed to enhance the performance of the feature extraction and fusion network with three parallel branches.

[0051] Optionally, image input I = R H×W×3 First, it passes through a global average pooling layer to produce the output The formula is Then the output It is recombined with another three-channel 1×1 convolution layer to obtain the weight distribution across different channels. The convolution result is activated by the sigmoid function and the weight vector is defined as V∈R 1×1×3 Finally, the outer product of input I and V is performed to obtain the final result Y, where Y∈RH×W×3 can be expressed as follows, where represents the outer product, σ represents the sigmoid function, and φ represents the 1×1 convolution. The formula for Y is The intermediate results are obtained through a network containing five layers of attention modules. Then the final result F is obtained. g , effectively integrating the deep and shallow features. Then, for this output F g Processing is done through an MLP layer and an average pooling layer to finally obtain a global feature vector G g .

[0052] Step S206, determining a second global feature based on the shape prior data, wherein the second global feature includes a geometric feature of the shape prior data;

[0053] In the technical solution provided in step S206, geometric features can be extracted from the shape prior data. Optionally, pointnet++ encoder can be used to extract geometric features from the category. The extracted features are denoted as F p With dimension m×64, it is fed into the self-attention layer for feature enhancement. This mechanism allows to enhance F by calculating the weight distribution between features. p The features of , thereby strengthening the mutual relationship between them. Subsequently, the average pooling layer is used to extract the global feature G p .

[0054] Step S208: Determine the target deformation field and target matching matrix of the target object based on the first global feature and the second global feature, and determine the pose prediction result of the target object based on the target deformation field and the target matching matrix, wherein the target deformation field is used to adjust the shape of the target object, the target matching matrix is ​​used to determine the position and orientation of the target object, and the pose prediction result includes pose information and size information of the target object.

[0055] In the technical solution provided in step S208, the step of determining the target deformation field and the target matching matrix of the target object according to the first global feature and the second global feature includes: processing the first global feature and the second global feature by using a feature exchange network to obtain initial first fusion features and initial second fusion features; processing the first fusion features and the second fusion features by using a model recursive recovery network to obtain an initial deformation field and an initial matching matrix; determining recursive features according to the initial deformation field and shape prior data, and iteratively processing the recursive features, the first global feature and the second global feature to obtain the target deformation field and the target matching matrix, wherein in each iteration, the feature exchange network is used to process the first global feature, the second global feature and the recursive features to obtain the first fusion features and the second fusion features, and the deformation field and the matching matrix are determined according to the first fusion features and the second fusion features, and the recursive features in the next iteration are determined according to the deformation field and the shape prior data in the case where the iteration termination condition is not met.

[0056] In some embodiments of the present application, the step of determining the recursive features according to the initial deformation field and the shape prior data, and iteratively processing the recursive features, the first global feature and the second global feature includes:

[0057] Firstly, the recursive features, the first global feature and the second global feature are processed by using the feature exchange network to obtain the first fusion features and the second fusion features;

[0058] Secondly, the deformation field and the matching matrix are determined according to the first fusion features and the second fusion features;

[0059] Thirdly, it is determined whether the iteration termination condition is met, and in the case where it is determined that the iteration termination condition is met, the fifth step is jumped to, otherwise the fourth step is jumped to, wherein the iteration termination condition includes that the iteration number reaches a preset number threshold, or the deformation field residual corresponding to the deformation field and the matching matrix residual corresponding to the matching matrix meet a preset requirement;

[0060] Fourthly, the recursive features are determined again according to the deformation field and the shape prior data, and the first step is jumped to;

[0061] Fifthly, the deformation field obtained in the current iteration is determined as the target deformation field, and the matching matrix obtained in the current iteration is determined as the target matching matrix.

[0062] In some embodiments of the present application, after the global features G p and G g are obtained, the feature exchange network can be used to process these global features to generate exchanged fusion features F m and F n . In the first generation of fusion features, the input of the feature exchange network is the global feature G p and Gg In the subsequent iteration process, the input of the feature exchange network includes the recursively recovered feature Fr and the global feature G p and G g The output is F m and F n . F m The calculation formula is F n The calculation formula is Where η represents the nonlinear change function in the feature exchange network.

[0063] like Figure 3 As shown, when the fusion feature F is first generated m and F n After processing by the MLP layer, the initial deformation field and matching matrix D can be obtained. 0 and M 0 . Then you can use D 0 Merge to V C Update the NOCS model in the MLP layer, and the feature obtained is F r . F r The formula is Then F r Feedback to the feature exchange network to obtain F m1 and F n1 In addition, the optimization target of the network is the deformation field and matching matrix The residual. The formula is The formula is By optimizing the residual, the deformation field can be and matching matrix The final estimate of gt Represents the true value of the deformation field, M gt Represents the true value of the matching matrix.

[0064] As an optional implementation manner, the preset requirement that the residual needs to meet may be that the residuals of the deformation field and the matching matrix converge.

[0065] In some embodiments of the present application, the step of determining the pose prediction result of the target object based on the target deformation field and the target matching matrix includes: establishing a normalized object coordinate space model based on the target deformation field and shape prior data; and determining the pose prediction result based on the normalized object coordinate space model and the target point cloud data.

[0066] As an optional implementation, the step of determining the pose prediction result of the target object according to the target deformation field and the target matching matrix comprises: determining coordinate values of each point in the target point cloud data in the normalized object coordinate space according to the target point cloud data and the target matching matrix; determining similarity transformation parameters according to the coordinate values of each point in the point cloud data and the normalized object coordinate space model, and determining the pose prediction result according to the similarity transformation parameters.

[0067] In some embodiments of the present application, after obtaining the final deformation field and the matching matrix , the equation can be used to obtain the 6D pose of the object. Optionally, the shape prior is applied to reconstruct the NOCS model Vm. The point cloud V and the NOCS model Vm are located in the NOCS and camera coordinate space respectively. Estimating the 6D pose of the object is equivalent to finding the similarity transformation parameters between the observed point cloud V and Vm. That is, the similarity transformation can be applied to Vm and the NOCS coordinates of each point in v can be calculated. Finally, the Umeyama algorithm can be used to calculate the similarity transformation and the RANSAC algorithm can be used to remove outliers, so as to achieve robust estimation.

[0068] As an optional implementation, the process of determining the pose prediction result according to the similarity transformation parameters can include the following steps:

[0069] Step 1: Apply similarity transformation.

[0070] The similarity transformation parameters consist of a rotation matrix (R), a translation vector (T) and a scaling factor (S), which collectively describe the exact spatial relationship between the target point cloud data and the normalized object coordinate space model. Once these parameters are calculated, they can be directly applied to adjust the normalized object coordinate space model to align it with the object in the target point cloud data.

[0071] Step 2: Convert coordinates.

[0072] Convert the points in the normalized object coordinate space model using the obtained similarity transformation parameters.

[0073] Step 3: Precise matching.

[0074] The converted model point cloud (P') should match the target point cloud data as perfectly as possible, which means the points of the model should fall in the same positions as the target object, considering possible errors, this matching can be achieved by minimizing some form of distance metric (such as Euclidean distance), ensuring that the rotation, translation and scaling operations can most accurately reflect the true pose and size of the target object.

[0075] Step 4: Outlier rejection and optimization.

[0076] In practical applications, due to sensor noise, occlusion or other factors, the target point cloud data may contain some outliers, which will interfere with pose estimation. Therefore, a robust algorithm (such as RANSAC) is usually used to reject these outliers, and only the points well matched with the model are retained for pose estimation, thereby improving the accuracy and stability of the estimation.

[0077] Step 5: Output the pose prediction result.

[0078] Finally, the results obtained by the above rotation, translation and scaling operations can be output as the pose prediction result of the target object, including its rotation matrix (R), translation vector (T) and scaling factor (S), which together constitute the 6D pose and 3D size information of the object.

[0079] Optionally, Figure 3 A 6D pose estimation network for performing the method provided by the embodiments of the present application is shown, which includes a feature extraction network, a feature exchange network and a model recursive recovery network, wherein the model recursive recovery network includes Figure 3 three MLP layers on the right. The loss function of the 6D pose estimation network includes a reconstruction loss L cd , which uses the deformation field D and the chamfer distance of the object instance model as the reconstruction loss. A smooth L1 loss L corr is used to reflect the deviation of the reconstructed NOCS model coordinates from the true value. A cross-entropy loss Lentropy, and an L2 regularization loss Ldef are used to maintain the shape and semantic consistency of the model.

[0080] In addition, since the recursive recovery network of the model needs to be supervised in each iteration, a deep supervision mechanism is also used in the embodiments of the present application to supervise each iteration. This involves learning the residual function of each round of recursion, and the loss function is represented as:

[0081]

[0082] In the above loss function, the coefficients a, b, g, d are used to balance the weights of different loss terms to guide the model to consider geometric reconstruction, feature consistency, semantic classification and shape regularization in the iterative optimization process.

[0083] In some embodiments of the present application, the process of estimating the 6D pose information and size information of the object by the model shown in Figure 3 is shown in Figure 4 . From Figure 4As can be seen, the first global feature fused with the RGB image feature and the point cloud feature can be extracted through the feature extraction network first. Then the first global feature and the shape prior are input into the corresponding second global feature into the feature exchange network. Then the feature exchange network and the model recursive network will iteratively process the input features, so as to obtain the 6D pose estimation result, including the 6D pose information and the size information.

[0084] By adopting the target image, the target point cloud data corresponding to the target image, and the shape prior data corresponding to the target image, wherein the target image is an image of a target object collected, the target point cloud data is point cloud data of the target object, and the shape prior data includes shape information of the target object; determining the first global feature according to the target image and the target point cloud data, wherein the first global feature includes features of the target image and features of the target point cloud data; determining the second global feature according to the shape prior data, wherein the second global feature includes geometric features of the shape prior data; determining the target deformation field and the target matching matrix of the target object according to the first global feature and the second global feature, and determining the pose prediction result of the target object according to the target deformation field and the target matching matrix, wherein the target deformation field is used to adjust the shape of the target object, and the target matching matrix is used to determine the position and direction of the target object. By introducing the shape prior data, the image data and the point cloud data to determine the deformation field and the matching matrix together, and determining the pose prediction result according to the deformation field and the matching matrix, the purpose of avoiding the interference of the external environment on the prediction result is achieved, thereby realizing the technical effect of improving the accuracy of the pose prediction result, and further solving the technical problem that the pose prediction method in the related art cannot accurately predict the motion state of the object due to the inability to guarantee the accuracy of the prediction result in a complex environment.

[0085] The embodiment of the present application provides a pose prediction device, Figure 5 is a structural schematic diagram of the device. From Figure 5As can be seen, the device comprises: a first processing module 50 configured to acquire a target image, target point cloud data corresponding to the target image, and shape prior data corresponding to the target image, wherein the target image is an image of a target object collected, the target point cloud data is point cloud data of the target object, and the shape prior data comprises shape information of the target object; a second processing module 52 configured to determine a first global feature according to the target image and the target point cloud data, wherein the first global feature comprises features of the target image and features of the target point cloud data; a third processing module 54 configured to determine a second global feature according to the shape prior data, wherein the second global feature comprises geometric features of the shape prior data; and a fourth processing module 56 configured to determine a target deformation field and a target matching matrix of the target object according to the first global feature and the second global feature, and determine a pose prediction result of the target object according to the target deformation field and the target matching matrix, wherein the target deformation field is used to adjust the shape of the target object, the target matching matrix is used to determine the position and direction of the target object, and the pose prediction result comprises pose information and size information of the target object.

[0086] In some embodiments of the present application, the feature extraction network comprises a first feature extraction branch, a second feature extraction branch, and a feature processing branch, the first feature extraction branch is configured to extract image features of the target image, the second feature extraction branch is configured to extract point cloud features of the target point cloud data, and the feature processing branch is configured to process intermediate processing results of the first feature extraction branch and the second feature extraction branch; the step of determining the first global feature according to the target image and the target point cloud data by the second processing module 52 comprises: processing the target image and the target point cloud data by the feature extraction network to obtain a first output result of the first feature extraction branch, a second output result of the second feature extraction branch, and a third output result of the feature processing branch; and fusing the first output result, the second output result, and the third output result to obtain the first global feature.

[0087] In some embodiments of the present application, the first feature extraction branch and the feature processing branch, and the second feature extraction branch and the feature processing branch are connected through an attention mechanism module, wherein the intermediate processing result of the first feature extraction branch and the intermediate processing result of the second feature extraction branch are input into corresponding convolution layers of the feature processing branch after being processed by the attention mechanism module.

[0088] In some embodiments of the present application, the fourth processing module 56 determines the target deformation field and the target matching matrix of the target object according to the first global feature and the second global feature, including: processing the first global feature and the second global feature by using a feature exchange network to obtain initial first fusion features and initial second fusion features; processing the first fusion features and the second fusion features by using a model recursive recovery network to obtain an initial deformation field and an initial matching matrix; determining recursive features according to the initial deformation field and shape prior data, and iteratively processing the recursive features, the first global feature and the second global feature to obtain the target deformation field and the target matching matrix, wherein in each iteration, the first global feature, the second global feature and the recursive features are processed by using the feature exchange network to obtain the first fusion features and the second fusion features, and the deformation field and the matching matrix are determined according to the first fusion features and the second fusion features, and the recursive features in the next iteration are determined according to the deformation field and the shape prior data in the case where the iteration termination condition is not met.

[0089] In some embodiments of the present application, the fourth processing module 56 determines the recursive features according to the initial deformation field and the shape prior data, and iteratively processes the recursive features, the first global feature and the second global feature, including:

[0090] Firstly, the recursive features, the first global feature and the second global feature are processed by using the feature exchange network to obtain the first fusion features and the second fusion features;

[0091] Secondly, the deformation field and the matching matrix are determined according to the first fusion features and the second fusion features;

[0092] Thirdly, it is determined whether the iteration termination condition is met, and in the case where it is determined that the iteration termination condition is met, the fifth step is jumped to, otherwise the fourth step is jumped to, wherein the iteration termination condition includes that the iteration number reaches a preset number threshold, or the deformation field residual corresponding to the deformation field and the matching matrix residual corresponding to the matching matrix meet a preset requirement;

[0093] Fourthly, the recursive features are determined again according to the deformation field and the shape prior data, and the first step is jumped to;

[0094] Fifthly, the deformation field obtained in the current iteration is determined as the target deformation field, and the matching matrix obtained in the current iteration is determined as the target matching matrix.

[0095] In some embodiments of the present application, the fourth processing module 56 determines the pose prediction result of the target object according to the target deformation field and the target matching matrix, including: establishing a normalized object coordinate space model according to the target deformation field and the shape prior data; and determining the pose prediction result according to the normalized object coordinate space model and the target point cloud data.

[0096] In some embodiments of the present application, the fourth processing module 56 determines the pose prediction result of the target object according to the target deformation field and the target matching matrix, and the step includes: determining the coordinate value of each point in the target point cloud data in the normalized object space coordinate according to the target point cloud data and the target matching matrix; determining the similarity transformation parameter according to the coordinate value of each point in the point cloud data and the normalized object coordinate space model, and determining the pose prediction result according to the similarity transformation parameter.

[0097] It should be noted that each module in the above pose prediction device can be a program module (for example, a set of program instructions for implementing a certain specific function) or a hardware module. For the latter, it can be in the following form, but is not limited to: each module is in the form of a processor, or the functions of each module are implemented by a processor.

[0098] According to the embodiments of the present application, a non-volatile storage medium is also provided, and the non-volatile storage medium stores a program. When the program is running, the device in which the non-volatile storage medium is located performs the following pose prediction method: obtaining a target image, target point cloud data corresponding to the target image, and shape prior data corresponding to the target image, wherein the target image is an image of a target object collected, the target point cloud data is point cloud data of the target object, and the shape prior data includes shape information of the target object; determining a first global feature according to the target image and the target point cloud data, wherein the first global feature includes features of the target image and features of the target point cloud data; determining a second global feature according to the shape prior data, wherein the second global feature includes geometric features of the shape prior data; determining a target deformation field and a target matching matrix of the target object according to the first global feature and the second global feature, and determining a pose prediction result of the target object according to the target deformation field and the target matching matrix, wherein the target deformation field is used to adjust the shape of the target object, the target matching matrix is used to determine the position and direction of the target object, and the pose prediction result includes pose information and size information of the target object.

[0099] According to the embodiment of the present application, an electronic device is also provided, comprising a memory and a processor, the processor being configured to run a program stored in the memory, wherein the program performs the following pose prediction method when running: obtaining a target image, target point cloud data corresponding to the target image, and shape prior data corresponding to the target image, wherein the target image is an image of a target object collected, the target point cloud data is point cloud data of the target object, and the shape prior data comprises shape information of the target object; determining first global features according to the target image and the target point cloud data, wherein the first global features comprise features of the target image and features of the target point cloud data; determining second global features according to the shape prior data, wherein the second global features comprise geometric features of the shape prior data; determining a target deformation field and a target matching matrix of the target object according to the first global features and the second global features, and determining a pose prediction result of the target object according to the target deformation field and the target matching matrix, wherein the target deformation field is used to adjust the shape of the target object, the target matching matrix is used to determine the position and direction of the target object, and the pose prediction result comprises pose information and size information of the target object.

[0100] According to the embodiment of the present application, a computer program product is also provided, comprising a computer program, which, when executed by a processor, implements the following pose prediction method: obtaining a target image, target point cloud data corresponding to the target image, and shape prior data corresponding to the target image, wherein the target image is an image of a target object collected, the target point cloud data is point cloud data of the target object, and the shape prior data comprises shape information of the target object; determining first global features according to the target image and the target point cloud data, wherein the first global features comprise features of the target image and features of the target point cloud data; determining second global features according to the shape prior data, wherein the second global features comprise geometric features of the shape prior data; determining a target deformation field and a target matching matrix of the target object according to the first global features and the second global features, and determining a pose prediction result of the target object according to the target deformation field and the target matching matrix, wherein the target deformation field is used to adjust the shape of the target object, the target matching matrix is used to determine the position and direction of the target object, and the pose prediction result comprises pose information and size information of the target object.

[0101] In the above embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0102] In several embodiments provided in the present application, it should be understood that the disclosed technology can be implemented by other ways. Among them, the above-mentioned device embodiments are only schematic, for example, the division of the units can be a logical function division, and actual implementation can have another division mode, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be indirect coupling or communication connection through some interfaces, units or modules, which can be electrical or other forms.

[0103] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed to multiple units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0104] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0105] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the related art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and various program codes that can be stored in the medium.

[0106] The above is only the preferred embodiment of the present application, and it should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should be considered as the protection scope of the present application.

Claims

1. A pose prediction method, characterized by, The method comprises the following steps: obtaining a target image, target point cloud data corresponding to the target image, and shape prior data corresponding to the target image, wherein the target image is an image of a target object collected, the target point cloud data is point cloud data of the target object, and the shape prior data comprises shape information of the target object; determining a first global feature according to the target image and the target point cloud data, wherein the first global feature comprises features of the target image and features of the target point cloud data; determining a second global feature according to the shape prior data, wherein the second global feature comprises geometric features of the shape prior data; determining a target deformation field and a target matching matrix of the target object according to the first global feature and the second global feature, and determining a pose prediction result of the target object according to the target deformation field and the target matching matrix, wherein the target deformation field is used to adjust the shape of the target object, the target matching matrix is used to determine the position and direction of the target object, and the pose prediction result comprises pose information and size information of the target object.

2. The pose prediction method of claim 1, wherein, The feature extraction network comprises a first feature extraction branch, a second feature extraction branch, and a feature processing branch, the first feature extraction branch is used to extract image features of the target image, the second feature extraction branch is used to extract point cloud features of the target point cloud data, and the feature processing branch is used to process intermediate processing results of the first feature extraction branch and the second feature extraction branch; determining a first global feature according to the target image and the target point cloud data comprises: processing the target image and the target point cloud data through the feature extraction network to obtain a first output result of the first feature extraction branch, a second output result of the second feature extraction branch, and a third output result of the feature processing branch; fusing the first output result, the second output result, and the third output result to obtain the first global feature.

3. The pose prediction method of claim 2, wherein, The first feature extraction branch and the feature processing branch, and the second feature extraction branch and the feature processing branch are connected through attention mechanism modules, wherein the intermediate processing results of the first feature extraction branch and the second feature extraction branch are input into corresponding convolution layers of the feature processing branch after being processed by the attention mechanism modules.

4. The pose prediction method of claim 1, wherein, determining a target deformation field and a target matching matrix of the target object according to the first global feature and the second global feature comprises: processing the first global feature and the second global feature through a feature exchange network to obtain an initial first fusion feature and an initial second fusion feature; processing the first fusion feature and the second fusion feature through a model recursive recovery network to obtain an initial deformation field and an initial matching matrix; determining a recursive feature according to the initial deformation field and the shape prior data, and performing iterative processing on the recursive feature, the first global feature and the second global feature, to obtain the target deformation field and the target matching matrix, wherein in each iteration, the feature exchange network is used to process the first global feature, the second global feature and the recursive feature, to obtain first fusion features and second fusion features, and a deformation field and a matching matrix are determined according to the first fusion features and the second fusion features, and in the case where an iteration termination condition is not met, a recursive feature in the next iteration is determined according to the deformation field and the shape prior data.

5. The pose prediction method of claim 4, wherein, determining a recursive feature according to the initial deformation field and the shape prior data, and performing iterative processing on the recursive feature, the first global feature and the second global feature, to obtain the target deformation field and the target matching matrix, wherein in each iteration, the feature exchange network is used to process the first global feature, the second global feature and the recursive feature, to obtain first fusion features and second fusion features, and a deformation field and a matching matrix are determined according to the first fusion features and the second fusion features, and in the case where an iteration termination condition is not met, a recursive feature in the next iteration is determined according to the deformation field and the shape prior data. The method comprises: a first step of processing the recursive feature, the first global feature and the second global feature by the feature exchange network to obtain the first fusion features and the second fusion features; a second step of determining a deformation field and a matching matrix according to the first fusion features and the second fusion features; a third step of determining whether an iteration termination condition is met, and in the case where the iteration termination condition is met, jumping to a fifth step, otherwise jumping to a fourth step, wherein the iteration termination condition comprises that an iteration number reaches a preset number threshold, or a deformation field residual corresponding to the deformation field and a matching matrix residual corresponding to the matching matrix meet a preset requirement; a fourth step of determining the recursive feature again according to the deformation field and the shape prior data, and jumping to the first step; 6. The posture prediction method according to claim 1, characterized in that a fifth step of determining the deformation field obtained in the current iteration as the target deformation field, and determining the matching matrix obtained in the current iteration as the target matching matrix. The method comprises: establishing a normalized object coordinate space model according to the target deformation field and the shape prior data; 7. The pose prediction method of claim 6, wherein, determining the pose prediction result according to the normalized object coordinate space model and the target point cloud data. The method comprises: determining coordinate values of each point in the target point cloud data in a normalized object space coordinate according to the target point cloud data and the target matching matrix; 8. An apparatus for pose prediction, the apparatus comprising: determining similarity transformation parameters according to the coordinate values of each point in the point cloud data and the normalized object coordinate space model, and determining the pose prediction result according to the similarity transformation parameters. The method comprises: a first processing module configured to acquire a target image, target point cloud data corresponding to the target image and shape prior data corresponding to the target image, wherein the target image is an image of a target object, the target point cloud data is point cloud data of the target object, and the shape prior data comprises shape information of the target object; determine a first global feature according to the target image and the target point cloud data, wherein the first global feature comprises a feature of the target image and a feature of the target point cloud data; determine a second global feature according to the shape prior data, wherein the second global feature comprises a geometric feature of the shape prior data; determine a target deformation field and a target matching matrix of the target object according to the first global feature and the second global feature, and determine a pose prediction result of the target object according to the target deformation field and the target matching matrix, wherein the target deformation field is used to adjust a shape of the target object, the target matching matrix is used to determine a position and a direction of the target object, and the pose prediction result comprises pose information and size information of the target object.

9. A non-volatile storage medium, comprising: The non-volatile storage medium stores a program, wherein the program controls a device in which the non-volatile storage medium is located to perform the pose prediction method in any one of claims 1 to 7 when the program is running.

10. An electronic device, comprising: comprise: a memory and a processor, wherein the processor is configured to execute a program stored in the memory, and the program performs the pose prediction method in any one of claims 1 to 7 when the program is running.

11. A computer program product, characterised in that, comprise a computer program, wherein the computer program is configured to implement the pose prediction method in any one of claims 1 to 7 when executed by a processor.

Images