Registration relationship determination method and apparatus, computer device, and storage medium

Abstract

The application relates to a registration relationship determination method and device, computer equipment and a storage medium. The method comprises the following steps: obtaining a to-be-matched point cloud and a reference point cloud of a moving object, performing spatial registration on the to-be-matched point cloud and the reference point cloud, and determining a registration relationship between the to-be-matched point cloud and the reference point cloud by taking the minimum change of the to-be-matched point cloud under non-rigid registration as a registration constraint. The spatial registration comprises simultaneous rigid registration and non-rigid registration, the joint estimation of rigid motion and non-rigid motion of the moving object is realized, the spatial registration process is more in line with the actual motion of the moving object, the change of the to-be-matched point cloud under non-rigid registration in the spatial registration is constrained to be minimum, the estimation accuracy of the rigid motion and the non-rigid motion of the moving object is improved, and the accuracy of the determined registration relationship is further improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus, computer equipment, and storage medium for determining registration relationships. Background Technology

[0002] The general process of image spatial registration is to determine a set of spatial registration relationships so that, after spatial transformation, corresponding points in one image are spatially consistent with those in another image. Image spatial registration also has wide applications in the medical field, with typical applications including Simultaneous Localization and Mapping (SLAM) in the surgical environment, head tracking with facial expressions, Non-Rigid Structure from Motion (NRSfM), and deformable medical image registration, etc.

[0003] When the research object exhibits both rigid and non-rigid motion, a non-rigid registration algorithm is required for registration. In related techniques, it is common practice to first assume the research object undergoes rigid motion to coarsely match the spatial registration relationship, and then use a non-rigid registration algorithm to constrain the coarse matching, thereby achieving spatial registration of the research object before and after its motion.

[0004] However, the coarse matching process in related technologies is based on the assumption of rigid body motion, which does not conform to the actual motion state of the research object. Therefore, the accuracy of the spatial registration relationship determined in the registration process is poor. Summary of the Invention

[0005] Therefore, it is necessary to provide a registration relationship determination method, apparatus, computer equipment, computer-readable storage medium, and computer program product to address the above-mentioned technical problems.

[0006] Firstly, this application provides a method for determining registration relationships, including:

[0007] Obtain the point cloud to be matched and the reference point cloud of the moving object;

[0008] Spatial registration is performed between the point cloud to be matched and the reference point cloud, and the registration relationship between the point cloud to be matched and the reference point cloud is determined by minimizing the change of the point cloud to be matched under non-rigid registration. Spatial registration includes both rigid registration and non-rigid registration performed simultaneously.

[0009] In one embodiment, spatial registration is performed between the point cloud to be matched and the reference point cloud, and the registration relationship between the point cloud to be matched and the reference point cloud is determined with the constraint of minimizing the change of the point cloud to be matched under non-rigid registration, including:

[0010] The point cloud to be matched and the reference point cloud are input into the registration model for solution to obtain the registration relationship between the point cloud to be matched and the reference point cloud. The registration model aims to match the positions of the point cloud to be matched and the reference point cloud after spatial registration, and minimizes the change of the point cloud to be matched under non-rigid registration as the model constraint.

[0011] In one embodiment, the method further includes:

[0012] The model objective is to match the positions of the point cloud to be matched and the reference point cloud after spatial registration, and the model constraint is to minimize the change between the point cloud to be matched and the reference point cloud under non-rigid registration in spatial registration. A registration model is constructed.

[0013] In one embodiment, the model objective satisfies the following relationship:

[0014] After spatial registration, the sum of the distances between the target point cloud to be matched and the corresponding target reference point cloud meets the preset conditions; the target point cloud to be matched and the target reference point cloud are successfully matched point clouds and reference point clouds by minimizing the sum of their distances.

[0015] In one embodiment, the model constraints satisfy the following relationship:

[0016] After spatial registration, the sum of the changes in the point cloud to be matched is minimized under non-rigid registration.

[0017] In one embodiment, the sum of the variation distances of the point cloud to be matched under non-rigid registration is minimized, including:

[0018] A non-rigid transformation is performed on the point cloud to be matched based on the affine transformation matrix and translation matrix of the auxiliary points of the point cloud to be matched, resulting in the transformed point cloud. Under non-rigid registration, the sum of the transformation distances between the point cloud to be matched and the transformed point cloud is minimized.

[0019] In one embodiment, acquiring the point cloud to be matched and the reference point cloud of the moving object includes:

[0020] Acquire tracking images of the moving object, and obtain at least two depth images based on the tracking images;

[0021] The point cloud to be matched and the reference point cloud are determined based on at least two depth images.

[0022] Secondly, this application also provides a registration relationship determination device, comprising:

[0023] The point cloud acquisition module is used to acquire the point cloud to be matched and the reference point cloud of the moving object;

[0024] The spatial registration module is used to perform spatial registration between the point cloud to be matched and the reference point cloud, and determines the registration relationship between the point cloud to be matched and the reference point cloud by taking the minimum change of the point cloud to be matched under non-rigid registration as a constraint; wherein, spatial registration includes rigid registration and non-rigid registration performed simultaneously.

[0025] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the above methods.

[0026] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above methods.

[0027] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0028] The aforementioned registration relationship determination method, apparatus, computer equipment, storage medium, and computer program product acquire the point cloud to be matched and the reference point cloud of a moving object, perform spatial registration between the two point clouds, and determine the registration relationship between them by minimizing the change of the point cloud to be matched under non-rigid registration as the registration constraint. Spatial registration includes simultaneous rigid and non-rigid registration, achieving joint estimation of the rigid and non-rigid motion of the moving object. This makes the spatial registration process more consistent with the actual motion of the moving object, while simultaneously constraining the point cloud to be matched to minimize the change under non-rigid registration in spatial registration, improving the estimation accuracy of the rigid and non-rigid motion of the moving object, and thus improving the accuracy of the determined registration relationship. Attached Figure Description

[0029] Figure 1 This is an internal structural diagram of a computer device in one embodiment;

[0030] Figure 2 This is a flowchart illustrating a registration relationship determination method in one embodiment;

[0031] Figure 3 This is a flowchart illustrating the non-rigid transformation of the point cloud to be matched in one embodiment.

[0032] Figure 4 This is a schematic diagram of non-rigid registration of point clouds to be matched in one embodiment;

[0033] Figure 5 This is a flowchart illustrating the process of determining the point cloud to be matched and the reference point cloud in one embodiment;

[0034] Figure 6 This is a flowchart illustrating the registration relationship determination method in another embodiment;

[0035] Figure 7(a) shows the angle error distribution under different registration methods in one embodiment;

[0036] Figure 7(b) shows the distance error distribution under different registration methods in one embodiment;

[0037] Figure 8(a) shows the angle error distribution under different registration methods in another embodiment;

[0038] Figure 8(b) shows the distance error distribution under different registration methods in another embodiment;

[0039] Figure 9 This is a structural block diagram of a registration relationship determination device in one embodiment. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0041] The registration relationship determination method provided in this application embodiment can be applied to, for example, Figure 1 The computer device shown can be a terminal. The computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a registration relationship determination method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0042] Those skilled in the art will understand that Figure 1The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0043] In one embodiment, such as Figure 2 As shown, a method for determining registration relationships is provided, which can be applied to... Figure 1 Taking a computer device as an example, the explanation includes the following steps:

[0044] S210. Obtain the point cloud to be matched and the reference point cloud of the moving object.

[0045] The point cloud to be matched and the reference point cloud are point cloud data of the moving object at different acquisition times. The moving object is the research object that exhibits both rigid and non-rigid motion. For example, the moving object can be a human head with facial expression changes; the movement of the human head is rigid motion, while the changes in facial expression are non-rigid motion.

[0046] It should be noted that due to the motion of the moving object, there is a pose difference between the point cloud to be matched and the reference point cloud. Spatial registration is required to unify the pose for subsequent applications, such as target tracking and 3D reconstruction of the target. Typically, the point cloud to be matched is registered to the reference point cloud.

[0047] Optionally, the point cloud to be matched and the reference point cloud can be obtained by scanning the moving object using a scanning radar. Specifically, the computer device can scan the moving object using its own onboard scanning radar to obtain point cloud data of the moving object. The point cloud data at acquisition time T1 (i.e., the scanning time) can be used as the point cloud to be matched, and the point cloud data at acquisition time T2 (later than acquisition time T1) can be used as the reference point cloud. The computer device can also communicate with the scanning radar to obtain the point cloud to be matched and the reference point cloud of the moving object from the scanning radar.

[0048] S220. Perform spatial registration between the point cloud to be matched and the reference point cloud, and determine the registration relationship between the point cloud to be matched and the reference point cloud with the aim of minimizing the change of the point cloud to be matched under non-rigid registration.

[0049] Spatial registration includes simultaneous rigid and non-rigid registration. Rigid registration corresponds to rigid motion, meaning that the moving object only changes its position during motion, without any transformation in shape or size. The registration relationship obtained through rigid registration can include translation and rotation. Non-rigid registration corresponds to non-rigid motion, meaning that the moving object undergoes more complex transformations during motion. The registration relationship obtained through non-rigid registration can include scaling, affine, and transmission.

[0050] Optionally, rigid and non-rigid registration can be implemented based on the Iterative Closest Point (ICP) algorithm. Rigid registration is used to determine rigid registration parameters T, such as translation and rotation; non-rigid transformation is used to determine non-rigid registration parameters θ, such as the affine transformation matrix. It should be noted that both T and θ are generic terms.

[0051] Optionally, the computer device can simultaneously perform rigid registration and non-rigid registration spatial registration on the point cloud to be matched and the reference point cloud, and determine the rigid registration parameters and non-rigid registration parameters between the point cloud to be matched and the reference point cloud with the constraint of minimizing the change of the point cloud to be matched before and after non-rigid registration, and use the rigid registration parameters and non-rigid registration parameters as the registration relationship between the point cloud to be matched and the reference point cloud.

[0052] In this embodiment, the computer device acquires the point cloud to be matched and the reference point cloud of the moving object, performs spatial registration on the point cloud to be matched and the reference point cloud, and determines the registration relationship between the point cloud to be matched and the reference point cloud by minimizing the change between them under non-rigid registration. The spatial registration includes simultaneous rigid registration and non-rigid registration, realizing joint estimation of the rigid and non-rigid motion of the moving object. This makes the spatial registration process more consistent with the actual motion of the moving object, while simultaneously constraining the point cloud to be matched to minimize the change under non-rigid registration in the spatial registration, improving the estimation accuracy of the rigid and non-rigid motion of the moving object, and thus improving the accuracy of the determined registration relationship.

[0053] To improve registration efficiency, in one embodiment, the above-described S220 involves spatially registering the point cloud to be matched and the reference point cloud, and determining the registration relationship between the point cloud to be matched and the reference point cloud with the constraint of minimizing the change of the point cloud to be matched under non-rigid registration, including:

[0054] The point cloud to be matched and the reference point cloud are input into the registration model for solution to obtain the registration relationship between the point cloud to be matched and the reference point cloud.

[0055] In the registration model, the objective is to match the position of the point cloud to be matched with the reference point cloud after spatial registration, and the constraint is to minimize the change of the point cloud to be matched under non-rigid registration.

[0056] It should be noted that the registration model is a mathematical model used to achieve spatial registration between the point cloud to be matched and the reference point cloud. The process of spatial registration between the point cloud to be matched and the reference point cloud is to determine the registration relationship between the point cloud to be matched and the reference point cloud, and then perform spatial transformation on the point cloud to be matched based on the registration relationship, so that the transformed point cloud to be matched matches the position of the reference point cloud.

[0057] Optionally, the computer device can input the target point cloud and reference point cloud of the moving object into the pre-constructed registration model. The registration model aims to match the positions of the point cloud to be matched and the reference point cloud after spatial registration, and uses the minimum change of the point cloud to be matched under non-rigid registration as the model constraint to solve the problem, thereby realizing the spatial registration between the point cloud to be matched and the reference point cloud and obtaining the registration relationship.

[0058] Optionally, the function solution process of the above registration model is a nonlinear constrained optimization. The rigid registration parameters and non-rigid registration parameters can be obtained by using the Lagrange multiplier method, the enhanced Lagrange multiplier method, and the Alternating Direction Method of Multipliers (ADMM), which serve as the registration relationship between the point cloud to be matched and the reference point cloud.

[0059] In this embodiment, the computer device inputs the point cloud to be matched and the reference point cloud into the registration model for solution, thereby obtaining the registration relationship between the point cloud to be matched and the reference point cloud. The registration model aims to achieve positional matching between the point cloud to be matched and the reference point cloud after spatial registration, and minimizes the change of the point cloud to be matched under non-rigid registration as the model constraint. This method transforms the process of determining the registration relationship into mathematical calculations for constraint solving, significantly improving program execution efficiency, saving registration time, and thus improving registration efficiency.

[0060] Before spatial registration of the point cloud to be matched and the reference point cloud, the aforementioned registration model can be pre-constructed. Based on this, in one embodiment, the method further includes the process of constructing the registration model:

[0061] The model objective is to match the positions of the point cloud to be matched and the reference point cloud after spatial registration, and the model constraint is to minimize the change of the point cloud to be matched under non-rigid registration during spatial registration. A registration model is constructed.

[0062] Optionally, the computer device can predetermine the registration algorithm used for rigid registration and the registration algorithm used for non-rigid registration in spatial registration, and take the matching of the positions of the point cloud to be matched and the reference point cloud after spatial registration as the model objective in the registration model, and take the minimum change of the point cloud to be matched under non-rigid registration in spatial registration as the model constraint in the registration model, and thus construct the registration model.

[0063] In this embodiment, the computer device aims to match the positions of the point cloud to be matched and the reference point cloud after spatial registration, and uses minimizing the changes in the point cloud to be matched under non-rigid registration during spatial registration as a model constraint to construct a registration model. This registration model essentially constrains the non-rigid registration process. For moving objects with both rigid and non-rigid motion, it maximizes the limitation of changes in the point cloud to be matched under non-rigid registration and uses rigid registration for spatial registration. The constraint on non-rigid registration further separates the rigid and non-rigid motion of the moving object, greatly improving the accuracy of the determined registration parameters and thus improving the overall accuracy of the registration relationship.

[0064] In an alternative embodiment, when rigid registration is achieved using the ICP algorithm, the model objective satisfies the following relationship:

[0065] After spatial registration, the sum of the distances between the target point cloud to be matched and the corresponding target reference point cloud meets the preset conditions; the target point cloud to be matched and the target reference point cloud are successfully matched point clouds and reference point clouds by minimizing the sum of their distances.

[0066] Optionally, the computer device pre-matches the point cloud to be matched and the reference point cloud using the ICP algorithm based on the spatial coordinates of the point cloud to be matched and the spatial coordinates of the reference point cloud, so as to determine the point cloud to be matched and the reference point cloud that are successfully matched by minimizing the sum of the distances, and use them as the target point cloud to be matched P and the target reference point cloud Q.

[0067] Optionally, the computer device can construct a model target based on the fact that the sum of the distances between the target point cloud to be matched and the corresponding target reference point cloud after spatial registration meets a preset condition. For example, the preset condition includes that the sum of the distances between the target point cloud to be matched and the corresponding target reference point cloud after spatial registration is minimized, such as the sum of the distances being 0 or less than a preset threshold.

[0068] The model objective satisfies the following equation:

[0069]

[0070] p i p represents a point in the point cloud P to be matched. i ∈P, q i q represents a point in the point cloud Q to be matched. i ∈Q,TΓ9p i |θ) represents the relationship between p i Perform rigid registration (determine rigid registration parameter T) while simultaneously performing non-rigid registration (determine non-rigid registration parameter θ).

[0071] In an alternative embodiment, the model constraints satisfy the following relationship:

[0072] After spatial registration, the sum of the changes in the point cloud to be matched is minimized under non-rigid registration.

[0073] Optionally, the computer device can construct model constraints based on minimizing the positional changes of the point cloud to be matched before and after non-rigid registration in spatial registration. For example, the model constraints can be constructed by minimizing the sum of the distances between the position of the point cloud to be matched before non-rigid registration in spatial registration and the corresponding position of the point cloud to be matched after non-rigid registration in spatial registration.

[0074] The model constraints satisfy the following equation:

[0075]

[0076] Γ(p i |θ) represents the relationship between p i Perform non-rigid registration.

[0077] Optionally, the registration model constructed by combining the above model objectives and model reductions satisfies the following equation:

[0078]

[0079] In this embodiment, the constructed registration model satisfies the relationship that, after spatial registration, the sum of the distances between the target point cloud to be matched and the corresponding target reference point cloud satisfies a preset condition. The model constraints satisfy the relationship that, after spatial registration, the sum of the changed distances of the point cloud to be matched under non-rigid registration is minimized. Here, the target point cloud to be matched and the target reference point cloud are the point cloud to be matched and the reference point cloud that have been successfully paired by minimizing the sum of their distances. The above registration model does not require external hyperparameters to balance the relationship between model constraints and the model target, and the spatial registration process does not require additional parameter tuning, thus improving the applicability of the registration model.

[0080] In one embodiment, where the non-rigid registration parameters θ determined by non-rigid registration include the affine transformation matrix A and the translation matrix t, such as Figure 3 As shown, the sum of the variation distances of the point clouds to be matched under the above non-rigid registration is minimized, including:

[0081] S310. Perform a non-rigid transformation on the point cloud to be matched based on the affine transformation matrix and translation matrix of the auxiliary points of the point cloud to be matched, and obtain the transformed point cloud.

[0082] In this context, auxiliary nodes (also called nodes) are sparsely distributed virtual points pre-defined in space. For example... Figure 4 As shown, each auxiliary node g j Corresponding to an affine transformation matrix A j and a translation matrix t j A j∈R 3×3 , t j ∈R 3 .

[0083] Optionally, for any point in the point cloud to be matched, the computer device can obtain a preset number of auxiliary nodes that are closest to that point, and use them as auxiliary nodes corresponding to that point. For example... Figure 4 As shown, for any point p in the point cloud P to be matched i The computer equipment can search for the corresponding point p. i The four auxiliary nodes g j .

[0084] Optionally, after obtaining the affine transformation matrix and translation matrix of each auxiliary node, the computer device can perform a non-rigid transformation on the point cloud to be matched based on the affine transformation matrix and translation matrix of the auxiliary points to obtain the transformed point cloud. Specifically, for non-rigid registration Γ(p i |A1,t1…A m ,t m To match point p in point cloud P i The process of performing a non-rigid transformation satisfies the following equation, p i The result can be obtained after this non-rigid transformation. This results in the formation of a transformed point cloud.

[0085]

[0086] Among them, w j Represents weight. An auxiliary node g j Corresponding to a weight w j .

[0087] S320. Under non-rigid registration, the sum of the change distances between the point cloud to be matched and the transformed point cloud is minimized.

[0088] Optionally, in the case of achieving non-rigid registration, the sum of the change distances between the point cloud to be matched before the non-rigid transformation and the transformed point cloud obtained by the non-rigid transformation is minimized.

[0089] For example, in non-rigid registration Γ(p) i |A1,t1…A m ,t m In the case of ), the above registration model can be implemented as follows:

[0090]

[0091] in,

[0092]

[0093] Rot(Aj )=(c1·c2) 2 +(c1·c3) 2 +(c2·c3) 2 +(c1·c1-1) 2 +(c²·c²-1) 2

[0094] +(c3·c3-1) 2

[0095]

[0096]

[0097]

[0098] Where c1~c3 represent the affine transformation matrix A j The column vector, α jk The weight representing consistency between nodes is usually set to 1; [·, 1] The formula converts 3×1 points into a homogeneous coordinate system. E rot and E reg It is a regularization term used to constrain the deformation field and make its deformation smooth.

[0099] In this embodiment, the Lagrange multiplier method can be used to solve the above registration model to obtain the translation matrix t. j and affine transformation matrix A j .

[0100] In this embodiment, when the non-rigid registration parameters determined by non-rigid registration include affine transformation matrices and translation matrices, the computer device performs a non-rigid transformation on the point cloud to be matched based on the affine transformation matrices and translation matrices of the auxiliary points of the point cloud to be matched, obtaining a transformed point cloud. This achieves the minimum sum of the change distances between the point cloud to be matched and the transformed point cloud under non-rigid registration. In the above method, using the minimum sum of the change distances between the point cloud to be matched and the transformed point cloud under non-rigid registration as a model constraint maximizes the restriction of the changes in the point cloud to be matched under non-rigid registration, thus separating the rigid motion and non-rigid motion of the moving object, improving the accuracy of non-rigid registration, and correspondingly improving the accuracy of the determined affine transformation matrices and translation matrices.

[0101] When applied to scenarios involving tracking moving objects, in one embodiment, such as Figure 5 As shown, S210 above, acquiring the point cloud to be matched and the reference point cloud of the moving object, includes:

[0102] S510. Acquire tracking images of the moving object and acquire at least two depth images based on the tracking images.

[0103] Depth images, also known as range images, are images that use the distance (depth) between the image acquisition device and each point on the research object as pixel values, directly reflecting the geometry of the visible surface of the research object.

[0104] Optionally, the computer device can use its built-in depth camera to track and photograph a moving object, acquiring depth images of the moving object for a preset duration to form a tracking image. Alternatively, the computer device can communicate with a depth camera, which tracks and photographs the moving object, capturing depth images for a preset duration to form a tracking image, which the computer device then acquires from the depth camera.

[0105] It should be noted that during the tracking of a moving object, the moving object undergoes both rigid and non-rigid motion randomly.

[0106] Optionally, the computer device can acquire at least two depth images from the tracking image of the moving object to determine the point cloud to be matched and the reference point cloud. For example, the computer device can randomly acquire two depth images from the tracking image to determine the point cloud to be matched and the reference point cloud, or it can acquire multiple consecutive depth images at the acquisition time to determine the point cloud to be matched and the reference point cloud.

[0107] S520. Determine the point cloud to be matched and the reference point cloud based on at least two depth images.

[0108] Optionally, the computer device converts the at least two acquired depth images into a point cloud to be matched and a reference point cloud based on a predetermined conversion relationship between the depth images and point cloud data.

[0109] Optionally, the computer can acquire two depth images from the tracking image, and convert the depth image acquired earlier into point cloud data as the point cloud to be matched, based on the conversion relationship between the depth image and the point cloud data, and convert the depth image acquired later into non-point cloud data as the reference point cloud.

[0110] Optionally, the computer can acquire multiple temporally consecutive depth images from the tracking image, merge the multiple temporally consecutive depth images to form a keyframe depth image, convert the keyframe depth image into point cloud data according to the conversion relationship between the depth image and point cloud data, use it as the point cloud to be matched, and convert the subsequently acquired depth images into point cloud data, use it as the reference point cloud.

[0111] In this embodiment, the computer device acquires tracking images of the moving object and obtains at least two depth images based on the tracking images to determine the point cloud to be matched and the reference point cloud. This method enables the acquisition of the point cloud to be matched and the target point cloud during the tracking of the moving object, allowing the registration relationship determination method to be applied in tracking scenarios and helping to improve the accuracy and precision of tracking.

[0112] To facilitate understanding by those skilled in the art, in one embodiment, the registration relationship determination method provided in this application will be described in detail below, such as... Figure 6 As shown, the method may include:

[0113] S601. After spatial registration, the sum of the distances between the target point cloud to be matched and the corresponding target reference point cloud that meets the preset conditions is the model target; the target point cloud to be matched and the target reference point cloud are the point cloud to be matched and the reference point cloud that are successfully matched by minimizing the sum of the distances.

[0114] S602. The model constraint is to minimize the sum of the changes in the point cloud to be matched under non-rigid registration after spatial registration.

[0115] S603. Construct a registration model based on the model objectives and model constraints;

[0116] S604. Acquire tracking images of the moving object, and acquire at least two depth images based on the tracking images;

[0117] S605. Determine the point cloud to be matched and the reference point cloud based on at least two depth images;

[0118] S606. Input the point cloud to be matched and the reference point cloud into the registration model and solve it to obtain the registration relationship between the point cloud to be matched and the reference point cloud.

[0119] It should be noted that the descriptions in S601-S606 above can be found in the relevant descriptions in the above embodiments, and their effects are similar, so they will not be repeated here.

[0120] Taking the tracking of a moving object's head by a depth camera as an example, the registration relationship obtained by the above registration relationship determination method and the ICP algorithm in related technologies can be used to spatially transform the tracked keyframe point cloud (composed of multiple frames) to determine the registration error between the transformed keyframe point cloud and the current frame point cloud. The registration errors generated by the two methods are compared to verify the accuracy of the registration relationship obtained by the above registration relationship determination method.

[0121] Figures 7(a) to 7(b)This represents the angle error and distance error determined by the two methods mentioned above for the head of the moving object in case 1 (i.e., non-rigid motion without rigid motion but with changes in facial expression); Figures 8(a) to 8(b) This represents the angle and distance errors determined by the two methods described above under Case 2 (i.e., simulating rigid motion with non-rigid motion involving facial expression changes). Solid lines represent the registration errors generated by registration based on the registration relationships obtained using the methods described above, while dashed lines represent the registration errors generated by row registration based on the ICP algorithm.

[0122] As shown in the figure, in most cases, the angle and distance errors produced by registration based on the registration relationship determined by the above method are smaller than those produced by registration based on the ICP algorithm. Furthermore, it can be seen that the moving object undergoes facial expression changes (such as deliberately frowning violently) in frames 3700-3900, 4720-4920, and 6200-6400. In these three facial expression changes, the registration process based on the registration relationship determined by the above method isolates the rigid body motion, resulting in a registration error far smaller than that of the ICP algorithm.

[0123] right Figures 7(a) to 7(b) as well as Figures 8(a) to 8(b) After organizing the data, the mean error under different conditions is shown in the table below:

[0124]

[0125] The results show that the average registration error generated by registration based on the registration relationship determined by the above method is less than the average registration error generated by registration based on the ICP algorithm.

[0126] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0127] Based on the same inventive concept, this application also provides a registration relationship determination apparatus for implementing the registration relationship determination method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more registration relationship determination apparatus embodiments provided below can be found in the limitations of the registration relationship determination method described above, and will not be repeated here.

[0128] In one embodiment, such as Figure 9 As shown, a registration relationship determination device is provided, including: a point cloud acquisition module 901 and a target registration module 902, wherein:

[0129] Point cloud acquisition module 901 is used to acquire the point cloud to be matched and the reference point cloud of the moving object;

[0130] The target registration module 902 is used to perform spatial registration between the point cloud to be matched and the reference point cloud, and determines the registration relationship between the point cloud to be matched and the reference point cloud by taking the minimum change of the point cloud to be matched under non-rigid registration as a constraint; wherein, spatial registration includes rigid registration and non-rigid registration performed simultaneously.

[0131] In one embodiment, the target registration module 902 includes:

[0132] The solver submodule is used to input the point cloud to be matched and the reference point cloud into the registration model for solving, and obtain the registration relationship between the point cloud to be matched and the reference point cloud. The registration model aims to match the positions of the point cloud to be matched and the reference point cloud after spatial registration, and the model constraint is to minimize the change of the point cloud to be matched under non-rigid registration.

[0133] In one embodiment, the above-described apparatus further includes: a model building module;

[0134] The model building module is used to construct a registration model with the goal of matching the positions of the point cloud to be matched and the reference point cloud after spatial registration, and with the constraint of minimizing the changes of the point cloud to be matched under non-rigid registration in spatial registration.

[0135] In one embodiment, the model objective satisfies the following relationship:

[0136] After spatial registration, the sum of the distances between the target point cloud to be matched and the corresponding target reference point cloud meets the preset conditions; the target point cloud to be matched and the target reference point cloud are successfully matched point clouds and reference point clouds by minimizing the sum of their distances.

[0137] In one embodiment, the model constraints satisfy the following relationship:

[0138] After spatial registration, the sum of the changes in the point cloud to be matched under non-rigid registration is minimized.

[0139] In one embodiment, the model constraints satisfy the following relationship:

[0140] A non-rigid transformation is performed on the point cloud to be matched based on the affine transformation matrix and translation matrix of the auxiliary points of the point cloud to be matched, resulting in the transformed point cloud. Under non-rigid registration, the sum of the transformation distances between the point cloud to be matched and the transformed point cloud is minimized.

[0141] In one embodiment, the point cloud acquisition module 901 includes:

[0142] The depth submodule is used to acquire tracking images of moving objects and obtain at least two depth images based on the tracking images;

[0143] The transformation submodule is used to determine the point cloud to be matched and the reference point cloud based on at least two frames of depth images.

[0144] Each module in the above-mentioned registration relationship determination device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0145] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0146] Acquire the point cloud to be matched and the reference point cloud of the moving object; perform spatial registration on the point cloud to be matched and the reference point cloud, and determine the registration relationship between the point cloud to be matched and the reference point cloud by taking the minimum change of the point cloud to be matched under non-rigid registration as a constraint; wherein, spatial registration includes rigid registration and non-rigid registration performed simultaneously.

[0147] In one embodiment, the processor further performs the following steps when executing the computer program:

[0148] The point cloud to be matched and the reference point cloud are input into the registration model for solution to obtain the registration relationship between the point cloud to be matched and the reference point cloud. The registration model aims to match the positions of the point cloud to be matched and the reference point cloud after spatial registration, and minimizes the change of the point cloud to be matched under non-rigid registration as the model constraint.

[0149] In one embodiment, the processor further performs the following steps when executing the computer program:

[0150] The model objective is to match the positions of the point cloud to be matched and the reference point cloud after spatial registration, and the model constraint is to minimize the change of the point cloud to be matched under non-rigid registration in spatial registration. A registration model is constructed.

[0151] In one embodiment, the model objective satisfies the following relationship:

[0152] After spatial registration, the sum of the distances between the target point cloud to be matched and the corresponding target reference point cloud meets the preset conditions; the target point cloud to be matched and the target reference point cloud are successfully matched point clouds and reference point clouds by minimizing the sum of their distances.

[0153] In one embodiment, the model constraints satisfy the following relationship:

[0154] After spatial registration, the sum of the changes in the point cloud to be matched under non-rigid registration is minimized.

[0155] In one embodiment, the model constraints satisfy the following relationship:

[0156] A non-rigid transformation is performed on the point cloud to be matched based on the affine transformation matrix and translation matrix of the auxiliary points of the point cloud to be matched, resulting in the transformed point cloud. Under non-rigid registration, the sum of the transformation distances between the point cloud to be matched and the transformed point cloud is minimized.

[0157] In one embodiment, the processor further performs the following steps when executing the computer program:

[0158] Acquire tracking images of the moving object and determine the point cloud to be matched and the reference point cloud based on at least two depth images in the tracking images.

[0159] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0160] Acquire the point cloud to be matched and the reference point cloud of the moving object; perform spatial registration on the point cloud to be matched and the reference point cloud, and determine the registration relationship between the point cloud to be matched and the reference point cloud by taking the minimum change of the point cloud to be matched under non-rigid registration as a constraint; wherein, spatial registration includes rigid registration and non-rigid registration performed simultaneously.

[0161] In one embodiment, when the computer program is executed by the processor, it further performs the following steps:

[0162] The point cloud to be matched and the reference point cloud are input into the registration model for solution to obtain the registration relationship between the point cloud to be matched and the reference point cloud. The registration model aims to match the positions of the point cloud to be matched and the reference point cloud after spatial registration, and minimizes the change of the point cloud to be matched under non-rigid registration as the model constraint.

[0163] In one embodiment, when the computer program is executed by the processor, it further performs the following steps:

[0164] The model objective is to match the positions of the point cloud to be matched and the reference point cloud after spatial registration, and the model constraint is to minimize the change of the point cloud to be matched under non-rigid registration in spatial registration. A registration model is constructed.

[0165] In one embodiment, the model objective satisfies the following relationship:

[0166] After spatial registration, the sum of the distances between the target point cloud to be matched and the corresponding target reference point cloud meets the preset conditions; the target point cloud to be matched and the target reference point cloud are successfully matched point clouds and reference point clouds by minimizing the sum of their distances.

[0167] In one embodiment, the model constraints satisfy the following relationship:

[0168] After spatial registration, the sum of the changes in the point cloud to be matched under non-rigid registration is minimized.

[0169] In one embodiment, the model constraints satisfy the following relationship:

[0170] A non-rigid transformation is performed on the point cloud to be matched based on the affine transformation matrix and translation matrix of the auxiliary points of the point cloud to be matched, resulting in the transformed point cloud. Under non-rigid registration, the sum of the transformation distances between the point cloud to be matched and the transformed point cloud is minimized.

[0171] In one embodiment, when the computer program is executed by the processor, it further performs the following steps:

[0172] Acquire tracking images of the moving object, and acquire at least two depth images based on the tracking images; determine the point cloud to be matched and the reference point cloud based on the at least two depth images.

[0173] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0174] Acquire the point cloud to be matched and the reference point cloud of the moving object; perform spatial registration on the point cloud to be matched and the reference point cloud, and determine the registration relationship between the point cloud to be matched and the reference point cloud by taking the minimum change of the point cloud to be matched under non-rigid registration as a constraint; wherein, spatial registration includes rigid registration and non-rigid registration performed simultaneously.

[0175] In one embodiment, when the computer program is executed by the processor, it further performs the following steps:

[0176] The point cloud to be matched and the reference point cloud are input into the registration model for solution to obtain the registration relationship between the point cloud to be matched and the reference point cloud. The registration model aims to match the positions of the point cloud to be matched and the reference point cloud after spatial registration, and minimizes the change of the point cloud to be matched under non-rigid registration as the model constraint.

[0177] In one embodiment, when the computer program is executed by the processor, it further performs the following steps:

[0178] The model objective is to match the positions of the point cloud to be matched and the reference point cloud after spatial registration, and the model constraint is to minimize the change of the point cloud to be matched under non-rigid registration in spatial registration. A registration model is constructed.

[0179] In one embodiment, the model objective satisfies the following relationship:

[0180] After spatial registration, the sum of the distances between the target point cloud to be matched and the corresponding target reference point cloud meets the preset conditions; the target point cloud to be matched and the target reference point cloud are successfully matched point clouds and reference point clouds by minimizing the sum of their distances.

[0181] In one embodiment, the model constraints satisfy the following relationship:

[0182] After spatial registration, the sum of the changes in the point cloud to be matched under non-rigid registration is minimized.

[0183] In one embodiment, the model constraints satisfy the following relationship:

[0184] A non-rigid transformation is performed on the point cloud to be matched based on the affine transformation matrix and translation matrix of the auxiliary points of the point cloud to be matched, resulting in the transformed point cloud. Under non-rigid registration, the sum of the transformation distances between the point cloud to be matched and the transformed point cloud is minimized.

[0185] In one embodiment, when the computer program is executed by the processor, it further performs the following steps:

[0186] Acquire tracking images of the moving object, and acquire at least two depth images based on the tracking images; determine the point cloud to be matched and the reference point cloud based on the at least two depth images.

[0187] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0188] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0189] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for determining registration relationships, characterized in that, The method includes: Acquire multiple temporally consecutive depth images from the tracking image of a moving object, and merge the multiple temporally consecutive depth images to form a keyframe depth image; Based on the conversion relationship between depth images and point cloud data, the keyframe depth images are converted into point cloud data as point clouds to be matched, and the subsequently acquired depth images are converted into point cloud data as reference point clouds; the point clouds to be matched and the reference point clouds are point cloud data of the moving object at different acquisition times; the moving object exhibits both rigid and non-rigid motion. The point cloud to be matched and the reference point cloud are input into the registration model for solution to obtain the registration relationship between the point cloud to be matched and the reference point cloud; wherein, the registration model takes the matching of the positions of the point cloud to be matched and the reference point cloud after spatial registration as the model objective, and takes the minimum change of the point cloud to be matched under non-rigid registration as the model constraint; the spatial registration includes rigid registration and non-rigid registration performed simultaneously.

2. The method according to claim 1, characterized in that, The method further includes: The registration model is constructed with the goal of matching the positions of the point cloud to be matched and the reference point cloud after spatial registration, and with the constraint of minimizing the change of the point cloud to be matched under non-rigid registration in the spatial registration.

3. The method according to claim 2, characterized in that, The model objective satisfies the following relationship: After spatial registration, the sum of the distances between the target point cloud to be matched and the corresponding target reference point cloud satisfies a preset condition; the target point cloud to be matched and the target reference point cloud are point clouds to be matched and reference point clouds that have been successfully paired by minimizing the sum of their distances.

4. The method according to claim 2, characterized in that, The model constraints satisfy the following relationship: After spatial registration, the sum of the variation distances of the point cloud to be matched under the non-rigid registration is minimized.

5. The method according to claim 4, characterized in that, The sum of the variation distances of the point cloud to be matched under the non-rigid registration is minimized, including: The point cloud to be matched is subjected to a non-rigid transformation based on the affine transformation matrix and translation matrix corresponding to the auxiliary points of the point cloud to be matched, so as to obtain the transformed point cloud. Under the non-rigid registration, the sum of the change distances between the point cloud to be matched and the transformed point cloud is minimized.

6. A registration relationship determination device, characterized in that, The device includes: The point cloud acquisition module is used to acquire multiple temporally consecutive depth images from the tracking image of a moving object, merge the multiple temporally consecutive depth images to form a keyframe depth image, convert the keyframe depth image into point cloud data according to the conversion relationship between depth image and point cloud data, and use it as the point cloud to be matched, and convert the subsequently acquired depth image into point cloud data as the reference point cloud; the point cloud to be matched and the reference point cloud are the point cloud data of the moving object at different acquisition times; the moving object exhibits both rigid motion and non-rigid motion; The spatial registration module is used to input the point cloud to be matched and the reference point cloud into the registration model for solution, so as to obtain the registration relationship between the point cloud to be matched and the reference point cloud; wherein, the registration model takes the matching of the positions of the point cloud to be matched and the reference point cloud after spatial registration as the model objective, and takes the minimization of the change of the point cloud to be matched under non-rigid registration as the model constraint; the spatial registration includes rigid registration and non-rigid registration performed simultaneously.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images