Method and apparatus for generating three-dimensional image data, and electronic device and storage medium

By performing rigid and non-rigid alignment on multi-frame 3D image data of the target object, and utilizing facial key points and segmentation masks, the problem of inconsistent topological structures in 3D scanning equipment was solved, achieving efficient and accurate alignment and representation of 3D image data.

WO2024245278A1PCT designated stage expired Publication Date: 2024-12-05BEIJING ZITIAO NETWORK TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2024/096021
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-30
Filing Date
2024-05-29
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

In existing technologies, when 3D scanning devices scan a target object, the inconsistent topological structure of the 3D mesh sequence caused by changes in the state of the target object over a period of time affects the topological consistency of subsequent applications.

Method used

By acquiring multi-frame 3D image data of the target object, rigid and non-rigid alignment is performed. Facial key points and segmentation masks are used for preprocessing. Deformation parameters are optimized by combining the ICP method and the target loss function to achieve topological consistency alignment of 3D image data.

Benefits of technology

It improves the topological consistency and alignment accuracy of 3D image data, ensures the continuous representation of the morphological changes of the target object over a time period, and enhances the accuracy and efficiency of subsequent applications.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024096021_05122024_PF_FP_ABST
    Figure CN2024096021_05122024_PF_FP_ABST
Patent Text Reader

Abstract

The present disclosure relates to a method and apparatus for generating three-dimensional image data, and an electronic device and a storage medium. The method comprises: acquiring N frames of three-dimensional image data to be processed of a target object; performing rigid alignment on reference three-dimensional image data on the basis of a first frame of three-dimensional image data to be processed among the N frames of three-dimensional image data to be processed; and on the basis of the N frames of three-dimensional image data to be processed, performing non-rigid alignment on the reference three-dimensional image data, which has been subjected to rigid alignment, so as to determine a target three-dimensional image data sequence of the target object.
Need to check novelty before this filing date? Find Prior Art

Description

Method, device, electronic device and storage medium for generating three-dimensional image data

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on the Chinese application with application number 202310627858.X and application date May 30, 2023, and claims its priority. The disclosed content of the Chinese application is hereby introduced as a whole into this application. Technical Field

[0003] The present disclosure relates to the field of computer vision technology, and in particular to a method, device, electronic device, and storage medium for generating three-dimensional image data. Background Art

[0004] Three-dimensional (3D) reconstruction involves reconstructing a 3D model from one or more two-dimensional (2D) images. The use of 3D scanning technology to create realistic models of target objects has become mainstream in related industries, and as the price of 3D scanning hardware decreases, this technology will become increasingly popular.

[0005] Summary of the Invention

[0006] The present disclosure provides a method, device, electronic device, and storage medium for generating three-dimensional image data.

[0007] According to a first aspect of the present disclosure, a method for generating three-dimensional image data is provided, the method comprising:

[0008] Acquire N frames of to-be-processed three-dimensional image data of a target object; the N frames of to-be-processed three-dimensional image data include a sequence of to-be-processed three-dimensional image data of a target portion of the target object obtained within a target time period; wherein N is a positive integer greater than 1;

[0009] performing rigid alignment on the reference three-dimensional image data based on a first frame of the N frames of three-dimensional image data to be processed;

[0010] Based on the N frames of 3D image data to be processed, non-rigid alignment is performed on the rigidly aligned reference 3D image data to determine a target 3D image data sequence of the target object; wherein, based on the N frames of 3D image data to be processed, performing non-rigid alignment on the rigidly aligned reference 3D image data includes, based on the i-th frame of 3D image data to be processed among the N frames of 3D image data to be processed, performing non-rigid alignment on the reference 3D image data that has been aligned with the i-1-th frame of 3D image data to be processed, to obtain reference 3D image data aligned with the i-th frame of 3D image data to be processed; wherein i is a positive integer greater than 1.

[0011] According to a second aspect of the present disclosure, a device for generating three-dimensional image data is provided, characterized in that the device includes:

[0012] a data acquisition module configured to acquire N frames of to-be-processed three-dimensional image data of a target object; the N frames of to-be-processed three-dimensional image data comprising a sequence of to-be-processed three-dimensional image data of a target portion of the target object acquired within a target time period; wherein N is a positive integer greater than 1;

[0013] A rigid alignment module, configured to perform rigid alignment on the reference three-dimensional image data based on a first frame of the N frames of three-dimensional image data to be processed;

[0014] A non-rigid alignment module is configured to perform non-rigid alignment on rigidly aligned reference three-dimensional image data based on the N frames of to-be-processed three-dimensional image data to determine a target three-dimensional image data sequence of the target object; wherein, performing non-rigid alignment on rigidly aligned reference three-dimensional image data based on the N frames of to-be-processed three-dimensional image data includes, based on the i-th frame of to-be-processed three-dimensional image data among the N frames of to-be-processed three-dimensional image data, performing non-rigid alignment on reference three-dimensional image data that has been aligned with the i-1-th frame of to-be-processed three-dimensional image data to obtain reference three-dimensional image data aligned with the i-th frame of to-be-processed three-dimensional image data; wherein i is a positive integer greater than 1.

[0015] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above method when executing the program.

[0016] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above method of the present disclosure is implemented.

[0017] The disclosed embodiments provide a method, apparatus, electronic device, and storage medium for generating three-dimensional image data. These methods obtain N frames of to-be-processed three-dimensional image data of a target object, rigidly align reference three-dimensional image data based on the first frame of the N frames of to-be-processed three-dimensional image data, and non-rigidly align the rigidly aligned reference three-dimensional image data based on the N frames of to-be-processed three-dimensional image data to determine a target three-dimensional image data sequence for the target object. During the alignment process, the disclosed embodiments first rigidly align the reference three-dimensional image data with the first frame of the first frame of to-be-processed three-dimensional image data. Based on this rigid alignment, the rigidly aligned reference three-dimensional image data is non-rigidly aligned based on the N frames of to-be-processed three-dimensional image data to obtain a target three-dimensional image data sequence for the target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0019] FIG1 is a flowchart of a method for generating three-dimensional image data provided by an exemplary embodiment of the present disclosure;

[0020] FIG2 is a flow chart of step S120 in FIG1 ;

[0021] FIG3 is a schematic block diagram of functional modules of a device for generating three-dimensional image data provided by an exemplary embodiment of the present disclosure;

[0022] FIG4 is a structural block diagram of an electronic device provided by an exemplary embodiment of the present disclosure;

[0023] FIG5 is a structural block diagram of a computer system provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0025] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0026] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0027] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0028] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0029] In related technologies, a 3D scanning device is used to scan a target object, and after a period of scanning, a 3D mesh sequence of the target object can be obtained. However, since the 3D mesh sequence is obtained over a period of time, and the state of the target object may change during this period, the topological structure of the 3D mesh corresponding to different moments in the period may be inconsistent, which is not conducive to the subsequent practical application of the 3D mesh sequence.

[0030] To address the above-mentioned issues, the present disclosure provides a method for generating three-dimensional image data. In an embodiment provided herein, a target portion of a target object can be scanned using a three-dimensional scanning device. The target object can be a person, for example, the target portion can be a person's face or hand. The embodiment uses a face as an example, but the embodiment is not limited thereto. Multiple frames of to-be-processed three-dimensional image data of the person's face are obtained, and corresponding multiple frames of images are obtained. These multiple frames of images can be captured by a camera. In an embodiment, the multiple frames of to-be-processed three-dimensional image data and the multiple frames of images can have a multi-frame count greater than or equal to 2. In an embodiment, the multi-frame count refers to to-be-processed three-dimensional image data or images acquired over a period of time with a temporal relationship. The image format can be RGB (RGB color mode), and the target object can be a specific person. In an embodiment, the to-be-processed three-dimensional image data can be three-dimensional data describing the target portion of the target object using a set of vertices and polygons of a polyhedron. The polygons can be at least one of triangles, quadrilaterals, or other simple convex polygons. Furthermore, the to-be-processed three-dimensional image data can also include three-dimensional data composed of ordinary polygons with holes, but the embodiment is not limited thereto.

[0031] The multiple frames of 3D image data to be processed obtained by scanning with a 3D scanning device may scan out non-target parts of the human body such as clothes, collars and hair during the scanning process. Therefore, these multiple frames of 3D image data to be processed need to be preprocessed to remove these parts.

[0032] In this embodiment, facial key points and a facial segmentation mask are extracted from the above-mentioned image, and the facial key point vertices are marked on the reference 3D image data. Key points are local feature points on the face, and can be feature points at specific facial features such as the eyes, nose, mouth, and chin, or at specific details of the face such as the eye sockets and nose wings. The present embodiment does not limit the number of facial key points, and can be 49, 68, or 81, for example. The embodiment uses 68 facial key points as an example for illustration. Alternatively, the reference 3D image data can be a standard face model from the 3DMM (3D Morphable Model) library. It should be noted that the present embodiment uses a human as the target object and a face as the target part for illustrative purposes. The target object or target part can also be other objects, such as a living being or an object, and the target part can also be other parts, and the embodiment is not limited to this. In the case of other target objects or target parts, key points and a segmentation mask for the target part area can also be extracted based on their local features, thereby aligning the corresponding reference 3D image data with multiple frames of to-be-processed 3D image data corresponding to the target part.

[0033] In this embodiment, for each frame of image, a face segmentation model can be used to segment the face region from the image to obtain a face segmentation mask. A certain number of facial key points are extracted from the image and correspondingly marked on the reference 3D image data. In this way, the facial key points can be used to align some frames of the multiple frames of 3D image data to be processed with the reference 3D image data. For example, the reference 3D image data can be aligned based on the first frame of the multiple frames of 3D image data to be processed.

[0034] For example, taking the number of facial key points as 68, in the benchmark 3D image data M t Mark the key vertex P of the face * ∈R 68×3 , for the i-th frame image I i ∈R H×W×3 , by extracting facial key points P i ∈R 68×3 And obtain the face segmentation mask S i ∈R H×W , and process the multiple frames of the target object's three-dimensional image data to be processed obtained by scanning, and remove the parts that do not belong to the face. For example, for each frame of the three-dimensional image data M to be processed i , the three-dimensional image data M to be processed i The vertices are projected to the three-dimensional image data M to be processed through the projection matrix i The corresponding image I i The image coordinate system where it is located, and by using each frame image I iThe corresponding face segmentation mask S i , determine each frame image I i The non-face area in the 3D image data M i The vertices projected in the non-face area are removed, and the processed three-dimensional image data can be recorded as Where H and W represent the height and width of the image respectively, and i is a positive integer.

[0035] After preprocessing the multiple frames of initial to-be-processed three-dimensional image data of the scanned target object in the above manner, multiple frames of to-be-processed three-dimensional image data of the target object are obtained. Based on the facial key points extracted from each frame of the image, the reference three-dimensional image data is rigidly aligned based on the first frame of to-be-processed three-dimensional image data among these multiple frames of to-be-processed three-dimensional image data.

[0036] Specifically, rigid alignment can be performed using existing methods, for example, the ICP (Iterative Closest Point) method can be used to rigidly align the baseline three-dimensional image data marked with facial key points with the first frame of the target object's processed three-dimensional image data, and obtain standard three-dimensional image data after rigid alignment, such as a human head model, etc. The embodiment is not limited to this.

[0037] For example, by using the corresponding facial key points, the scale s, rotation R and translation T are obtained so that the reference three-dimensional image data M t The three-dimensional image data to be processed can be obtained after the above processing Roughly aligned, it can be optimized using the following formula.

[0038] Where s is a scalar, R is a 3×3 matrix, and T is a 1×3 vector; As the three-dimensional face key points, obtain the face key points of the first frame image I1, and pass the first frame of the three-dimensional image data to be processed The vertices of the projected points are projected into the above-mentioned image coordinate system, and the projection points closest to the key points of each face in the first frame image I1 are determined from the projection points obtained after the projection, and the three-dimensional image data to be processed corresponding to these closest projection points are The vertex is P1 3d .

[0039] Through the rigid alignment, the base three-dimensional image data is deformed to a large extent, so that the base three-dimensional image data can be basically aligned with the first frame of to-be-processed three-dimensional image data of the target object.

[0040] In an embodiment, rigid alignment can be performed on the reference 3D image data based on the first frame of 3D image data to be processed among multiple frames of 3D image data to be processed of the target object. The alignment result of the rigid alignment between the reference 3D image data and the first frame of 3D image data to be processed is then used to perform non-rigid alignment on the rigidly aligned reference 3D image data based on the first frame of 3D image data to be processed. During the rigid alignment of the first frame of 3D image data to be processed with the reference 3D image data, the reference 3D image data will be significantly deformed. When the non-rigid alignment of the rigidly aligned reference 3D image data using the first frame of 3D image data to be processed is performed, the deformation of the rigidly aligned reference 3D image data will be reduced.

[0041] After rigidly aligning the reference 3D image data based on the facial key points of the target object, the rigidly aligned reference 3D image data can be used to perform non-rigid alignment with multiple frames of 3D image data to be processed.

[0042] Specifically, first, for the first frame of three-dimensional image data to be processed among the multiple frames of three-dimensional image data to be processed of the target object, non-rigid alignment is performed on the rigidly aligned reference three-dimensional image data based on the first frame of three-dimensional image data to be processed. Using the alignment result of the rigidly aligned reference three-dimensional image data and the first frame of three-dimensional image data to be processed, non-rigid alignment is performed on the aligned reference three-dimensional image data based on the second frame of three-dimensional image data to be processed. During the rigid alignment of the reference three-dimensional image data and the first frame of three-dimensional image data to be processed, the reference three-dimensional image data will be deformed to a large extent. By using the alignment result of the previous frame of three-dimensional image data to be processed, and then performing non-rigid alignment on the reference three-dimensional image data that has been aligned with the previous frame of three-dimensional image data based on the current frame of three-dimensional image data, alignment can be achieved with a smaller deformation. In this way, the current frame of 3D image data to be processed is used to perform non-rigid alignment on the reference 3D image data that has been aligned with the previous frame of 3D image data to be processed, until the non-rigid alignment based on the last frame of 3D image data to be processed is completed. This allows the use of the alignment result of the previous frame, and on this basis, further deformation of the reference 3D image data aligned with the previous frame to make it fit the current frame of 3D image data to be processed, thereby ensuring the topological consistency of the target 3D image data sequence finally obtained.

[0043] It should be noted that the multiple frames of to-be-processed 3D image data in the embodiments are sequences of to-be-processed 3D image data obtained by scanning a target portion of a target object within a time period. This sequence of to-be-processed 3D image data includes time sequence information. For example, when the target portion is a human face, the multiple frames of to-be-processed 3D image data can represent the process of facial expression changes of the target object within a time period. The previous frame of to-be-processed 3D image data and the current frame of to-be-processed 3D image data have a time sequence relationship. Therefore, based on the fact that the multiple frames of to-be-processed 3D image data represent the target portion of the same target object and have a time sequence relationship, the multiple frames of to-be-processed 3D image data can represent the continuous changes in the contour morphology of the same target portion over a period of time. In particular, the contour morphologies represented by adjacent frames of to-be-processed 3D image data are relatively similar. Therefore, the alignment result of the previous frame of to-be-processed 3D image data can be used to further align the baseline 3D image data that has been aligned with the previous frame of to-be-processed 3D image data, thereby obtaining the alignment result corresponding to the current frame of to-be-processed 3D image data. In the embodiments, only the first frame of to-be-processed 3D image data needs to be used to rigidly align the baseline 3D image data; there is no need to rigidly align each frame of to-be-processed 3D image data with the baseline 3D image data. Starting with the second frame of to-be-processed 3D image data, the alignment of the previously aligned reference 3D image data is performed based on alignment with the previous frame of to-be-processed 3D image data, resulting in a consistent topological structure for the resulting target 3D image data sequence. Furthermore, during the rigid alignment of the reference 3D image data using the first frame of to-be-processed 3D image data, the reference 3D image data undergoes significant deformation, but basic alignment of the reference 3D image data is already achieved. Subsequent alignment of the target 3D image data sequence using each frame of to-be-processed 3D image data requires only minor deformation of the rigidly aligned reference 3D image data, improving overall alignment efficiency.

[0044] In an embodiment, multiple frames of to-be-processed 3D image data are obtained by scanning a target object, and rigid alignment is performed on the reference 3D image data based on the first frame of to-be-processed 3D image data. Non-rigid alignment is then performed on the basis of the rigid alignment, and the non-rigid alignment result of the previous frame is utilized. On this basis, the reference 3D image data that has been aligned with the previous frame of to-be-processed 3D image data is non-rigidly aligned using the current frame of to-be-processed 3D image data. For the reference 3D image data that has been rigidly aligned with the first frame of to-be-processed 3D image data, non-rigid alignment is performed again on the rigidly aligned reference 3D image data based on the first frame of to-be-processed 3D image data, so that the reference 3D image data that has undergone the above-mentioned rigid alignment and non-rigid alignment can be made to fit more closely with the first frame of to-be-processed 3D image data. For the target 3D image data corresponding to the 3D image data to be processed of the remaining frames, since it is further deformed based on the non-rigid alignment result of the previous frame, it can also better fit the 3D image data to be processed of the remaining frames.

[0045] Specifically, for N v The rigidly aligned 3D image data vertices By optimizing the deformation parameters and translation parameters By constructing the target loss function and using the Adam optimizer to optimize the deformation parameter A and translation parameter b. Among them, the target loss function L ` Specifically:

[0046] Among them, for the first frame of 3D image data to be processed, V pre is the vertex V of the reference 3D image data after rigid alignment t ; For the remaining frames of 3D image data to be processed, V pre ∈ R⁻¹ represents the vertex corresponding to the non-rigid alignment result of the previous frame, and H represents the identity matrix. α and γ are regularization terms, and their values ​​can be set relatively high to ensure smoothness of the generated 3D image data. β and ε can be set relatively low. The specific weight coefficients α, β, γ, and ε can be set to 1000, 10, 500, and 10, respectively.

[0047] The above objective loss function L' consists of five parts:

[0048] (1) Nearest neighbor constraint function: ||(AV pre +b)-V nearest ||2 (3)

[0049] Among them, in the Mth i When the frame to be processed 3D image data is non-rigidly aligned with the reference 3D image data, V nearnest For the Mth i Frame distance to be processed 3D image data (AV pre+b) nearest vertex, a KDTree (k-dimensional, KD tree) can be used to accelerate the acquisition of nearest neighbor points; ||·|| represents the second norm, that is, the square root of the sum of all elements. A is the deformation parameter, and b is the translation parameter. When performing non-rigid alignment on the rigidly aligned reference 3D image data based on the first frame of the 3D image data to be processed, the nearest neighbor constraint function can be used to constrain the distance between the vertices in the first frame of the 3D image data to be processed and the nearest vertices in the non-rigidly aligned reference 3D image data. Also, when performing non-rigid alignment on the reference 3D image data aligned with the previous frame of the 3D image data to be processed based on the current frame of the 3D image data using the non-rigid alignment results of the previous frame, the nearest neighbor constraint function can be used to constrain the distance between the vertices in the current frame of the 3D image data to be processed and the nearest vertices in the non-rigidly aligned reference 3D image data. In this way, through non-rigid deformation, the vertices in the non-rigidly aligned reference 3D image data tend to overlap with the vertices in the 3D image data to be processed of the corresponding frame, making the non-rigidly aligned reference 3D image data more closely aligned with the 3D image data to be processed of the target object.

[0050] (2) Laplace loss function: ||L(AV pre +b)-LV t ||2 (4) Where L is the Laplace matrix, which is expressed as follows:

[0051] Among them, d i is the degree of the i-th vertex (i.e., the number of vertices connected to it). By constraining the local Laplace coordinates to be invariant, the smoothness of the baseline 3D data after non-rigid deformation is guaranteed. This Laplace loss function can also be called a smoothness constraint function.

[0052] (3) Function of key point constraint:

[0053] The facial key points of the deformed reference 3D image data are projected onto the image of the current frame of the target object, and the distance between the facial key points of the deformed reference 3D image data and the key points in the image of the current frame of the corresponding target object is as close as possible. It is the facial key point determined based on the reference three-dimensional image data that has been aligned with the previous frame of three-dimensional image data to be processed, and P is the facial key point result of the image corresponding to the current frame of three-dimensional image data to be processed; for example, it can be the facial key point P extracted from the current frame image, or it can also be the facial key point P of the current frame determined by other means.

[0054] (4) Function of non-deformation region constraint: ||Mnf (A1-H)||2+||M nf b1||2 (6)

[0055] Among them, M nf is the segmentation mask of the non-deformed region, A1 is the deformation parameter, and b1 is the translation parameter. By using the constraint function of the non-deformed region, when non-rigidly aligning the rigidly aligned reference 3D image data based on the first frame of the 3D image data to be processed, and when non-rigidly aligning the reference 3D image data aligned with the previous frame of the 3D image data to be processed based on the non-rigid alignment result of the previous frame, the non-deformed region (e.g., the back of the head and teeth) in the non-rigidly aligned reference 3D image data can be minimized from deformation. Therefore, for this non-deformed region, when the deformation parameter is a matrix, the deformation matrix corresponding to the non-deformed region is the unit matrix, and the deformation offset is minimized to zero.

[0056] (5) Function of stiffness constraint:

[0057] in, and are the deformation parameters corresponding to the i-th vertex and the j-th vertex with vertex connection relationship, and are the translation parameters corresponding to the i-th vertex and the j-th vertex with vertex connection relationship respectively.

[0058] By using a stiffness-constrained function, when non-rigid alignment is performed on the rigidly aligned reference three-dimensional image data based on the first frame of three-dimensional image data to be processed, and when non-rigid alignment is performed on the reference three-dimensional image data that has been aligned with the previous frame of three-dimensional image data to be processed based on the current frame of three-dimensional image data using the non-rigid alignment result of the previous frame, the changes between vertices with a connection relationship in the non-rigidly aligned reference three-dimensional image data can be made as close as possible, that is, the difference in the deformation term and the difference in the translation term between the two vertices are as small as possible.

[0059] In the above manner, based on the N-frame sequence image I of the target object i and the corresponding N frames of 3D image data M to be processed i , and the projection matrix P of the K cameras j , by comparing the reference 3D image data M t Perform deformation and finally output N three-dimensional image data with the same topological structure Among them, M i ={V i ,E i},i∈{1,...,N},V i ,E i are the vertices and edges of the 3D image data to be processed of the target object in the i-th frame, j∈{1,...,K}, M t ={V t ,E t}, N and K are both positive integers.

[0060] Based on the above embodiment, the present disclosure further provides a method for generating three-dimensional image data, as shown in FIG1 . The method may include the following steps:

[0061] In step S110, N frames of to-be-processed 3D image data of the target object are acquired. The N frames of to-be-processed 3D image data include a sequence of to-be-processed 3D image data of a target part of the target object acquired within a target time period. N is a positive integer greater than 1.

[0062] In the embodiments provided by the present disclosure, a target part of a target object can be scanned by a 3D scanning device within a target time period, and N frames of scanned 3D image data to be processed can be obtained. In the embodiments, the 3D image data to be processed can be 3D mesh data. For example, within a target time period T, a target part of a target object can be 3D scanned at different times in chronological order, and 3D image data to be processed can be obtained at different times, such as time t1, time t2, time t3, time t4, ..., t x The 3D image data M to be processed at time x , and each moment corresponds to at least one frame of image I. Wherein, x is a positive integer.

[0063] In the embodiment, from the above time t1 to t x At any moment in time, K cameras set at different positions can be used to scan and obtain the corresponding three-dimensional image data M to be processed at that moment. i . Where i is a positive integer.

[0064] It should be noted that the N frames of to-be-processed three-dimensional image data obtained in the embodiment can be from the time t1 to the time t x Part of the time period, for example, is from time t1 to time t4, or time t2, time t4, time t5, etc., and the embodiments are not limited thereto. The N frames of 3D image data to be processed satisfy a temporal sequence relationship, and the first frame of 3D image data to be processed can be understood as the frame of 3D image data to be processed that is located at the earliest time sequence among the N frames of 3D image data to be processed, and the second frame is the frame of 3D image data to be processed that is located after the first frame and before the remaining frames of 3D image data to be processed.

[0065] In the embodiments provided herein, within the aforementioned time period T, if the target object is a person, the target part can be a face or a hand, though this disclosure does not limit this. Taking the face as an example, since the facial expression can change within the time period T, the resulting target 3D image data sequence can represent the changing expression of the target object and have a consistent topology.

[0066] In the embodiment provided by the present disclosure, when the target part is a human face, the face of the target object can be scanned by a three-dimensional scanning device to obtain N frames of scanned three-dimensional image data to be processed. Since the scanned N frames of three-dimensional image data to be processed may include non-target areas such as human clothes, collar or hair, the scanned N frames of three-dimensional image data to be processed can be pre-processed to obtain N frames of three-dimensional image data to be processed of the target object.

[0067] Specifically, N frames of images of the target object can be obtained using the above method. These N frames of images correspond to N frames of to-be-processed 3D image data of the target object. The obtained 3D scan data of the target object can be projected into the coordinate system corresponding to the N frames of images to obtain a non-target region within the N frames of to-be-processed 3D image data. By removing vertices located in the non-target region, the N frames of to-be-processed 3D image data of the target object are obtained.

[0068] In step S120 , rigid alignment is performed on the reference three-dimensional image data based on the first frame of three-dimensional image data to be processed among the N frames of three-dimensional image data to be processed.

[0069] In this embodiment, key points can be obtained from the first frame of 3D image data to be processed, as well as key points from the reference 3D image data. The reference 3D image data can then be transformed so that each key point in the transformed reference 3D image data approximates the corresponding key point in the first frame of 3D image data to be processed. The reference 3D image data in this embodiment can be a self-created 3D model or an existing standard 3D model, but the embodiment is not limited thereto.

[0070] In step S130 , based on the N frames of to-be-processed 3D image data, non-rigid alignment is performed on the rigidly aligned reference 3D image data to determine a target 3D image data sequence of the target object.

[0071] The non-rigid alignment of the rigidly aligned standard three-dimensional data based on the N frames of three-dimensional image data to be processed includes performing non-rigid alignment on the reference three-dimensional image data aligned with the i-1 frame of three-dimensional image data to be processed based on the i-th frame of three-dimensional image data to be processed in the N frames of three-dimensional image data to be processed, to obtain the reference three-dimensional image data aligned with the i-th frame of three-dimensional image data to be processed; wherein i is a positive integer greater than 1. For example, based on the first frame of three-dimensional image data M1, the reference three-dimensional image data M t After performing rigid and non-rigid alignment, reference 3D image data M1' is obtained, which is aligned with M1. Based on the second frame of processed 3D image data M2, non-rigid alignment is performed on M1' to obtain aligned reference 3D image data M2', and so on, until a target 3D image data sequence of the target object is obtained. In this embodiment, the target 3D image data sequence of the target object is determined based on the aligned reference 3D image data M1', M2', etc., and their topological structures are consistent.

[0072] During the rigid alignment of the first frame of to-be-processed 3D image data with the rigidly aligned reference 3D image data, the reference 3D image data will be significantly deformed. Using the alignment results of the previous frame of to-be-processed 3D image data, and then using the current frame of to-be-processed 3D image data to perform non-rigid alignment on the reference 3D image data that has been aligned with the previous frame of to-be-processed 3D image data, non-rigid alignment can be achieved with smaller deformations. This allows the alignment results of the previous frame to be used to further deform the previously aligned reference 3D image data to fit the current frame of to-be-processed 3D image data, thereby ensuring the topological consistency of the resulting target 3D image data sequence.

[0073] In the embodiments provided herein, in step S130, non-rigid alignment can be performed on the rigidly aligned reference 3D image data based on the first frame of 3D image data to be processed. This allows the deformed reference 3D image data to fit the first frame of 3D image data to be processed. Furthermore, the non-rigid alignment of the rigidly aligned reference 3D image data using the second frame of 3D image data to be processed is performed based on the alignment of the reference 3D image data using the first frame of 3D image data to be processed. If the alignment accuracy of the first frame of 3D image data to the reference 3D image data is high, this will improve the alignment accuracy and efficiency of subsequent frames.

[0074] Based on the above embodiment, in another embodiment provided by the present disclosure, as shown in FIG2 , the above step S120 may further include the following steps:

[0075] In step S121 , a first frame image of the target object is acquired.

[0076] The first frame image corresponds to the first frame of three-dimensional image data to be processed.

[0077] For example, the N-frame sequence image I of the target object can be obtained by the above method. i and the corresponding N frames of 3D image data M to be processed i .

[0078] In step S122 , key points in the first frame image are extracted.

[0079] In an embodiment, key points in an image, such as key points of a human face, may be extracted using algorithms such as SIFT (Scale-invariant feature transform).

[0080] In step S123, three-dimensional key points corresponding to the key points in the first frame of three-dimensional image data to be processed are obtained.

[0081] In the embodiment, the facial key points of the first frame image I1 can be obtained by converting the first frame of the three-dimensional image data to be processed into The vertices of the projected points are projected into the above-mentioned image coordinate system, and the projection points closest to the key points of each face in the first frame image I1 are determined from the projection points obtained after the projection, and the first frame of the three-dimensional image data to be processed corresponding to these closest projection points are respectively determined. The vertex is P1 3d .

[0082] In step S124 , the annotated 3D key points of the reference 3D image data are obtained.

[0083] The annotated key points of the reference three-dimensional image data correspond to the key points on the target object's three-dimensional image data to be processed, such as the feature points of the specific facial features such as the eyes, nose, mouth, chin, etc. involved in the above embodiments, or the specific detail positions in the human face such as the eye sockets and nostrils, etc.

[0084] In step S125 , deformation processing is performed on the reference 3D image data based on the 3D key points and the annotated 3D key points.

[0085] In an embodiment, the reference three-dimensional image data is deformed so that the positions of the annotated key points of the reference three-dimensional image data are consistent with the positions of the three-dimensional key points corresponding to the key points in the first frame of the three-dimensional image data to be processed, thereby achieving the purpose of rigid alignment of the reference three-dimensional image data.

[0086] Based on the above embodiment, in another embodiment provided by the present disclosure, step S130 may specifically include performing non-rigid alignment on the rigidly aligned reference 3D image data based on the N frames of 3D image data to be processed and target constraints. The target constraints include at least one of a nearest neighbor constraint, a smoothness constraint, a key point constraint, a non-deformation region constraint, and a stiffness constraint.

[0087] In an embodiment, when the target constraints include a nearest neighbor constraint, a smoothness constraint, a keypoint constraint, a non-deformation region constraint, and a stiffness constraint, the weighted sum of the nearest neighbor constraint function, the smoothness constraint function, the stiffness constraint function, the keypoint constraint function, the non-deformation region constraint function, and the stiffness constraint function is minimized. The nearest neighbor constraint function is used to constrain the distance between vertices in the N frames of to-be-processed 3D image data and the nearest vertices in the non-rigidly aligned reference 3D image data; the smoothness constraint function is used to constrain the Laplace coordinates of vertices in the non-rigidly aligned reference 3D image data; the stiffness constraint function is used to constrain the degree of variation between connected vertices in the non-rigidly aligned reference 3D image data; the keypoint constraint function is used to constrain the distance between keypoints in the N frames of to-be-processed 3D image data and keypoints in the non-rigidly aligned reference 3D image data; the non-deformation region constraint function is used to constrain the degree of variation in the non-deformation region in the non-rigidly aligned reference 3D image data; and the stiffness constraint function is used to constrain the degree of variation between connected vertices in the non-rigidly aligned reference 3D image data. In the embodiment, the target constraint condition may specifically be the target loss function constructed in the above embodiment. For details, please refer to the specific description in the above embodiment, which will not be repeated here.

[0088] Based on the above embodiment, in another embodiment provided by the present disclosure, the method may further include the following steps:

[0089] Step S211: Acquire N frames of initial to-be-processed 3D image data and N frames of images of the target object, wherein the N frames of initial to-be-processed 3D image data correspond to the N frames of images respectively.

[0090] In an embodiment, the N frames of initial 3D image data to be processed may be initial 3D image data of a target object obtained at different times within a target time period. For example, within the target time period, a target portion of the target object may be 3D scanned at different times in chronological order to obtain 3D image data to be processed at each time. The N frames of image data are images of the target object obtained at the aforementioned times, and the N frames of initial 3D image data to be processed have a corresponding relationship with the N frames of image data. For example, if the target object is a person, the initial 3D image data to be processed may include the person's face. Since the initial 3D image data to be processed may also include the person's hair, collar, etc., the initial 3D image data to be processed may require preprocessing.

[0091] Step S212 , obtaining segmentation masks corresponding to the target parts of N frames of images respectively.

[0092] In an embodiment, for each frame of image, a face region may be segmented from the image using a face segmentation model to obtain a face segmentation mask.

[0093] Step S213 : projecting the N frames of initial 3D image data to be processed into the coordinate system corresponding to the N frames of image respectively, and determining the vertices in the N frames of initial 3D image data to be processed that do not belong to the target part based on the segmentation mask.

[0094] For example, for each frame of the three-dimensional image data M to be processed, i , project it to the image coordinate system using the projection matrix, and use each frame image I i The corresponding face segmentation mask S i , determine each frame image I i The identity matrix H in .

[0095] Step S214 : removing vertices that do not belong to the target part from the N frames of initial 3D image data to be processed, and obtaining N frames of 3D image data to be processed of the target object.

[0096] In an embodiment, N frames of to-be-processed 3D image data of the target object can be obtained by removing vertices that do not belong to the target part from each frame of the initial to-be-processed 3D image data, for example, removing vertices that do not belong to the face area.

[0097] The method for generating three-dimensional image data provided by the embodiments of the present disclosure can be used to obtain facial training data, such as for use in the training of a facial 3D key point detection model and a virtual image expression driving model. Since the corresponding training data obtained by this method has a consistent topology, the accuracy of the model obtained by training the training data obtained by this method can also be further improved.

[0098] In the case of dividing each functional module into corresponding functional modules, an embodiment of the present disclosure provides a device for generating three-dimensional image data. The device for generating three-dimensional image data can be a server or a chip applied to a server. Figure 3 is a schematic block diagram of the functional modules of the device for generating three-dimensional image data provided by an exemplary embodiment of the present disclosure. As shown in Figure 3, the device for generating three-dimensional image data includes:

[0099] a data acquisition module configured to acquire N frames of to-be-processed three-dimensional image data of a target object; the N frames of to-be-processed three-dimensional image data comprising a sequence of to-be-processed three-dimensional image data of a target portion of the target object acquired within a target time period; wherein N is a positive integer greater than 1;

[0100] A rigid alignment module, configured to perform rigid alignment on the reference three-dimensional image data based on a first frame of the N frames of three-dimensional image data to be processed;

[0101] A non-rigid alignment module is configured to perform non-rigid alignment on rigidly aligned reference three-dimensional image data based on the N frames of to-be-processed three-dimensional image data to determine a target three-dimensional image data sequence of the target object; wherein, performing non-rigid alignment on rigidly aligned reference three-dimensional image data based on the N frames of to-be-processed three-dimensional image data includes, based on the i-th frame of to-be-processed three-dimensional image data among the N frames of to-be-processed three-dimensional image data, performing non-rigid alignment on reference three-dimensional image data that has been aligned with the i-1-th frame of to-be-processed three-dimensional image data to obtain reference three-dimensional image data aligned with the i-th frame of to-be-processed three-dimensional image data; wherein i is a positive integer greater than 1.

[0102] Optionally, the non-rigid alignment module is further configured to:

[0103] Based on the N frames of to-be-processed three-dimensional image data and the target constraint conditions, non-rigid alignment is performed on the rigidly aligned reference three-dimensional image data; wherein the target constraint conditions include at least one of the following: nearest neighbor constraint, smoothness constraint, stiffness constraint and key point constraint.

[0104] Optionally, the non-rigid alignment module is further configured to:

[0105] performing non-rigid alignment on the rigidly aligned reference 3D image data according to the N frames of to-be-processed 3D image data, based on a condition where a weighted sum of the function of the nearest neighbor constraint, the function of the smoothness constraint, the function of the stiffness constraint, and the function of the key point constraint is minimized;

[0106] Among them, the function of the nearest neighbor constraint is used to constrain the distance between the vertices in the N frames of to-be-processed three-dimensional image data and the nearest vertices in the non-rigidly aligned reference three-dimensional image data; the function of the smoothness constraint is used to constrain the Laplace coordinates of the vertices in the non-rigidly aligned reference three-dimensional image data; the function of the stiffness constraint is used to constrain the degree of change between vertices with connection relationships in the non-rigidly aligned reference three-dimensional image data; the function of the key point constraint is used to constrain the distance between the key points in the N frames of to-be-processed three-dimensional image data and the key points in the non-rigidly aligned reference three-dimensional image data.

[0107] Optionally, the target constraint condition further includes a non-deformation region constraint; the function of the non-deformation region constraint is used to constrain the degree of change of the non-deformation region in the reference three-dimensional image data after the non-rigid alignment; the non-rigid alignment module is further used to:

[0108] Based on the fact that the weighted sum of the function of the nearest neighbor constraint, the function of the smoothness constraint, the function of the key point constraint, the function of the non-deformation area constraint and the stiffness constraint is minimized, non-rigid alignment is performed on the rigidly aligned reference three-dimensional image data according to the N frames of three-dimensional image data to be processed.

[0109] Optionally, the rigid alignment module is further configured to:

[0110] Acquire a first frame image of the target object; the first frame image corresponds to the first frame of three-dimensional image data to be processed;

[0111] Extracting key points from the first frame of image;

[0112] Obtaining a three-dimensional key point corresponding to the key point in the first frame of three-dimensional image data to be processed;

[0113] Acquire annotated three-dimensional key points of the reference three-dimensional image data;

[0114] The reference three-dimensional image data is deformed based on the three-dimensional key points and the standard three-dimensional key points.

[0115] Optionally, the rigid alignment module is further configured to:

[0116] According to the first frame of to-be-processed three-dimensional image data, non-rigid alignment is performed on the rigidly aligned reference three-dimensional image data.

[0117] In another embodiment provided by the present disclosure, the apparatus further includes:

[0118] a data acquisition module, configured to acquire N frames of initial to-be-processed three-dimensional image data and N frames of images of the target object; wherein the N frames of initial to-be-processed three-dimensional image data correspond to the N frames of images respectively;

[0119] A segmentation mask acquisition module, configured to respectively acquire segmentation masks corresponding to the target parts of the N frames of images;

[0120] a determination module, configured to project the N frames of initial to-be-processed three-dimensional image data into coordinate systems corresponding to the N frames of images, and determine, based on the segmentation mask, vertices in the N frames of initial to-be-processed three-dimensional image data that do not belong to the target part;

[0121] The removal module is configured to remove vertices that do not belong to the target part from the N frames of initial 3D image data to be processed, thereby obtaining N frames of 3D image data to be processed of the target object.

[0122] Since the device part in the embodiment corresponds to the method part above, please refer to the description of the above method embodiment for details, which will not be repeated here.

[0123] The present invention provides a device for generating 3D image data, which obtains N frames of to-be-processed 3D image data of a target object, performs rigid alignment on reference 3D image data based on the first frame of the N frames of to-be-processed 3D image data, and then performs non-rigid alignment on the rigidly aligned reference 3D image data based on the N frames of to-be-processed 3D image data to determine a target 3D image data sequence for the target object. In the alignment process, the present invention first rigidly aligns the reference 3D image data with the first frame of the first frame of to-be-processed 3D image data, and then, based on the rigid alignment, performs non-rigid alignment on the rigidly aligned reference 3D image data based on the N frames of to-be-processed 3D image data to obtain a target 3D image data sequence for the target object. In this way, during the alignment process, the alignment results of the previous frame can be utilized to further deform the previously aligned reference 3D image data to conform to the to-be-processed 3D image data of the current frame, thereby ensuring the topological consistency of the resulting target 3D image data sequence.

[0124] An embodiment of the present disclosure further provides an electronic device, comprising: at least one processor; a memory for storing instructions executable by the at least one processor; wherein the at least one processor is configured to execute the instructions to implement the above method disclosed in the embodiment of the present disclosure.

[0125] Figure 4 is a schematic diagram of the structure of an electronic device provided by an exemplary embodiment of the present disclosure. As shown in Figure 4, the electronic device 1800 includes at least one processor 1801 and a memory 1802 coupled to the processor 1801. The processor 1801 can execute the corresponding steps of the above method disclosed in the embodiment of the present disclosure.

[0126] The processor 1801 can also be referred to as a central processing unit (CPU), which can be an integrated circuit chip with signal processing capabilities. Each step in the method disclosed in the embodiments of the present disclosure can be completed by hardware integrated logic circuits in the processor 1801 or by software instructions. The processor 1801 can be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of the present disclosure can be directly implemented as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in the memory 1802, such as a storage medium mature in the art, such as a random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The processor 1801 reads the information in the memory 1802 and, in conjunction with its hardware, completes the steps of the method.

[0127] In addition, when various operations / processes according to the present disclosure are implemented via software and / or firmware, the programs constituting the software can be installed from a storage medium or a network to a computer system having a dedicated hardware structure, such as computer system 1900 shown in FIG5 . When the various programs are installed, the computer system can perform various functions, including those described above. FIG5 is a block diagram of the structure of a computer system provided by an exemplary embodiment of the present disclosure.

[0128] Computer system 1900 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are intended to be examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0129] As shown in FIG5 , computer system 1900 includes a computing unit 1901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1902 or a computer program loaded from a storage unit 1908 into a random access memory (RAM) 1903. Various programs and data required for the operation of computer system 1900 may also be stored in RAM 1903. Computing unit 1901, ROM 1902, and RAM 1903 are connected to each other via a bus 1904. An input / output (I / O) interface 1905 is also connected to bus 1904.

[0130] Several components within computer system 1900 are connected to I / O interface 1905, including an input unit 1906, an output unit 1907, a storage unit 1908, and a communication unit 1909. Input unit 1906 can be any type of device capable of inputting information into computer system 1900. Input unit 1906 can receive input numeric or character information and generate key input signals related to user settings and / or function control of an electronic device. Output unit 1907 can be any type of device capable of presenting information and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. Storage unit 1908 may include, but is not limited to, a magnetic disk or an optical disk. Communication unit 1909 allows computer system 1900 to exchange information / data with other devices over a network, such as the Internet, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver and / or chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0131] The computing unit 1901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1901 performs the various methods and processes described above. For example, in some embodiments, the above-mentioned methods disclosed in the embodiments of the present disclosure may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1908. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1900 via the ROM 1902 and / or the communication unit 1909. In some embodiments, the computing unit 1901 may be configured to perform the above-mentioned methods disclosed in the embodiments of the present disclosure by any other appropriate means (e.g., by means of firmware).

[0132] An embodiment of the present disclosure further provides a computer-readable storage medium, wherein, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the above method disclosed in the embodiment of the present disclosure.

[0133] The computer-readable storage medium in the embodiments of the present disclosure can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. The above-mentioned computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. More specifically, the above-mentioned computer-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0134] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0135] The embodiments of the present disclosure further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor, the method disclosed in the embodiments of the present disclosure is implemented.

[0136] In embodiments of the present disclosure, computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or combinations thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer.

[0137] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0138] The modules, components, or units described in the embodiments of the present disclosure may be implemented in software or hardware. The names of the modules, components, or units do not necessarily limit the modules, components, or units themselves.

[0139] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, and without limitation, exemplary hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0140] The above descriptions are merely some embodiments of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in the present disclosure.

[0141] Although some specific embodiments of the present disclosure have been described in detail by way of examples, those skilled in the art will appreciate that the above examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Those skilled in the art will appreciate that modifications may be made to the above embodiments without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A method for generating three-dimensional image data, the method comprising: Acquire N frames of to-be-processed three-dimensional image data of the target object; The N frames of to-be-processed three-dimensional image data include a sequence of to-be-processed three-dimensional image data of a target part of the target object obtained within a target time period; wherein N is a positive integer greater than 1; Based on the first frame of the N frames of the three-dimensional image data to be processed, rigidly aligning the reference three-dimensional image data; Based on the N frames of three-dimensional image data to be processed, non-rigid alignment is performed on the rigidly aligned reference three-dimensional image data to determine a target three-dimensional image data sequence of the target object; wherein, based on the N frames of three-dimensional image data to be processed, non-rigid alignment is performed on the rigidly aligned reference three-dimensional image data including, based on the i-th frame of three-dimensional image data to be processed among the N frames of three-dimensional image data to be processed, non-rigid alignment is performed on the reference three-dimensional image data that has been aligned with the i-1-th frame of three-dimensional image data to be processed to obtain reference three-dimensional image data aligned with the i-th frame of three-dimensional image data to be processed; wherein i is a positive integer greater than 1.

2. The method according to claim 1, wherein: The non-rigid alignment of the rigidly aligned reference three-dimensional image data comprises: Based on the N frames of 3D image data to be processed and the target constraint conditions, non-rigid alignment is performed on the rigidly aligned reference 3D image data; wherein the target constraint conditions include at least one of the following: nearest neighbor constraint, smoothness constraint, stiffness constraint and key point constraint.

3. The method according to claim 2, wherein: The non-rigid alignment of the rigidly aligned reference three-dimensional image data based on the N frames of three-dimensional image data to be processed and the target constraint condition comprises: Based on the fact that the weighted sum of the function of the nearest neighbor constraint, the function of the smoothness constraint, the function of the stiffness constraint and the function of the key point constraint is minimized, non-rigidly aligning the rigidly aligned reference 3D image data according to the N frames of 3D image data to be processed; Among them, the function of the nearest neighbor constraint is used to constrain the distance between the vertices in the N frames of three-dimensional image data to be processed and the nearest vertices in the non-rigidly aligned reference three-dimensional image data; the function of the smoothness constraint is used to constrain the Laplace coordinates of the vertices in the non-rigidly aligned reference three-dimensional image data; the function of the stiffness constraint is used to constrain the degree of change between vertices with connection relationships in the non-rigidly aligned reference three-dimensional image data; the function of the key point constraint is used to constrain the distance between the key points in the N frames of three-dimensional image data to be processed and the key points in the non-rigidly aligned reference three-dimensional image data.

4. The method according to claim 2 or 3, wherein: The target constraint condition further includes a non-deformation region constraint, wherein a function of the non-deformation region constraint is used to constrain a degree of change of the non-deformation region in the reference three-dimensional image data after the non-rigid alignment; The non-rigid alignment of the rigidly aligned reference three-dimensional image data based on the N frames of three-dimensional image data to be processed and the target constraint condition comprises: Based on the fact that the weighted sum of the function of the nearest neighbor constraint, the function of the smoothness constraint, the function of the key point constraint, the function of the non-deformation area constraint and the function of the stiffness constraint is minimized, non-rigid alignment is performed on the rigidly aligned reference three-dimensional image data according to the N frames of three-dimensional image data to be processed.

5. The method according to any one of claims 1 to 4, wherein: The step of rigidly aligning the first frame of to-be-processed three-dimensional image data with the reference three-dimensional image data based on the key points of the first frame of to-be-processed three-dimensional image data of the target part comprises: Acquire a first frame image of the target object; the first frame image corresponds to the first frame of three-dimensional image data to be processed; Extracting key points from the first frame of image; Obtaining a three-dimensional key point corresponding to the key point in the first frame of three-dimensional image data to be processed; Acquire annotated three-dimensional key points of the reference three-dimensional image data; The reference three-dimensional image data is deformed based on the three-dimensional key points and the annotated three-dimensional key points.

6. The method according to any one of claims 1 to 5, wherein: The non-rigid alignment of the rigidly aligned reference three-dimensional image data based on the N frames of to-be-processed three-dimensional image data comprises: According to the first frame of to-be-processed three-dimensional image data, non-rigid alignment is performed on the rigidly aligned reference three-dimensional image data.

7. The method according to any one of claims 1 to 6, wherein: The method further comprises: Acquire N frames of initial to-be-processed three-dimensional image data and N frames of images of the target object; wherein the N frames of initial to-be-processed three-dimensional image data correspond to the N frames of images respectively; Respectively obtaining segmentation masks of the N frames of images corresponding to the target parts; Projecting the N frames of initial three-dimensional image data to be processed into coordinate systems corresponding to the N frames of images respectively, and determining vertices in the N frames of initial three-dimensional image data to be processed that do not belong to the target part based on the segmentation mask; The vertices that do not belong to the target part in the N frames of initial three-dimensional image data to be processed are removed to obtain N frames of three-dimensional image data to be processed of the target object.

8. A device for generating three-dimensional image data, the device comprising: A data acquisition module, used to acquire N frames of to-be-processed three-dimensional image data of a target object; The N frames of to-be-processed three-dimensional image data include a sequence of to-be-processed three-dimensional image data of a target part of the target object obtained within a target time period. Column; where N is a positive integer greater than 1; A rigid alignment module, configured to perform rigid alignment on the reference three-dimensional image data based on the first frame of the N frames of three-dimensional image data to be processed; A non-rigid alignment module is used to perform non-rigid alignment on rigidly aligned reference three-dimensional image data based on the N frames of three-dimensional image data to be processed, so as to determine a target three-dimensional image data sequence of the target object; wherein, based on the N frames of three-dimensional image data to be processed, performing non-rigid alignment on rigidly aligned reference three-dimensional image data includes, based on the i-th frame of three-dimensional image data to be processed among the N frames of three-dimensional image data to be processed, performing non-rigid alignment on reference three-dimensional image data that has been aligned with the i-1-th frame of three-dimensional image data to be processed, so as to obtain reference three-dimensional image data aligned with the i-th frame of three-dimensional image data to be processed; wherein i is a positive integer greater than 1.

9. An electronic device, comprising: at least one processor; a memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, wherein: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method as claimed in any one of claims 1 to 7.

11. A computer program comprising: Instructions, when executed by a processor of an electronic device, cause the processor to perform the method according to any one of claims 1-7.

Citation Information

Patent Citations

  • 3D image data registration method and device

    CN105844582A

  • Method and device applied to vascular interventional operation navigation based on 3D / 2D registration

    CN111784727A

  • Three-dimensional reconstruction method, electronic equipment and storage medium

    CN112991524A

  • Techniques for removing and synthesizing secondary dynamics in facial performance capture

    US11151767B1