Three-dimensional image data generation method and device, electronic equipment and storage medium
By performing rigid and non-rigid alignment of the three-dimensional scan data, the topological structure inconsistency caused by the change of the target object state in three-dimensional scan is solved, and the topological consistency and efficient alignment of the three-dimensional image data sequence are achieved.
Patent Information
- Application Number
- CN202310627858.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-30
- Publication Date
- 2024-12-03
AI Technical Summary
In the existing three-dimensional scanning technology, the three-dimensional grid sequence topology caused by the change in the state of the target object during the scanning time period affects subsequent applications.
By obtaining the target object's multi-frame pending three-dimensional image data, rigid alignment is performed first, and then non-rigid alignment is performed. The reference three-dimensional image data is slightly deformed using the previous frame alignment result to make it fit the current frame data, thereby ensuring three-dimensional Topological consistency of image data sequences.
The topological consistency of the three-dimensional image data sequence is achieved, the alignment efficiency and accuracy are improved, and it is suitable for characterizing the continuous changes of the target object during the time period.
Smart Images

Figure CN119068024A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer vision technology, and in particular to a method, device, electronic device and storage medium for generating three-dimensional image data. Background Art
[0002] Three-dimensional (3D) reconstruction refers to the reconstruction of a 3D model from one or more two-dimensional (2D) images. In related industries, the use of 3D scanning technology to obtain models with realistic target objects has become mainstream, and as the price of 3D scanning hardware decreases, this technology will become more and more popular.
[0003] In the related art, when a target object is scanned by a 3D scanning device, a 3D grid sequence of the target object can be obtained after a period of scanning. Since the 3D grid sequence is obtained within a period of time, and the state of the target object may change during the period of time, the topological structure of the 3D grid corresponding to different moments in the period of time may be inconsistent, which is not conducive to the subsequent practical application of the 3D grid sequence. Summary of the invention
[0004] The present disclosure provides a method, device, electronic device and storage medium for generating three-dimensional image data.
[0005] According to a first aspect of the present disclosure, a method for generating three-dimensional image data is provided, the method comprising:
[0006] Acquire N frames of to-be-processed three-dimensional image data of a target object; the N frames of to-be-processed three-dimensional image data include a sequence of to-be-processed three-dimensional image data of a target part of the target object obtained within a target time period; wherein N is a positive integer greater than 1;
[0007] Based on the first frame of the N frames of the three-dimensional image data to be processed, rigidly aligning the reference three-dimensional image data;
[0008] Based on the N frames of three-dimensional image data to be processed, non-rigid alignment is performed on the rigidly aligned reference three-dimensional image data to determine a target three-dimensional image data sequence of the target object; wherein, based on the N frames of three-dimensional image data to be processed, non-rigid alignment is performed on the rigidly aligned reference three-dimensional image data including, based on the i-th frame of three-dimensional image data to be processed among the N frames of three-dimensional image data to be processed, non-rigid alignment is performed on the reference three-dimensional image data that has been aligned with the i-1-th frame of three-dimensional image data to be processed to obtain reference three-dimensional image data aligned with the i-th frame of three-dimensional image data to be processed; wherein i is a positive integer greater than 1.
[0009] According to a second aspect of the present disclosure, a device for generating three-dimensional image data is provided, wherein the device comprises:
[0010] A data acquisition module, configured to acquire N frames of to-be-processed three-dimensional image data of a target object; the N frames of to-be-processed three-dimensional image data include a sequence of to-be-processed three-dimensional image data of a target part of the target object obtained within a target time period; wherein N is a positive integer greater than 1;
[0011] A rigid alignment module, configured to perform rigid alignment on the reference three-dimensional image data based on the first frame of the N frames of three-dimensional image data to be processed;
[0012] A non-rigid alignment module is used to perform non-rigid alignment on rigidly aligned reference three-dimensional image data based on the N frames of three-dimensional image data to be processed, so as to determine a target three-dimensional image data sequence of the target object; wherein, based on the N frames of three-dimensional image data to be processed, performing non-rigid alignment on rigidly aligned reference three-dimensional image data includes, based on the i-th frame of three-dimensional image data to be processed among the N frames of three-dimensional image data to be processed, performing non-rigid alignment on reference three-dimensional image data that has been aligned with the i-1-th frame of three-dimensional image data to be processed, so as to obtain reference three-dimensional image data aligned with the i-th frame of three-dimensional image data to be processed; wherein i is a positive integer greater than 1.
[0013] According to a third aspect of the present disclosure, an electronic device is provided, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, the method described above is implemented.
[0014] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above method of the present disclosure is implemented.
[0015] The method, device, electronic device and storage medium for generating three-dimensional image data provided by the embodiments of the present disclosure obtain N frames of three-dimensional image data to be processed of the target object, perform rigid alignment on the reference three-dimensional image data based on the first frame of the three-dimensional image data to be processed of the N frames of the three-dimensional image data to be processed, and perform non-rigid alignment on the reference three-dimensional image data after rigid alignment based on the N frames of the three-dimensional image data to be processed, so as to determine the target three-dimensional image data sequence of the target object. In the alignment process, the embodiment of the present disclosure first performs rigid alignment on the reference three-dimensional image data with the first frame of the first frame of the three-dimensional image data to be processed, and on the basis of the rigid alignment, performs non-rigid alignment on the reference three-dimensional image data after rigid alignment based on the N frames of the three-dimensional image data to be processed, so as to obtain the target three-dimensional image data sequence of the target object, so that the alignment result of the previous frame can be used in the alignment process, and on this basis, the reference three-dimensional image data aligned in the previous frame is further deformed to make it fit the three-dimensional image data to be processed of the current frame, thereby ensuring the topological consistency of the target three-dimensional image data sequence finally obtained. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:
[0017] Figure 1 A flowchart of a method for generating three-dimensional image data provided by an exemplary embodiment of the present disclosure;
[0018] Figure 2 for Figure 1 Flow chart of step S120;
[0019] Figure 3 A schematic block diagram of functional modules of a device for generating three-dimensional image data provided by an exemplary embodiment of the present disclosure;
[0020] Figure 4 A structural block diagram of an electronic device provided for an exemplary embodiment of the present disclosure;
[0021] Figure 5 A structural block diagram of a computer system provided for an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION
[0022] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.
[0023] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.
[0024] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0025] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0026] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.
[0027] In the embodiments provided by the present disclosure, a target part of a target object can be scanned by a three-dimensional scanning device. The target object can be a person. For example, the target part can be a person's face or hand, etc. In the embodiments, a human face is used as an example for explanation, and the embodiments are not limited thereto. By obtaining multiple frames of three-dimensional image data to be processed of a human face, and obtaining corresponding multiple frames of images, these multiple frames of images can be obtained by shooting with a camera. In the embodiments, the multiple frames of the three-dimensional image data to be processed and the multiple frames of images can be greater than or equal to 2. The multiple frames in the embodiments refer to the three-dimensional image data or images to be processed with a time sequence relationship obtained within a period of time. The format of the image can be RGB (RGB color mode), and the target object can be a specific person. The three-dimensional image data to be processed in the embodiments can be a set formed by the vertices and polygons of a polyhedron to describe the three-dimensional data of the target part of the target object, and the polygon is at least one of a triangle, a quadrilateral or other simple convex polygons; in addition, the three-dimensional image data to be processed can also include three-dimensional data composed of ordinary polygons with holes, and the embodiments are not limited thereto.
[0028] The multiple frames of 3D image data to be processed obtained by scanning with a 3D scanning device may scan out non-target parts of the human body such as clothes, collars and hair during the scanning process. Therefore, these multiple frames of 3D image data to be processed need to be preprocessed to remove these parts.
[0029] In the embodiment, the key points of the face and the face segmentation mask are extracted from the above image, and the vertices of the key points of the face are marked on the reference three-dimensional image data. Among them, the key points are local feature points in the face, which can be the feature points of the specific facial features such as eyes, nose, mouth, chin, etc. or the specific details of the face such as eye sockets, nose wings, etc. The number of key points of the face is not limited in the embodiment of the present disclosure, and can be 49, 68 or 81, etc., and the number of key points of the face is 68 as an example for explanation in the embodiment. In addition, the reference three-dimensional image data can be a standard face model in the 3DMM library. It should be noted that the embodiment of the present disclosure is schematically explained by taking the target object as a person and the target part as the face as an example, and the target object or the target part can also be other, for example, the target object can be a certain creature or object, and the target part can also be other parts, etc., which is not limited to this in the embodiment. In the case where the target object or the target part is other, the segmentation mask of the key points and the target part area can also be extracted according to its local features, so as to realize the alignment of the corresponding reference three-dimensional image data based on the multiple frames of to-be-processed three-dimensional image data corresponding to the target part.
[0030] In an embodiment, for each frame of image, a face region can be segmented from the image by a face segmentation model to obtain a face segmentation mask. A certain number of face key points in the image are extracted, and a corresponding number of face key points are marked on the reference 3D image data, so that some frames of multiple frames of 3D image data to be processed can be aligned with the reference 3D image data using the face key points, for example, the reference 3D image data can be aligned based on the first frame of multiple frames of 3D image data to be processed.
[0031] For example, taking the number of facial key points as 68, in the benchmark 3D image data M t Mark the key vertex P of the face * ∈R 68×3 , for the i-th frame image I i ∈R H×W×3 , by extracting facial key points P i ∈R 68×3 And obtain the face segmentation mask S i ∈R H×W , and process the multiple frames of the scanned target object's three-dimensional image data to be processed to remove the parts that do not belong to the face. For example, for each frame of the three-dimensional image data M to be processed i , the three-dimensional image data M to be processedi The vertices are projected to the three-dimensional image data M to be processed through the projection matrix i The corresponding image I i The image coordinate system where it is located, and by using each frame image I i The corresponding face segmentation mask S i , determine each frame image I i The non-face area in the 3D image data M i The vertices projected in the non-face area are removed, and the processed three-dimensional image data can be recorded as Where H and W represent the height and width of the image respectively, and i is a positive integer.
[0032] After preprocessing the multiple frames of initial three-dimensional image data to be processed of the scanned target object in the above manner, multiple frames of three-dimensional image data to be processed of the target object are obtained, and based on the facial key points of each frame of the image extracted as above, the reference three-dimensional image data is rigidly aligned based on the first frame of three-dimensional image data to be processed among these multiple frames of three-dimensional image data to be processed.
[0033] Specifically, rigid alignment can be performed using existing methods, such as the ICP (Iterative Closest Point) method, to rigidly align the baseline three-dimensional image data with facial key points marked with the first frame of the target object's three-dimensional image data to be processed, and obtain standard three-dimensional image data after rigid alignment, such as a human head model, etc. The embodiments are not limited to this.
[0034] For example, by using the above-mentioned corresponding facial key points, by obtaining the scale s, rotation R and translation T, the reference three-dimensional image data M t The three-dimensional image data to be processed can be obtained after the above processing Roughly aligned, it can be optimized using the following formula.
[0035] min s,R,T ||s*(RP * )+T-P1 3d || (1)
[0036] Where s is a scalar, R is a 3×3 matrix, and T is a 1×3 vector; P1 3d The facial key points of the first frame image I1 are obtained by converting the first frame of the three-dimensional image data M1 to be processed into process The vertices of the data are projected to the above-mentioned image coordinate system, and the projection points obtained after the projection are respectively determined to be the closest to each face key point in the first frame image I1, and the three-dimensional image data M1 to be processed corresponding to these closest projection points is processThe vertex is P1 3d .
[0037] Through the above rigid alignment, the deformation amplitude of the reference three-dimensional image data is relatively large, so that the reference three-dimensional image data can be basically aligned with the first frame of to-be-processed three-dimensional image data of the target object.
[0038] In an embodiment, the reference three-dimensional image data may be rigidly aligned according to the first frame of three-dimensional image data to be processed among multiple frames of three-dimensional image data to be processed of the target object. The alignment result of the rigid alignment between the reference three-dimensional image data and the first frame of three-dimensional image data to be processed is used to perform non-rigid alignment on the rigidly aligned reference three-dimensional image data according to the first frame of three-dimensional image data to be processed. In the process of rigidly aligning the first frame of three-dimensional image data to be processed with the reference three-dimensional image data, the reference three-dimensional image data will be deformed to a large extent. When the rigidly aligned reference three-dimensional image data is non-rigidly aligned using the first frame of three-dimensional image data to be processed, the deformation amplitude of the rigidly aligned reference three-dimensional image data will be reduced.
[0039] After rigidly aligning the reference three-dimensional image data based on the facial key points of the target object, the rigidly aligned reference three-dimensional image data can be used to perform non-rigid alignment with multiple frames of three-dimensional image data to be processed.
[0040] Specifically, first, for the first frame of three-dimensional image data to be processed among the multiple frames of three-dimensional image data to be processed of the target object, non-rigid alignment is performed on the rigidly aligned reference three-dimensional image data according to the first frame of three-dimensional image data to be processed. And using the alignment result of the rigidly aligned reference three-dimensional image data and the first frame of three-dimensional image data to be processed, non-rigid alignment is performed on the aligned reference three-dimensional image data according to the second frame of three-dimensional image data to be processed. In the process of rigidly aligning the reference three-dimensional image data with the first frame of three-dimensional image data to be processed, the reference three-dimensional image data will be deformed to a large extent. By using the alignment result of the previous frame of three-dimensional image data to be processed, and then performing non-rigid alignment on the reference three-dimensional image data that has been aligned with the previous frame of three-dimensional image data according to the current frame of three-dimensional image data, alignment can be achieved through a smaller deformation. In this way, the reference three-dimensional image data that has been aligned with the previous frame of three-dimensional image data to be processed is non-rigidly aligned through the current frame of three-dimensional image data to be processed, until the non-rigid alignment based on the last frame of three-dimensional image data to be processed is completed. It is then possible to utilize the alignment result of the previous frame, and on this basis further deform the reference three-dimensional image data that has been aligned with the current frame of three-dimensional image data to be processed, thereby ensuring the topological consistency of the target three-dimensional image data sequence finally obtained.
[0041] It should be noted that the multiple frames of to-be-processed three-dimensional image data in the embodiment are a sequence of to-be-processed three-dimensional image data obtained by scanning the target part of the target object within a time period, and the sequence of to-be-processed three-dimensional image data has time sequence information. For example, when the target part is a human face, the multiple frames of to-be-processed three-dimensional image data can represent the process of expression changes of the face of the target object within a time period, and the previous frame of to-be-processed three-dimensional image data and the current frame of to-be-processed three-dimensional image data have a time sequence relationship. Therefore, on the basis that the multiple frames of to-be-processed three-dimensional image data represent the target part of the same target object and have a time sequence relationship, the multiple frames of to-be-processed three-dimensional image data can represent the continuous changes in the contour morphology of the same target part within a period of time, especially the contour morphology represented by the to-be-processed three-dimensional image data of adjacent frames is relatively close, so the alignment result of the previous frame of to-be-processed three-dimensional image data can be used, and on this basis, the reference three-dimensional image data aligned with the previous frame of to-be-processed three-dimensional image data can be further aligned to obtain the alignment result corresponding to the current frame of to-be-processed three-dimensional image data. In the embodiment, only the first frame of to-be-processed three-dimensional image data needs to be used to rigidly align the reference three-dimensional image data, and there is no need to rigidly align each frame of to-be-processed three-dimensional image data with the reference three-dimensional image data. Starting from the second frame of 3D image data to be processed, since the aligned reference 3D image data is aligned on the basis of the alignment with the previous frame of 3D image data to be processed, the topological structure of the obtained target 3D image data sequence is consistent. In addition, in the process of rigidly aligning the reference 3D image data using the first frame of 3D image data to be processed, the deformation of the reference 3D image data is relatively large, and the basic alignment of the reference 3D image data can be achieved. When the final target 3D image data sequence is obtained through each frame of 3D image data to be processed, only a small deformation of the rigidly aligned reference 3D image data is required, which can improve the overall efficiency of the alignment.
[0042] In the embodiment, multiple frames of to-be-processed three-dimensional image data are obtained by scanning the target object, and rigid alignment is performed on the reference three-dimensional image data according to the first frame of to-be-processed three-dimensional image data, and non-rigid alignment is performed again on the basis of the rigid alignment, and the non-rigid alignment result of the previous frame is used, and on this basis, the reference three-dimensional image data that has been aligned with the previous frame of to-be-processed three-dimensional image data is non-rigidly aligned through the current frame of to-be-processed three-dimensional image data. For the reference three-dimensional image data that has been rigidly aligned with the first frame of to-be-processed three-dimensional image data, the reference three-dimensional image data that has been rigidly aligned is non-rigidly aligned again according to the first frame of to-be-processed three-dimensional image data, so that the reference three-dimensional image data that has been rigidly aligned can be made to fit the first frame of to-be-processed three-dimensional image data more closely after the above rigid alignment and non-rigid alignment. For the target 3D image data corresponding to the 3D image data to be processed of the remaining frames, since it is further deformed based on the non-rigid alignment result of the previous frame, it can also better fit the 3D image data to be processed of the remaining frames.
[0043] Specifically, for N v The vertices of the rigidly aligned reference 3D image data By optimizing the deformation parameters and translation parameters By constructing the target loss function, and using the Adam optimizer to optimize the deformation parameter A and the translation parameter b. The target loss function L' is specifically:
[0044]
[0045] Among them, for the first frame of 3D image data to be processed, V pre is the vertex V of the reference 3D image data after rigid alignment t ; For the remaining frames of 3D image data to be processed, V pre is the vertex corresponding to the result of non-rigid alignment of the previous frame, and H represents the unit matrix. In addition, α and γ are regularization terms, and their values can be set relatively high to ensure the smoothness of the generated 3D image data. The values of β and ε can be set relatively low. The specific weight coefficients α, β, γ and ε can be set to 1000, 10, 500 and 10 respectively.
[0046] The above objective loss function L' consists of five parts:
[0047] (1) Nearest neighbor constraint function:
[0048] ||(AV pre +b)-V nearest ||2 (3)
[0049] Among them, in the Mth i When the frame to-be-processed 3D image data is non-rigidly aligned with the reference 3D image data, V nearnest For the Mth i The distance between the frame and the 3D image data to be processed (AV pre+b) nearest vertex, KDTree (k-dimensional, KD tree) can be used to accelerate the acquisition of the nearest neighbor point; ||·|| represents the second norm, that is, the square root of the sum of each element. A is a deformation parameter, and b is a translation parameter. When non-rigid alignment is performed on the rigidly aligned reference three-dimensional image data according to the first frame of the three-dimensional image data to be processed, the function of the nearest neighbor constraint can be used to constrain the distance between the vertices in the first frame of the three-dimensional image data to be processed of the target object and the nearest vertices in the non-rigidly aligned reference three-dimensional image data, and when non-rigid alignment is performed on the reference three-dimensional image data that has been aligned with the previous frame of the three-dimensional image data to be processed according to the current frame of the three-dimensional image data to be processed using the non-rigid alignment result of the previous frame, the function of the nearest neighbor constraint can be used to constrain the distance between the vertices in the current frame of the three-dimensional image data to be processed and the nearest vertices in the non-rigidly aligned reference three-dimensional image data. In this way, through non-rigid deformation, the vertices in the non-rigidly aligned reference three-dimensional image data tend to coincide with the vertices of the corresponding frame of the three-dimensional image data to be processed, so that the non-rigidly aligned reference three-dimensional image data and the target object The three-dimensional image data to be processed is more closely fitted.
[0050] (2) Laplace loss function:
[0051] ||L(AV pre +b)-LV t ||2 (4)
[0052] Where L is the Laplace matrix, which can be expressed as follows:
[0053]
[0054] Among them, d i is the degree of the i-th vertex (i.e., the number of vertices connected to the point), and the smoothness of the reference 3D data after non-rigid deformation is ensured by constraining the local Laplace coordinates to be invariant. The Laplace loss function can also be called a smoothness constraint function.
[0055] (3) Function of key point constraints:
[0056]
[0057] The facial key points of the deformed reference 3D image data are projected onto the image of the current frame of the target object, and the distance between the facial key points of the deformed reference 3D image data and the key points in the image of the current frame of the corresponding target object is as close as possible. It is the facial key point determined based on the reference three-dimensional image data that has been aligned with the previous frame of three-dimensional image data to be processed, and P is the facial key point result of the image corresponding to the current frame of three-dimensional image data to be processed; for example, it can be the facial key point P extracted from the current frame image, or it can also be the facial key point P of the current frame determined by other means.
[0058] (4) Function of non-deformation area constraint:
[0059] ||M nf (A1-H)||2+||M nf b1||2 (6)
[0060] Among them, M nf is the segmentation mask of the non-deformed area, A1 is the deformation parameter, and b1 is the translation parameter. Through the constraint function of the non-deformed area, when the first frame of the 3D image data to be processed is used to perform non-rigid alignment on the reference 3D image data after rigid alignment, and when the non-rigid alignment result of the previous frame is used to perform non-rigid alignment on the reference 3D image data aligned with the previous frame of the 3D image data to be processed according to the current frame of the 3D image data, the non-deformed area (such as the back of the head and teeth, etc.) in the reference 3D image data after non-rigid alignment can be made as little deformed as possible. Therefore, for the non-deformed area, when the deformation parameter is a matrix, the deformation matrix corresponding to the non-deformed area is a unit matrix, and the deformation offset is as close to 0 as possible.
[0061] (5) Function of stiffness constraint:
[0062]
[0063] in, and are the deformation parameters corresponding to the i-th vertex and the j-th vertex with vertex connection relationship, and They are the translation parameters corresponding to the i-th vertex and the j-th vertex with vertex connection relationship respectively.
[0064] Through the function of the rigidity constraint, when non-rigid alignment is performed on the rigidly aligned reference three-dimensional image data according to the first frame of three-dimensional image data to be processed, and when non-rigid alignment is performed on the reference three-dimensional image data that has been aligned with the previous frame of three-dimensional image data to be processed according to the current frame of three-dimensional image data using the non-rigid alignment result of the previous frame, the changes between the vertices with a connection relationship in the non-rigidly aligned reference three-dimensional image data can be made as close as possible, that is, the difference in the deformation term and the difference in the translation term between the two vertices are as small as possible.
[0065] In the above manner, based on the N-frame sequence image I of the target object i and the corresponding N frames of 3D image data M to be processed i , and the projection matrix P of the K cameras j , by comparing the reference 3D image data M t Deform and finally output N three-dimensional image data with the same topological structure Among them, M i = {V i ,E i},i∈{1,...,N},V i ,E i are the vertices and edges of the 3D image data to be processed of the target object in the i-th frame, j∈{1,...,K}, M t = {V t ,E t}, N and K are both positive integers.
[0066] Based on the above embodiments, the present disclosure also provides a method for generating three-dimensional image data, such as Figure 1 As shown, the method may include the following steps:
[0067] In step S110, N frames of to-be-processed three-dimensional image data of the target object are obtained, wherein the N frames of to-be-processed three-dimensional image data include a sequence of to-be-processed three-dimensional image data of the target part of the target object obtained within a target time period, wherein N is a positive integer greater than 1.
[0068] In the embodiments provided by the present disclosure, a target part of a target object can be scanned by a three-dimensional scanning device within a target time period, and N frames of scanned three-dimensional image data to be processed can be obtained. In the embodiments, the three-dimensional image data to be processed can be three-dimensional grid data. For example, within a target time period T, a target part of a target object is three-dimensionally scanned at different times in a time sequence, and the three-dimensional image data to be processed at different times can be obtained, for example, obtaining time t1, time t2, time t3, time t4, ..., t x The three-dimensional image data M to be processed at time x , and each moment corresponds to at least one frame of image I. Wherein, x is a positive integer.
[0069] In the embodiment, from the above time t1 to t x At any time in the time period, K cameras set at different positions can be used to scan and obtain the corresponding 3D image data M to be processed at that time i . Where i is a positive integer.
[0070] It should be noted that the N frames of to-be-processed three-dimensional image data obtained in the embodiment can be the above-mentioned time t1 to time t x Part of the moments, for example, is from moment t1 to moment t4, or moment t2, moment t4, moment t5, etc., and the embodiment is not limited thereto. The N frames of three-dimensional image data to be processed satisfy a sequential relationship in time sequence, and the first frame of three-dimensional image data to be processed can be understood as a frame of three-dimensional image data to be processed that is located at the front of the time sequence among the N frames of three-dimensional image data to be processed, and the second frame is a frame of three-dimensional image data to be processed that is located after the first frame and before the remaining frames of three-dimensional image data to be processed.
[0071] In the embodiments provided by the present disclosure, within the above time period T, if the target object is a person, the target part can be a face or a hand, and the present disclosure does not limit this. Taking the face as an example, since the expression of the face can change within the time period T, the target three-dimensional image data sequence finally obtained can represent the process of the expression change of the target object and has a consistent topology.
[0072] In the embodiments provided in the present disclosure, when the target part is a human face, the face of the target object can be scanned by a three-dimensional scanning device to obtain N frames of scanned three-dimensional image data to be processed. Since the scanned N frames of three-dimensional image data to be processed may include non-target areas such as human clothes, collar or hair, the scanned N frames of three-dimensional image data to be processed can be pre-processed to obtain N frames of three-dimensional image data to be processed of the target object.
[0073] Specifically, N frames of images of the target object can be obtained by the above method, and the N frames of images correspond to the N frames of three-dimensional image data to be processed of the target object. The obtained three-dimensional scanning data of the target object can be projected into the coordinate system corresponding to the N frames of images respectively to obtain the non-target part domain in the N frames of three-dimensional image data to be processed. By removing the vertices located in the non-target part area, the N frames of three-dimensional image data to be processed of the target object are obtained.
[0074] In step S120 , based on the first frame of 3D image data to be processed among the N frames of 3D image data to be processed, rigid alignment is performed on the reference 3D image data.
[0075] In the embodiment, by obtaining key points in the first frame of 3D image data to be processed and obtaining key points in the reference 3D image data, the reference 3D image data can be transformed so that each key point in the transformed reference 3D image data can be close to the corresponding key point in the first frame of 3D image data to be processed. The reference 3D image data in the embodiment can be a self-created 3D model or an existing standard 3D model, and the embodiment is not limited thereto.
[0076] In step S130, based on the N frames of to-be-processed 3D image data, non-rigid alignment is performed on the rigidly aligned reference 3D image data to determine a target 3D image data sequence of the target object.
[0077] Wherein, based on N frames of to-be-processed 3D image data, performing non-rigid alignment on the rigidly aligned standard 3D data includes, based on the i-th frame of to-be-processed 3D image data in the N frames of to-be-processed 3D image data, performing non-rigid alignment on the reference 3D image data that has been aligned with the i-1-th frame of to-be-processed 3D image data, to obtain reference 3D image data aligned with the i-th frame of to-be-processed 3D image data; wherein i is a positive integer greater than 1. For example, based on the first frame of to-be-processed 3D image data M1, the reference 3D image data M2 is aligned with the i-1-th frame of to-be-processed 3D image data. t After rigid alignment and non-rigid alignment, the reference three-dimensional image data M1' aligned with M1 is obtained. According to the second frame of the three-dimensional image data to be processed M2, non-rigid alignment is performed on M1' to obtain aligned reference three-dimensional image data M2'..., and so on, until the target three-dimensional image data sequence of the target object is obtained. In the embodiment, the target three-dimensional image data sequence of the target object is determined based on the aligned reference three-dimensional image data M1', M2'..., and their topological structures are consistent.
[0078] In the process of rigidly aligning the first frame of 3D image data to be processed with the rigidly aligned reference 3D image data, the reference 3D image data will be deformed to a large extent. By using the alignment result of the last frame of 3D image data to be processed, and then using the current frame of 3D image data to be processed to perform non-rigid alignment on the reference 3D image data that has been aligned with the last frame of 3D image data to be processed, non-rigid alignment can be achieved through a smaller deformation. In this way, the alignment result of the last frame can be used, and on this basis, the reference 3D image data that has been aligned with the last frame of 3D image data to be processed can be further deformed to fit the current frame of 3D image data to be processed, thereby ensuring the topological consistency of the target 3D image data sequence finally obtained.
[0079] In the embodiment provided by the present disclosure, in the above step S130, the rigidly aligned reference 3D image data can be non-rigidly aligned according to the first frame of 3D image data to be processed. In this way, the deformed reference 3D image data can be further aligned with the first frame of 3D image data to be processed; and the non-rigid alignment of the rigidly aligned reference 3D image data using the second frame of 3D image data to be processed is based on the alignment of the reference 3D image data using the first frame of 3D image data to be processed. If the alignment accuracy of the first frame of 3D image data to the reference 3D image data is high, the alignment accuracy and efficiency of subsequent frames will be improved.
[0080] Based on the above embodiments, in another embodiment provided by the present disclosure, Figure 2 As shown, the above step S120 may further include the following steps:
[0081] In step S121, a first frame image of the target object is acquired.
[0082] The first frame image corresponds to the first frame of three-dimensional image data to be processed.
[0083] For example, the N-frame sequence image I of the target object can be obtained by the above method. i and the corresponding N frames of 3D image data M to be processed i .
[0084] In step S122, key points in the first frame image are extracted.
[0085] In an embodiment, key points in an image, such as key points of a human face, may be extracted by using algorithms such as SIFT (Scale-invariant feature transform).
[0086] In step S123, three-dimensional key points corresponding to the key points in the first frame of three-dimensional image data to be processed are obtained.
[0087] In the embodiment, the facial key points of the first frame image I1 can be obtained by converting the first frame of the to-be-processed three-dimensional image data M1 into process The vertices of the image are projected to the above-mentioned image coordinate system, and the projection points obtained after the projection are respectively determined to be the closest projection points to each face key point in the first frame image I1, and the first frame of the three-dimensional image data M1 to be processed corresponding to these closest projection points process The vertex is P1 3d .
[0088] In step S124 , the annotated 3D key points of the reference 3D image data are obtained.
[0089] The annotated key points of the reference three-dimensional image data correspond to the key points on the target object's three-dimensional image data to be processed, such as the feature points of the specific facial features such as eyes, nose, mouth, chin, etc. involved in the above embodiments, or the specific detail positions of the human face such as eye sockets and nose wings, etc.
[0090] In step S125 , deformation processing is performed on the reference 3D image data based on the 3D key points and the annotated 3D key points.
[0091] In an embodiment, the benchmark 3D image data is deformed so that the positions of the annotated key points of the benchmark 3D image data are consistent with the positions of the 3D key points corresponding to the key points in the first frame of 3D image data to be processed, thereby achieving the purpose of rigid alignment of the benchmark 3D image data.
[0092] Based on the above embodiment, in another embodiment provided by the present disclosure, the above step S130 may specifically include: based on the N frames of 3D image data to be processed and the target constraint condition, non-rigidly aligning the rigidly aligned reference 3D image data. The target constraint condition includes: at least one of a nearest neighbor constraint, a smoothness constraint, a key point constraint, a non-deformation region constraint, and a rigidity constraint.
[0093] In an embodiment, when the target constraint conditions include: nearest neighbor constraint, smoothness constraint, key point constraint, non-deformation region constraint and stiffness constraint, the weighted sum of the nearest neighbor constraint function, smoothness constraint function, stiffness constraint function, key point constraint function, non-deformation region constraint function and stiffness constraint function is the smallest. Among them, the nearest neighbor constraint function is used to constrain the distance between the vertices in the N frames of to-be-processed three-dimensional image data and the nearest vertices in the non-rigidly aligned reference three-dimensional image data; the smoothness constraint function is used to constrain the Laplace coordinates of the vertices in the non-rigidly aligned reference three-dimensional image data; the stiffness constraint function is used to constrain the degree of change between the vertices with a connection relationship in the non-rigidly aligned reference three-dimensional image data; the key point constraint function is used to constrain the distance between the key points in the N frames of to-be-processed three-dimensional image data and the key points in the non-rigidly aligned reference three-dimensional image data. The non-deformation region constraint function is used to constrain the degree of change of the non-deformation region in the non-rigidly aligned reference three-dimensional image data; the stiffness constraint function is used to constrain the degree of change between the vertices with a connection relationship in the non-rigidly aligned reference three-dimensional image data. In the embodiment, the target constraint condition may specifically be the target loss function constructed in the above embodiment. For details, please refer to the specific description in the above embodiment, which will not be repeated here.
[0094] Based on the above embodiment, in another embodiment provided by the present disclosure, the method may further include the following steps:
[0095] Step S211, obtaining N frames of initial to-be-processed 3D image data and N frames of images of the target object, wherein the N frames of initial to-be-processed 3D image data correspond to the N frames of images respectively.
[0096] In an embodiment, the N frames of initial three-dimensional image data to be processed may be initial three-dimensional image data to be processed of a target object obtained at different times within a target time period. For example, within a target time period, a target part of a target object is subjected to a three-dimensional scan at different times in a chronological order, and three-dimensional image data to be processed at different times may be obtained. The N frames of images are images of the target object corresponding to the above-mentioned times, and the N frames of initial three-dimensional image data to be processed have a corresponding relationship with the N frames of images. For example, when the target object is a person, the initial three-dimensional image data to be processed may be three-dimensional image data to be processed including a face of the person. Since the initial three-dimensional image data to be processed may also include hair, collar, etc. of the person, the initial three-dimensional image data to be processed needs to be preprocessed.
[0097] Step S212, respectively obtain segmentation masks corresponding to the target parts of N frames of images.
[0098] In an embodiment, for each frame of image, a face region may be segmented from the image using a face segmentation model to obtain a face segmentation mask.
[0099] Step S213 , projecting the N frames of initial 3D image data to be processed into the coordinate system corresponding to the N frames of images respectively, and determining the vertices in the N frames of initial 3D image data to be processed that do not belong to the target part based on the segmentation mask.
[0100] For example, for each frame of the three-dimensional image data M to be processed, i , project it to the image coordinate system using the projection matrix, and use each frame image I i The corresponding face segmentation mask S i , determine each frame image I i The identity matrix H in .
[0101] Step S214, removing vertices that do not belong to the target part from the N frames of initial three-dimensional image data to be processed, and obtaining N frames of three-dimensional image data to be processed of the target object.
[0102] In the embodiment, by removing vertices that do not belong to the target part in each frame of the initial three-dimensional image data to be processed, for example, removing vertices that do not belong to the face area, N frames of three-dimensional image data to be processed of the target object can be obtained.
[0103] The method for generating three-dimensional image data provided by the embodiments of the present disclosure can be used to obtain face training data, for example, for use in the training of a face 3D key point detection model and a virtual image expression driving model. Since the corresponding training data obtained by this method has a consistent topology, the accuracy of the model obtained by training the training data obtained by this method can also be further improved.
[0104] In the case of dividing each functional module according to each function, an embodiment of the present disclosure provides a device for generating three-dimensional image data, which may be a server or a chip applied to a server. Figure 3 The following is a schematic block diagram of the functional modules of a device for generating three-dimensional image data provided by an exemplary embodiment of the present disclosure. Figure 3 As shown, the device for generating the three-dimensional image data includes:
[0105] A data acquisition module, configured to acquire N frames of to-be-processed three-dimensional image data of a target object; the N frames of to-be-processed three-dimensional image data include a sequence of to-be-processed three-dimensional image data of a target part of the target object obtained within a target time period; wherein N is a positive integer greater than 1;
[0106] A rigid alignment module, configured to perform rigid alignment on the reference three-dimensional image data based on the first frame of the N frames of three-dimensional image data to be processed;
[0107] A non-rigid alignment module is used to perform non-rigid alignment on rigidly aligned reference three-dimensional image data based on the N frames of three-dimensional image data to be processed, so as to determine a target three-dimensional image data sequence of the target object; wherein, based on the N frames of three-dimensional image data to be processed, performing non-rigid alignment on rigidly aligned reference three-dimensional image data includes, based on the i-th frame of three-dimensional image data to be processed among the N frames of three-dimensional image data to be processed, performing non-rigid alignment on reference three-dimensional image data that has been aligned with the i-1-th frame of three-dimensional image data to be processed, so as to obtain reference three-dimensional image data aligned with the i-th frame of three-dimensional image data to be processed; wherein i is a positive integer greater than 1.
[0108] Optionally, the non-rigid alignment module is further used for:
[0109] Based on the N frames of 3D image data to be processed and the target constraint conditions, non-rigid alignment is performed on the rigidly aligned reference 3D image data; wherein the target constraint conditions include at least one of the following: nearest neighbor constraint, smoothness constraint, stiffness constraint and key point constraint.
[0110] Optionally, the non-rigid alignment module is further used for:
[0111] Based on the fact that the weighted sum of the function of the nearest neighbor constraint, the function of the smoothness constraint, the function of the stiffness constraint and the function of the key point constraint is minimized, non-rigidly aligning the rigidly aligned reference 3D image data according to the N frames of 3D image data to be processed;
[0112] Among them, the function of the nearest neighbor constraint is used to constrain the distance between the vertices in the N frames of three-dimensional image data to be processed and the nearest vertices in the non-rigidly aligned reference three-dimensional image data; the function of the smoothness constraint is used to constrain the Laplace coordinates of the vertices in the non-rigidly aligned reference three-dimensional image data; the function of the stiffness constraint is used to constrain the degree of change between vertices with connection relationships in the non-rigidly aligned reference three-dimensional image data; the function of the key point constraint is used to constrain the distance between the key points in the N frames of three-dimensional image data to be processed and the key points in the non-rigidly aligned reference three-dimensional image data.
[0113] Optionally, the target constraint condition further includes a non-deformation region constraint; the function of the non-deformation region constraint is used to constrain the degree of change of the non-deformation region in the reference three-dimensional image data after the non-rigid alignment; the non-rigid alignment module is further used to:
[0114] Based on the fact that the weighted sum of the function of the nearest neighbor constraint, the function of the smoothness constraint, the function of the key point constraint, the function of the non-deformation area constraint and the function of the stiffness constraint is minimized, non-rigid alignment is performed on the rigidly aligned reference three-dimensional image data according to the N frames of three-dimensional image data to be processed.
[0115] Optionally, the rigid alignment module is further used for:
[0116] Acquire a first frame image of the target object; the first frame image corresponds to the first frame of three-dimensional image data to be processed;
[0117] Extracting key points from the first frame of image;
[0118] Obtaining a three-dimensional key point corresponding to the key point in the first frame of three-dimensional image data to be processed;
[0119] Acquire annotated three-dimensional key points of the reference three-dimensional image data;
[0120] The reference three-dimensional image data is deformed based on the three-dimensional key points and the standard three-dimensional key points.
[0121] Optionally, the rigid alignment module is further used for:
[0122] According to the first frame of to-be-processed three-dimensional image data, non-rigid alignment is performed on the rigidly aligned reference three-dimensional image data.
[0123] In another embodiment provided by the present disclosure, the device further includes:
[0124] A data acquisition module, used to acquire N frames of initial to-be-processed three-dimensional image data and N frames of images of the target object; wherein the N frames of initial to-be-processed three-dimensional image data correspond to the N frames of images respectively;
[0125] A segmentation mask acquisition module, used to respectively acquire segmentation masks of the N frames of images corresponding to the target parts;
[0126] A determination module, used for projecting the N frames of initial three-dimensional image data to be processed into coordinate systems corresponding to the N frames of images, and determining vertices in the N frames of initial three-dimensional image data to be processed that do not belong to the target part based on the segmentation mask;
[0127] The removal module is used to remove vertices that do not belong to the target part in the N frames of initial three-dimensional image data to be processed, so as to obtain N frames of three-dimensional image data to be processed of the target object.
[0128] Since the device part in the embodiment corresponds to the method part described above, please refer to the description of the method embodiment described above for details, which will not be repeated here.
[0129] The device for generating three-dimensional image data provided by the embodiment of the present disclosure obtains N frames of three-dimensional image data to be processed of the target object, and based on the first frame of three-dimensional image data to be processed of the N frames of three-dimensional image data to be processed, performs rigid alignment on the reference three-dimensional image data, and based on the N frames of three-dimensional image data to be processed, performs non-rigid alignment on the rigidly aligned reference three-dimensional image data to determine the target three-dimensional image data sequence of the target object. In the process of alignment, the embodiment of the present disclosure first performs rigid alignment on the reference three-dimensional image data with the first frame of the first frame of three-dimensional image data to be processed, and based on the rigid alignment, performs non-rigid alignment on the rigidly aligned reference three-dimensional image data based on the N frames of three-dimensional image data to be processed, to obtain the target three-dimensional image data sequence of the target object, so that the alignment result of the previous frame can be used in the alignment process, and on this basis, the aligned reference three-dimensional image data of the previous frame is further deformed to fit the three-dimensional image data to be processed of the current frame, thereby ensuring the topological consistency of the target three-dimensional image data sequence finally obtained.
[0130] An embodiment of the present disclosure also provides an electronic device, comprising: at least one processor; and a memory for storing instructions executable by the at least one processor; wherein the at least one processor is configured to execute the instructions to implement the above method disclosed in the embodiment of the present disclosure.
[0131] Figure 4 The structure diagram of an electronic device provided by an exemplary embodiment of the present disclosure is shown in FIG. Figure 4As shown, the electronic device 1800 includes at least one processor 1801 and a memory 1802 coupled to the processor 1801 , and the processor 1801 can execute corresponding steps in the above method disclosed in the embodiment of the present disclosure.
[0132] The processor 1801 may also be referred to as a central processing unit (CPU), which may be an integrated circuit chip having signal processing capabilities. Each step in the method disclosed in the embodiment of the present disclosure may be completed by an integrated logic circuit of hardware in the processor 1801 or by instructions in the form of software. The processor 1801 may be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present disclosure may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a memory 1802, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, or other mature storage media in the art. The processor 1801 reads the information in the memory 1802 and completes the steps of the method in combination with its hardware.
[0133] In addition, when various operations / processes according to the present disclosure are implemented by software and / or firmware, they can be transmitted from a storage medium or a network to a computer system having a dedicated hardware structure, such as Figure 5 The computer system 1900 shown is installed with the programs constituting the software. When the various programs are installed, the computer system can perform various functions, including the functions described above. Figure 5 A structural block diagram of a computer system provided for an exemplary embodiment of the present disclosure.
[0134] Computer system 1900 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.
[0135] like Figure 5 As shown, the computer system 1900 includes a computing unit 1901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1902 or a computer program loaded from a storage unit 1908 to a random access memory (RAM) 1903. In the RAM 1903, various programs and data required for the operation of the computer system 1900 can also be stored. The computing unit 1901, the ROM 1902, and the RAM 1903 are connected to each other via a bus 1904. An input / output (I / O) interface 1905 is also connected to the bus 1904.
[0136] A plurality of components in the computer system 1900 are connected to the I / O interface 1905, including: an input unit 1906, an output unit 1907, a storage unit 1908, and a communication unit 1909. The input unit 1906 may be any type of device capable of inputting information to the computer system 1900, and the input unit 1906 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 1907 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1908 may include, but is not limited to, a disk, an optical disk. The communication unit 1909 allows the computer system 1900 to exchange information / data with other devices over a network such as the Internet, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.
[0137] The computing unit 1901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1901 performs the various methods and processes described above. For example, in some embodiments, the above methods disclosed in the embodiments of the present disclosure may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1908. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1900 via the ROM 1902 and / or the communication unit 1909. In some embodiments, the computing unit 1901 may be configured to perform the above methods disclosed in the embodiments of the present disclosure in any other appropriate manner (e.g., by means of firmware).
[0138] The embodiment of the present disclosure further provides a computer-readable storage medium, wherein when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the above method disclosed in the embodiment of the present disclosure.
[0139] The computer-readable storage medium in the disclosed embodiments may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. The computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. More specifically, the computer-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0140] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.
[0141] The embodiments of the present disclosure further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor, the above method disclosed in the embodiments of the present disclosure is implemented.
[0142] In embodiments of the present disclosure, computer program codes for performing the operations of the present disclosure may be written in one or more programming languages or combinations thereof, including but not limited to object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer.
[0143] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0144] The modules, components or units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the names of the modules, components or units do not, in some cases, limit the modules, components or units themselves.
[0145] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.
[0146] The above descriptions are only some embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, a technical solution formed by replacing the above features with the technical features with similar functions disclosed in the present disclosure (but not limited to).
[0147] Although some specific embodiments of the present disclosure have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. It should be understood by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.
Claims
1. A method for generating three-dimensional image data, characterized in that: The method comprises: Acquire N frames of to-be-processed three-dimensional image data of a target object; the N frames of to-be-processed three-dimensional image data include a sequence of to-be-processed three-dimensional image data of a target part of the target object obtained within a target time period; wherein N is a positive integer greater than 1; Based on the first frame of the N frames of the three-dimensional image data to be processed, rigidly aligning the reference three-dimensional image data; Based on the N frames of three-dimensional image data to be processed, non-rigid alignment is performed on the rigidly aligned reference three-dimensional image data to determine a target three-dimensional image data sequence of the target object; wherein, based on the N frames of three-dimensional image data to be processed, non-rigid alignment is performed on the rigidly aligned reference three-dimensional image data including, based on the i-th frame of three-dimensional image data to be processed among the N frames of three-dimensional image data to be processed, non-rigid alignment is performed on the reference three-dimensional image data that has been aligned with the i-1-th frame of three-dimensional image data to be processed to obtain reference three-dimensional image data aligned with the i-th frame of three-dimensional image data to be processed; wherein i is a positive integer greater than 1.
2. The method according to claim 1, characterized in that The non-rigid alignment of the rigidly aligned reference three-dimensional image data comprises: Based on the N frames of 3D image data to be processed and the target constraint conditions, non-rigid alignment is performed on the rigidly aligned reference 3D image data; wherein the target constraint conditions include at least one of the following: nearest neighbor constraint, smoothness constraint, stiffness constraint and key point constraint.
3. The method according to claim 2, characterized in that The non-rigid alignment of the rigidly aligned reference 3D image data based on the N frames of 3D image data to be processed and the target constraint condition comprises: Based on the fact that the weighted sum of the function of the nearest neighbor constraint, the function of the smoothness constraint, the function of the stiffness constraint and the function of the key point constraint is minimized, non-rigidly aligning the rigidly aligned reference 3D image data according to the N frames of 3D image data to be processed; Among them, the function of the nearest neighbor constraint is used to constrain the distance between the vertices in the N frames of three-dimensional image data to be processed and the nearest vertices in the non-rigidly aligned reference three-dimensional image data; the function of the smoothness constraint is used to constrain the Laplace coordinates of the vertices in the non-rigidly aligned reference three-dimensional image data; the function of the stiffness constraint is used to constrain the degree of change between vertices with connection relationships in the non-rigidly aligned reference three-dimensional image data; the function of the key point constraint is used to constrain the distance between the key points in the N frames of three-dimensional image data to be processed and the key points in the non-rigidly aligned reference three-dimensional image data.
4. The method according to claim 2, characterized in that: The target constraint condition further includes a non-deformation region constraint, wherein a function of the non-deformation region constraint is used to constrain a degree of change of the non-deformation region in the reference three-dimensional image data after the non-rigid alignment; The non-rigid alignment of the rigidly aligned reference three-dimensional image data based on the N frames of three-dimensional image data to be processed and the target constraint condition comprises: Based on the fact that the weighted sum of the function of the nearest neighbor constraint, the function of the smoothness constraint, the function of the key point constraint, the function of the non-deformation area constraint and the function of the stiffness constraint is minimized, non-rigid alignment is performed on the rigidly aligned reference three-dimensional image data according to the N frames of three-dimensional image data to be processed.
5. The method according to claim 1, characterized in that The step of rigidly aligning the first frame of to-be-processed three-dimensional image data with the reference three-dimensional image data based on the key points of the first frame of to-be-processed three-dimensional image data of the target part comprises: Acquire a first frame image of the target object; the first frame image corresponds to the first frame of three-dimensional image data to be processed; Extracting key points from the first frame of image; Obtaining a three-dimensional key point corresponding to the key point in the first frame of three-dimensional image data to be processed; Acquire annotated three-dimensional key points of the reference three-dimensional image data; The reference three-dimensional image data is deformed based on the three-dimensional key points and the annotated three-dimensional key points.
6. The method according to claim 1, characterized in that The non-rigid alignment of the rigidly aligned reference three-dimensional image data based on the N frames of to-be-processed three-dimensional image data comprises: According to the first frame of to-be-processed three-dimensional image data, non-rigid alignment is performed on the rigidly aligned reference three-dimensional image data.
7. The method according to claim 1, characterized in that The method further comprises: Acquire N frames of initial to-be-processed three-dimensional image data and N frames of images of the target object; wherein the N frames of initial to-be-processed three-dimensional image data correspond to the N frames of images respectively; Respectively obtaining segmentation masks of the N frames of images corresponding to the target parts; Projecting the N frames of initial three-dimensional image data to be processed into coordinate systems corresponding to the N frames of images respectively, and determining vertices in the N frames of initial three-dimensional image data to be processed that do not belong to the target part based on the segmentation mask; The vertices that do not belong to the target part in the N frames of initial three-dimensional image data to be processed are removed to obtain N frames of three-dimensional image data to be processed of the target object.
8. A device for generating three-dimensional image data, characterized in that: The device comprises: A data acquisition module, configured to acquire N frames of to-be-processed three-dimensional image data of a target object; the N frames of to-be-processed three-dimensional image data include a sequence of to-be-processed three-dimensional image data of a target part of the target object obtained within a target time period; wherein N is a positive integer greater than 1; A rigid alignment module, configured to perform rigid alignment on the reference three-dimensional image data based on the first frame of the N frames of three-dimensional image data to be processed; A non-rigid alignment module is used to perform non-rigid alignment on rigidly aligned reference three-dimensional image data based on the N frames of three-dimensional image data to be processed, so as to determine a target three-dimensional image data sequence of the target object; wherein, based on the N frames of three-dimensional image data to be processed, performing non-rigid alignment on rigidly aligned reference three-dimensional image data includes, based on the i-th frame of three-dimensional image data to be processed among the N frames of three-dimensional image data to be processed, performing non-rigid alignment on reference three-dimensional image data that has been aligned with the i-1-th frame of three-dimensional image data to be processed, so as to obtain reference three-dimensional image data aligned with the i-th frame of three-dimensional image data to be processed; wherein i is a positive integer greater than 1.
9. An electronic device, characterized in that: include: at least one processor; a memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method as claimed in any one of claims 1 to 7.
Citation Information
Patent Citations
3D image data registration method and device
CN105844582A
Method and device applied to vascular interventional operation navigation based on 3D / 2D registration
CN111784727A
Three-dimensional reconstruction method, electronic equipment and storage medium
CN112991524A
Techniques for removing and synthesizing secondary dynamics in facial performance capture
US11151767B1
Three-dimensional facial model generation method and apparatus, device, and medium
WO2021120810A1
