A method for repairing 3D models of human organs based on graph convolutional generative networks

Abstract

The application discloses a kind of based on graph convolution generation network human organ three-dimensional model repair method, belong to computer vision technical field, comprising: S1, training phase: by structured data set training a generation model, after training, the decoder part is extracted separately, as generator;S2, inference phase: for the input data of defect, by optimization method, with generator iterative solution optimal fitting result;S3, post-processing phase: according to the fitting result of step S2, obtain repair model.The application adopts the above-mentioned one based on graph convolution generation network human organ three-dimensional model repair method, can be full automatic according to the input of defect face three-dimensional model, generate completion result, and the quality of repair area meets clinical use demand.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method for repairing three-dimensional models of human organs based on graph convolutional generative networks. Background Technology

[0002] In hospitals, patients with facial defects are frequently encountered. These defects, caused by various reasons (accidents, illnesses, etc.), result in varying degrees of facial damage. During diagnosis, a digital facial model is obtained after facial scanning and appropriate adjustments by the doctor. The purpose of this invention is to generate a complete facial model based on this defective model, ensuring the consistency of non-defective areas and the edge fit of the repaired area to meet clinical application standards. The doctor can then directly obtain a three-dimensional model of the repaired area generated by this invention for subsequent prosthesis design and manufacturing. This method is universal and can be applied to human organs or tissues such as the face and jawbone.

[0003] With the rapid development of deep learning, deep learning-based 3D model restoration methods have long surpassed traditional geometry-based methods in terms of efficiency and results. In computer science, a common approach to 3D model restoration (also known as point-cloud completion or shape completion) in recent years is to design an end-to-end neural network. This network extracts features from the missing point cloud model (or voxel structure model), compresses it to a low dimension, and then decodes and generates a complete point cloud model. Existing technologies include some work on shape completion in computer science. These methods are mainly evaluated on general-purpose 3D model datasets (such as ShapeNet). Because point cloud data itself lacks adjacency relationships between vertices, and voxel data often has resolution limitations due to its large memory footprint, existing methods often produce very coarse restoration results when applied to human organs (such as faces), and smoothing processes further lose a lot of detailed information.

[0004] In the medical field, there are generally two common methods for prosthesis design: one is the mirror method, which is relatively simple and straightforward. For example, if the patient's facial defect does not cross the midline, the shape of the mirrored area can be used as a reference for repair. This method can be used for symmetrical organs such as the face and jawbone. The other is the database method. Hospitals usually have a large case database, and during diagnosis, cases most similar to the patient can be selected from the database as a reference for prosthesis design. When the mirror method cannot be used, matching from the database often results in a lack of personalization in the repair results, and manual adjustments are required at the boundaries of the repair area to fit the patient's face. Summary of the Invention

[0005] The purpose of this invention is to provide a method for repairing three-dimensional models of human organs based on graph convolutional generative networks, so as to solve the problems existing in the above-mentioned background technology.

[0006] To achieve the above objectives, this invention provides a method for repairing three-dimensional models of human organs based on graph convolutional generative networks, comprising the following steps:

[0007] S1. Training phase: Train a generative model using a structured dataset. After training, extract the decoder part separately as the generator.

[0008] S2, Inference Stage: For incomplete input data (point cloud, grid, or any storage format), the optimal fitting result is obtained by iteratively solving the generator through optimization methods.

[0009] S3. Post-processing stage: Obtain the repair model based on the fitting results of step S2.

[0010] Preferably, the training phase in step S1 is used to obtain a generative model applicable to a specific human organ or region. Taking the face as an example, there are many publicly available facial model datasets, and hospitals also have non-public databases used for research. With ethical approval, non-rigid registration can be used to uniformly register the collected facial data to the same topological structure, thereby obtaining a structured dataset. There is no strict requirement for the size of this dataset; it is generally considered that several hundred to several thousand examples are sufficient to train a good model. Then, this dataset is used to train the model through a graph convolutional neural network.

[0011] Preferably, the graph convolutional neural network architecture extracts features from the 3D model both globally and locally. The basic feature extraction module consists of a three-layer structure: a graph convolutional layer, an activation layer, and a downsampling layer (local paths do not include a downsampling layer), while the generation module has the exact opposite structure. Since the repair results based on point cloud shape completion are quite coarse, the neural network in this repair method directly extracts features from the mesh structure. Because the mesh is an irregular structure, graph convolution is used to perform convolution operations on the mesh, fully utilizing the adjacency information stored in the mesh structure, making the method of this invention smoother and more refined in terms of results.

[0012] Preferably, the decoder in the training phase is used as the generator in the inference phase to perform generation tasks. It should be noted that in practical applications, this invention primarily uses the methods of the inference phase; the purpose of the training phase is only to train the generator for the corresponding part. For the defective input model M_d, the shape fitting process can be described as an optimization, as follows:

[0013]

[0014] st||z||2=1 (2)

[0015] Where (1) is the optimization objective: the average distance after rigid transformation, (2) is the constraint condition: the latent vector is normalized, z∈R^256 represents the latent space vector, and dec is the decoder. For a rigid transformation, D is the distance from the point to the surface.

[0016] Preferably, the optimal form is generally solved by iterative optimization. Specifically, the latent space vector z is first randomly initialized and a complete model is generated using a generator. Then, the loss is calculated using D (the formula for calculating the distance from a differentiable point to a surface is implemented, which is within the scope of programming and will not be elaborated here). The absolute difference between two adjacent loss calculations is used to determine whether the loss function has converged. If it has not converged, z is iteratively optimized. If it has converged, a complete model with a good fit is considered to have been generated.

[0017] Preferably, in the post-processing stage, for cases where there is inconsistency between the fitting results and the non-deficient regions between the input of the defective model, an engineering method is used to perform a series of processing on the fitting results, as follows:

[0018] S31. Identify the repair sites in the fitted results using the threshold method;

[0019] S32. Project the vertices of the non-defective region onto the surface of the input model along the normal direction;

[0020] S33. For each vertex of the missing region, find the K nearest neighbor in the extended part of the edge of the missing region, and translate it according to the displacement of the K nearest neighbor;

[0021] S34. Smooth all areas of the defective area and the extended part.

[0022] Therefore, the present invention employs the above-mentioned method for repairing three-dimensional models of human organs based on graph convolutional generative networks, which has the following beneficial effects:

[0023] (1) Compared with shape completion methods in the field of computer science, the repair method of the present invention directly acts on the mesh structure, making the repair result smoother and more refined;

[0024] (2) Compared with existing clinical prosthesis design methods, the repair method proposed in this invention is more personalized and automated, and is a novel and efficient auxiliary method;

[0025] (3) In the completion test of clinical nasal defect cases, the average edge fit of the repaired area of ​​20 patients with nasal defects was 0.33 mm, and the repaired shape was also visually reasonable and met the needs of clinical use.

[0026] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of a graph convolutional neural network, representing an embodiment of a method for repairing three-dimensional models of human organs based on graph convolutional generative networks according to the present invention.

[0028] Figure 2 This is a flowchart of the repair method of the present invention;

[0029] Figure 3 This is a schematic diagram of a face model repair example in this invention;

[0030] Figure 4 This is a schematic diagram illustrating the application of the present invention in the repair of nasal defects in clinical cases. Detailed Implementation

[0031] Example

[0032] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0033] like Figure 1-2 As shown, a method for repairing 3D models of human organs based on graph convolutional generative networks includes the following steps:

[0034] S1. Training Phase: A generative model is trained using a structured dataset. After training, the decoder is extracted separately and used as the generator. This phase is used to obtain a generative model applicable to a specific human organ. Taking the face as an example, there are many publicly available facial datasets, and hospitals also have non-public databases used for research. With ethical approval, non-rigid registration can be used to uniformly register the collected facial data to the same topology, thus obtaining a structured dataset. There is no strict requirement for the size of this dataset; generally, several hundred to several thousand examples are considered sufficient to train a good model. Then, this dataset is used to train the model using a graph convolutional neural network.

[0035] The graph convolutional neural network architecture extracts features from the 3D model both globally and locally. The basic feature extraction module consists of a three-layer structure: a graph convolutional layer, an activation layer, and a downsampling layer (local paths do not include a downsampling layer). The generation module has the exact opposite structure. Since the repair results based on point cloud shape completion are quite coarse, the neural network in this repair method directly extracts features from the mesh structure. Because the mesh is an irregular structure, graph convolution is used to perform convolution operations on the mesh, fully utilizing the adjacency information stored in the mesh structure, making the method of this invention smoother and more refined in terms of results.

[0036] S2, Inference Stage: For incomplete input data (point cloud, grid, or any storage format), the optimal fitting result is obtained by iteratively solving the generator through optimization methods.

[0037] The decoder in the training phase is used by the generator in the inference phase to perform generation tasks. It should be noted that in practical applications, this invention primarily uses the methods from the inference phase; the purpose of the training phase is simply to train the generator for the corresponding part. For the defective input model M_d, the shape fitting process can be described as an optimization process, as follows:

[0038]

[0039] st||z||2=1 (2)

[0040] Where (1) is the optimization objective: the average distance after rigid transformation, (2) is the constraint condition: the latent vector is normalized, z∈R^256 represents the latent space vector, and dec is the decoder. For a rigid transformation, D is the distance from the point to the surface.

[0041] The optimal form is generally solved by iterative optimization. Specifically, the latent space vector z is first randomly initialized and a complete model is generated using a generator. Then, the loss is calculated using D (the formula for calculating the distance from a differentiable point to a surface is implemented, which is within the scope of programming and will not be elaborated here). The absolute difference between two adjacent loss calculations is used to determine whether the loss function has converged. If it has not converged, z is iteratively optimized. If it has converged, a complete model with a good fit is considered to have been generated.

[0042] S3. Post-processing stage: Obtain the repair model based on the fitting results of step S2.

[0043] In the post-processing stage, to address inconsistencies between the fitted results and the input of the defective model regarding non-defective regions, an engineering approach is employed to process the fitted results in a series of ways, as detailed below:

[0044] S31. Identify the repair sites in the fitted results using the threshold method;

[0045] S32. Project the vertices of the non-defective region onto the surface of the input model along the normal direction;

[0046] S33. For each vertex of the missing region, find the K nearest neighbor in the extended part of the edge of the missing region, and translate it according to the displacement of the K nearest neighbor;

[0047] S34. Smooth all areas of the defective area and the extended part.

[0048] By describing the implementation method in face model repair as an example of the present invention, it can be shown that all human organs for which structured datasets can be obtained can be repaired in the same way.

[0049] First, a proven computer hardware and software configuration for training and inference of neural networks is provided:

[0050] operating system Ubuntu 22.04 CUDA version 12.1 Python version 3.10 PyTorch version 2.1.0 GPU model NVIDIA GeForce RTX 4090

[0051] It should be noted that, as long as the training and inference process in this invention can be achieved, the above configuration can be replaced with any similar configuration, and is not limited to the configuration used during the verification of this invention.

[0052] Secondly, the preparation of the structured dataset is crucial. For faces, there are many publicly available datasets, such as FaceScape and Headspace. Alternatively, ethically approved non-public datasets can be used, such as facial data collected by hospitals with patient consent. After collection, a non-rigid registration method is used to register all face data to a unified topological structure. Non-rigid registration is a method well-known in the computer science field, and its principles will not be detailed here. After obtaining the structured dataset, the model is trained using the scheme in the training phase of this invention to obtain the generator.

[0053] For incomplete facial data, the inference stage scheme of this invention can be used for model repair to obtain a complete repair result. The repaired portion of the result can then be extracted separately to assist in the design of facial prostheses.

[0054] This invention has undergone rigorous verification in face model restoration, such as... Figure 3 As shown, the first row represents the constructed defect example, the second row represents the repair result, and the third row represents the real model. Furthermore, this invention also presents repair results based on facial models of 20 patients with nasal defects provided by a hospital. The average edge fit of the repaired area in these 20 cases is 0.33 mm, meeting clinical requirements. Figure 4 As shown, this is just one case.

[0055] It is worth mentioning other alternatives to this invention. First, the training data is unlimited and can be either public or private, as long as it is registered to a unified topology. Second, regarding the design of the generative network, this invention only provides a neural network structure that has been experimentally verified to be effective, without imposing strict limitations on its structure. As long as a generator that can be trained to meet the condition of "generating a complete three-dimensional model based on latent space parameters" is within the scope of protection of this invention.

[0056] Therefore, the present invention adopts the above-mentioned method for repairing three-dimensional models of human organs based on graph convolutional generative networks, which provides an automated and personalized solution for the design of human organ prostheses in clinical practice. From the perspective of computers, it is to generate a structured complete three-dimensional model given a defective three-dimensional model, and to ensure the consistency of non-defective areas and the tightness of the edges of the repaired area.

[0057] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1.A method for repairing a three-dimensional model of a human organ based on a graph convolution generation network, characterized in that, The method comprises the following steps: S1, a training phase: a generative model is trained through a structured data set, after the collection is completed, all face data is registered to a unified topological structure using a non-rigid registration method, the decoder part is extracted separately after the training is completed, as a generator, and a generative model applied to a specific human organ part is obtained; S2, an inference phase: for input data of human organ or tissue defects, the optimal fitting result is iteratively solved by the generator through an optimization method; S3, a post-processing phase: a repair model is obtained according to the fitting result of step S2; The training phase of step S1 is used to obtain a generative model applied to a specific human organ part, the collected data is uniformly non-rigidly registered to the same topological structure, and a structured data set is obtained, and the data set is used for model training through a graph convolutional neural network; The graph convolutional neural network architecture extracts features from the global and local three-dimensional models respectively, the basic module of feature extraction is a three-layer structure composed of a graph convolution layer, an activation layer and a down-sampling layer, and the generation module is a completely opposite structure; The decoder of the training phase is used for the generator of the inference phase, and for the input model M_d, the shape fitting process is described as an optimization form, as follows: （1） （2） where (1) is the optimization objective: the average distance after rigid transformation, (2) is the constraint: the hidden vector unitization, represents the hidden space vector, dec is the decoder, is the rigid transformation, is the point-to-plane distance; The optimal form is solved using an iterative optimization method, specifically: first, the latent space vector is randomly initialized. And use the generator to generate the complete model, then use Calculate the loss and determine whether the loss function has converged by the absolute difference between two consecutive loss calculations. If it has not converged, continue iterative optimization. If convergence is achieved, a complete model with a good fit is considered to have been generated. In the post-processing phase, for the inconsistent case between the fitting result and the non-defect region of the input defect model, an engineering method is used to process the fitting result, as follows: S31, the repair part of the fitting result is identified by a threshold method; S32, the vertices of the non-defect region are projected onto the surface of the input model along the normal direction; S33, for each vertex of the defect region, K-nearest neighbors are found in the extended part of the defect region edge, and the K-nearest neighbors are translated according to the displacement; S34, all regions of the defect region and the extended part are smoothed.

Images