A prediction method, apparatus, device and readable storage medium

Abstract

The specification discloses a prediction method, device, equipment and readable storage medium, wherein a three-dimensional face image is obtained by performing three-dimensional face reconstruction on an acquired face image of a user, then, based on a pre-trained prediction model and the three-dimensional face image of the user, a skull point cloud output by the prediction model is obtained, and further, a predicted skull surface of the user is obtained according to the skull point cloud output by the model. It can be seen that the user's face is directly used as input to predict the skull by using the prediction model, which is more accurate and faster, and the user's normal skull is predicted by using the three-dimensional face image of the user before injury, so that the accuracy is improved without relying on the personal experience of doctors.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a prediction method, apparatus, device, and readable storage medium. Background Technology

[0002] Craniofacial trauma refers to injuries to the head and face. Currently, craniofacial reconstruction surgery can repair defective skull and facial bones in patients with craniofacial trauma. This surgery requires a normal skull model of the patient as a reference to assist the surgeon in preoperative planning. Therefore, obtaining a normal skull model for patients who have suffered craniofacial trauma is a pressing issue that needs to be addressed.

[0003] Currently, based on three-dimensional medical images of patients with craniofacial trauma and three-dimensional medical images of other normal individuals, the differences between the patient's skull and the normal individual's skull can be quantified to determine the degree of trauma. Then, based on the degree of trauma and the doctor's clinical experience, a normal skull model of the patient can be constructed.

[0004] However, the above methods rely on the doctor's personal experience, and the resulting skull models have limited accuracy and low efficiency. Summary of the Invention

[0005] This specification provides a prediction method, apparatus, device, and readable storage medium to partially solve the aforementioned problems existing in the prior art.

[0006] The following technical solution is adopted in this specification:

[0007] This specification provides a prediction method, including:

[0008] Acquire a user's facial image, wherein the user's facial image includes a two-dimensional facial image or a depth image of the face;

[0009] A 3D face reconstruction is performed based on the user's face image to obtain the user's 3D face image;

[0010] The three-dimensional face image is sampled to obtain a face point cloud, and the face point cloud is input into a pre-trained prediction model to obtain a skull point cloud output by the prediction model;

[0011] Based on the skull point cloud, the predicted skull surface of the user is obtained.

[0012] Optionally, the three-dimensional face image is sampled to obtain a face point cloud, and the face point cloud is input into a pre-trained prediction model to obtain a skull point cloud output by the prediction model, specifically including:

[0013] Iterative execution: Based on the density of the previous sampling of the 3D face image, determine the current sampling density, and perform current sampling on the 3D face image according to the current sampling density to obtain the current face point cloud. Input the current face point cloud and the skull point cloud output by the pre-trained prediction model into the prediction model to obtain the current skull point cloud output by the prediction model, until the current skull point cloud meets the preset conditions.

[0014] Wherein, the current sampling density is greater than the density of the previous sampling, and the current density of the skull point cloud is greater than the density of the skull point cloud output by the prediction model in the previous sampling.

[0015] Optionally, the current face point cloud and the skull point cloud previously output by the pre-trained prediction model are input into the prediction model to obtain the current skull point cloud output by the prediction model, specifically including:

[0016] Extract multiple key points from the skull point cloud output by the pre-trained prediction model in the previous step.

[0017] The current face point cloud and the skull point cloud output by the prediction model are merged to obtain the point cloud to be processed.

[0018] Based on the distance between the point cloud to be processed and each key point, the point cloud to be processed is clustered to obtain each point cloud cluster;

[0019] For each point cloud cluster, the point cloud to be processed contained in the point cloud cluster is input into the prediction model to obtain the point cloud corresponding to the point cloud cluster output by the prediction model.

[0020] The current skull point cloud is determined based on the point clouds corresponding to each point cloud cluster.

[0021] Optionally, before extracting multiple keypoints from the skull point cloud previously output by the pre-trained prediction model, the method further includes:

[0022] The number of key points is determined based on the current sampling density, and the number of key points is positively correlated with the current sampling density;

[0023] Multiple key points are extracted from the skull point cloud output by the pre-trained prediction model, specifically including:

[0024] Based on the number of key points, key points are extracted from the skull point cloud output by the pre-trained prediction model in the previous iteration.

[0025] Optionally, the predicted skull surface of the user is obtained based on the skull point cloud, specifically including:

[0026] Obtain a 3D skull template;

[0027] Based on the three-dimensional skull template and the skull point cloud, the predicted skull surface of the user is obtained.

[0028] Optionally, before sampling the three-dimensional face image to obtain a face point cloud, the method further includes:

[0029] The three-dimensional face image is registered with a preset three-dimensional medical image to obtain a registered three-dimensional face image.

[0030] The process of sampling the three-dimensional face image to obtain a face point cloud specifically includes:

[0031] The registered 3D face image is sampled to obtain a face point cloud.

[0032] Optionally, the prediction model is pre-trained, specifically including:

[0033] A three-dimensional medical image is acquired in advance, and a three-dimensional skull image is obtained by segmentation from the three-dimensional medical image;

[0034] Determine the sampling density of the face point cloud and the sampling density of the skull point cloud; wherein the sampling density of the skull point cloud is greater than the sampling density of the face point cloud.

[0035] The three-dimensional medical image is sampled according to the face point cloud sampling density to obtain the face point cloud as a training sample;

[0036] The three-dimensional skull image is sampled according to the skull point cloud sampling density to obtain the skull point cloud as the annotation of the training sample;

[0037] The prediction model is trained based on the training samples and their annotations.

[0038] Optionally, the prediction model is trained based on the training samples and their annotations, specifically including:

[0039] Obtain a specified skull point cloud with the same density as the training sample, wherein the specified skull point cloud is obtained based on a prediction model trained on a face point cloud with a density lower than that of the training sample.

[0040] Multiple key points are extracted from the specified skull point cloud, and the training sample and the specified skull point cloud are merged to obtain the point cloud to be processed.

[0041] Based on each key point, the point cloud to be processed is clustered to obtain each point cloud cluster;

[0042] For each point cloud cluster, the point cloud to be processed contained in the point cloud cluster is input into the prediction model to be trained, and the predicted skull point cloud corresponding to the point cloud cluster is output by the prediction model.

[0043] The prediction model is trained with the goal of minimizing the difference between the predicted skull point cloud corresponding to each point cloud cluster and the annotation of the training sample.

[0044] Optionally, the current sampling density is determined based on the density of the previous sampling of the three-dimensional face image, specifically including:

[0045] When there is no previous sampling of the 3D face image, the current sampling density is determined as the initial sampling density;

[0046] Before inputting the current face point cloud and the skull point cloud previously output by the pre-trained prediction model into the prediction model to obtain the current skull point cloud output by the prediction model, the method further includes:

[0047] When there is no previous output of a pre-trained prediction model, an initial skull point cloud with the same density as the current face point cloud is obtained and used as the skull point cloud of the previous output of the prediction model.

[0048] This specification provides a prediction device, comprising:

[0049] A face image acquisition module is used to acquire a user's face image, wherein the user's face image includes a two-dimensional face image or a depth image of a face;

[0050] The 3D reconstruction module is used to perform 3D face reconstruction based on the user's face image to obtain the user's 3D face image;

[0051] The skull point cloud prediction module is used to sample the three-dimensional face image to obtain a face point cloud, and input the face point cloud into a pre-trained prediction model to obtain the skull point cloud output by the prediction model.

[0052] The predicted skull surface determination module is used to obtain the predicted skull surface of the user based on the skull point cloud.

[0053] This specification provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described prediction method.

[0054] This specification provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the prediction method described above.

[0055] The above-mentioned technical solutions adopted in this specification can achieve the following beneficial effects:

[0056] The prediction method provided in this manual involves reconstructing a 3D face from the acquired user's facial image to obtain a 3D facial image. Then, based on a pre-trained prediction model and the user's 3D facial image, a skull point cloud is generated from the model's output. Finally, the predicted skull surface of the user is obtained from this skull point cloud. It is evident that by using the user's face as input, the prediction model directly predicts the skull, resulting in higher accuracy and faster speed. Furthermore, by using the user's 3D facial image before the injury to predict the user's normal skull, it eliminates reliance on the doctor's personal experience, thereby improving accuracy. Attached Figure Description

[0057] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and their descriptions, serving to explain this specification and do not constitute an undue limitation thereof.

[0058] In the picture:

[0059] Figure 1 This is a flowchart illustrating one of the prediction methods described in this specification.

[0060] Figure 2 This is a flowchart illustrating one of the prediction methods described in this specification.

[0061] Figure 3 This is a schematic diagram of one of the prediction models in this specification;

[0062] Figure 4 This is a flowchart illustrating one of the prediction methods described in this specification.

[0063] Figure 5 This is a schematic diagram of one of the prediction models in this specification;

[0064] Figure 6 This is a flowchart illustrating one of the prediction methods described in this specification.

[0065] Figure 7 This is a schematic diagram of a prediction device provided in this specification;

[0066] Figure 8 The corresponding information provided in this specification Figure 1 A schematic diagram of an electronic device. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0068] Additionally, it should be noted that all actions involving the acquisition of signals, information, or data in this invention are carried out in compliance with the relevant data protection laws and regulations of the locality and with authorization from the owner of the relevant device.

[0069] Craniofacial trauma refers to injuries to the head and face. These injuries may include fractures, contusions, lacerations, soft tissue injuries, and brain injuries. Craniofacial trauma is often caused by factors such as traffic accidents, falls, assaults, and tumors. Due to the increasing incidence of accidents, the incidence of craniofacial trauma is also increasing year by year. Furthermore, because of the numerous facial tissues and organs, craniofacial trauma can easily cause facial deformities or even functional impairments, and in severe cases, there is a risk of disfigurement.

[0070] Craniomaxillofacial (CMF) reconstruction surgery is specifically designed to reconstruct such defects by restoring the flawed facial bones to their normal shape or desired pre-traumatic state. It aims to repair facial and head injuries and restore their normal form and function. Craniomaxillofacial reconstruction surgery may require various techniques and methods, including bone grafts, bone marrow fillers, and artificial materials. These surgeries need to be performed by experienced specialists and require a lengthy recovery period. The success of CMF reconstruction surgery largely depends on accurate preoperative planning. Preoperative planning requires a normal, healthy skull model of the patient as a reference, assisting the surgeon in the planning process and possessing high clinical value.

[0071] Currently, there are several methods for obtaining normal skull models for patients who have suffered craniofacial trauma: One method involves using 3D medical images of the patient with craniofacial trauma and comparing them with 3D medical images of normal individuals. This quantifies the differences between the patient's skull and that of normal individuals to determine the degree of trauma. Based on the degree of trauma and the doctor's clinical experience, a normal skull model is then constructed. However, this method relies heavily on the doctor's clinical experience, resulting in limited accuracy of the constructed skull model. Furthermore, the efficiency of doctors constructing normal skull models is low. Another method uses sparse learning to construct datasets of normal human faces and skulls. The patient's face is then compared to the faces in the dataset, and a weighted sum of the faces in the dataset is used to represent the patient's face. The weights obtained in the previous method are then applied to the normal human skull data to represent the patient's skull. The disadvantage of this method is that a large number of weights are needed to ensure the accuracy of the patient's skull, making weight acquisition difficult and inefficient.

[0072] Based on this, this specification provides a prediction method that uses a prediction model pre-trained from facial point clouds and skull point clouds obtained from sampling three-dimensional medical images, along with a user's three-dimensional facial image, to obtain a skull point cloud output by the prediction model. Furthermore, the predicted skull surface of the user is obtained based on the skull point cloud output by the model. By using the user's face as input, the prediction model directly predicts the skull, resulting in higher accuracy and faster speed. Moreover, by using a three-dimensional facial image of the user before injury to predict the user's normal skull, it no longer relies on the doctor's personal experience, thereby improving accuracy.

[0073] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0074] Figure 1 This is a flowchart illustrating a prediction method provided in this specification.

[0075] S100: Acquire a user's face image, wherein the user's face image includes a two-dimensional face image or a depth image of the face.

[0076] S102: Perform three-dimensional face reconstruction based on the user's face image to obtain the user's three-dimensional face image.

[0077] This specification provides a prediction method in its embodiments, wherein the prediction model involved can be pre-trained. The prediction method can be executed by an electronic device such as a server used to obtain a user's predicted skull surface. The electronic device executing the training process of the prediction model and the electronic device executing the prediction method can be the same or different, and this specification does not limit this.

[0078] In the embodiments of this specification, the term "user's face" refers to a part of the human body supported by the skull, including the user's head and facial tissues. The term "skull" refers to the skull bones above the spine. In practical applications, there is a correspondence between a user's face and skull; that is, there is a strong correlation between them. Therefore, the user's face image can be used to constrain the predicted normal skull surface to improve the accuracy and usability of the predicted skull surface. Furthermore, since the skull surface that can provide a reference for doctors in craniofacial reconstruction surgery for patients is usually a three-dimensional model, it is necessary to obtain a three-dimensional face image of the user before the craniofacial trauma to constrain the prediction of the three-dimensional skull surface.

[0079] Therefore, it is necessary to acquire the user's three-dimensional facial image, which can be reconstructed from the user's two-dimensional facial image. In practical applications, since acquiring photographs is relatively easy, a photograph of the user can be used as the two-dimensional facial image. The photograph of the user as the two-dimensional facial image can at least represent the user's craniofacial information, and this photograph can be acquired through any existing photograph acquisition method before the user suffered craniofacial trauma. Furthermore, when performing three-dimensional reconstruction based on the user's photograph, the three-dimensional facial reconstruction scheme can be any existing method, such as a 3D facial deformation statistical model, a high-precision three-dimensional facial deformation model (faceverse), etc., and this specification does not limit this approach.

[0080] In addition, in the embodiments of this specification, the depth image of the user's face can also be used as the basis for three-dimensional reconstruction, and three-dimensional face reconstruction can be performed based on the depth represented by the gray value of each pixel in the depth image of the user's face to obtain the user's three-dimensional face image.

[0081] Optionally, in certain application scenarios, if the user underwent medical image acquisition of the head before suffering craniofacial trauma, the two-dimensional face image can be a two-dimensional medical image of the user's head acquired before the trauma, such as CT images obtained through computed tomography (CT), MRI images obtained through magnetic resonance imaging (MRI), images obtained through low-dose positron emission tomography / magnetic resonance imaging (PET), or other modalities of medical images. Then, three-dimensional image reconstruction is performed based on multiple two-dimensional medical images to obtain the user's three-dimensional face image. This specification does not limit the specific type of two-dimensional face image.

[0082] In an optional embodiment of this specification, the user's three-dimensional facial image may be obtained by a three-dimensional facial data acquisition device before the user suffers craniofacial trauma. The three-dimensional facial data acquisition device may be any existing acquisition device for acquiring three-dimensional image data, such as a three-dimensional motion-sensing camera, a dynamic three-dimensional image scanner, etc., and this specification does not limit it.

[0083] S104: Sample the three-dimensional face image to obtain a face point cloud, and input the face point cloud into a pre-trained prediction model to obtain the skull point cloud output by the prediction model.

[0084] Specifically, the pre-trained prediction model is obtained by using face point clouds acquired from 3D medical images as training samples and skull point clouds acquired from 3D medical images as annotations for the training samples, and then training it iteratively. The model structure of this prediction model can be a pyramid-shaped cascaded network structure or a single neural network model structure. The neural network included in the prediction model can be any existing machine learning network such as PointMLP network or PointNet++ network. This specification does not limit the specific model structure of the prediction model.

[0085] In practical applications, a set of facial point clouds obtained from sampling a 3D facial image can be directly input into the prediction model. The prediction model can then directly predict a set of skull point clouds based on the input facial point clouds. For example, sampling a 3D facial image yields 4000 facial point clouds, which are then used as input to the prediction model to obtain 8000 skull point clouds as output.

[0086] Furthermore, by considering the relationship between the point cloud density of the training samples used in the training process and the point cloud density of the labeled training samples, the relationship between the point cloud density of the input and output of the prediction model can be determined. Generally, if the point cloud density of the training samples used in the model training is less than the point cloud density of the labeled samples, then the point cloud density of the face input to the prediction model will also be less than the point cloud density of the skull output by the prediction model in application.

[0087] It should be noted that the training samples and annotations used to train the prediction model are from 3D medical images because medical images can display both the user's skull image and the user's human tissue image. In this way, the sampled face point cloud and skull point cloud have a strong correlation, and no alignment and correction are required.

[0088] In the embodiments of this specification, the three-dimensional face image obtained in step S102 can be regarded as a large number of points distributed in space. Existing three-dimensional point cloud sampling methods can be used to sample face point clouds from the three-dimensional face image. These three-dimensional point cloud sampling methods can be existing methods such as voxelized mesh sampling, uniform sampling, and geometric sampling. The number of points in the sampled face point cloud can be determined according to the specific application scenario; this specification does not specifically limit the point cloud sampling method or the number of points in the sampled face point cloud.

[0089] S106: Based on the skull point cloud, obtain the predicted skull surface of the user.

[0090] After obtaining the predicted skull point cloud, line segments between the points can be established based on the predicted skull point cloud to achieve the transformation from points to a surface (mesh), thus obtaining the user's predicted skull surface. The user's predicted skull surface is used as the user's normal skull surface and compared with the user's skull surface after suffering craniofacial trauma. This quantifies the user's trauma degree and location, providing doctors with medical reference for craniofacial reconstruction surgery and assisting doctors in preoperative repair planning.

[0091] The prediction method described in this document involves reconstructing a 3D face from the acquired user's facial image to obtain a 3D facial image. Then, based on a pre-trained prediction model and the user's 3D facial image, a skull point cloud is generated from the model's output. Finally, the predicted skull surface of the user is obtained from this skull point cloud. It is evident that directly predicting the skull using the user's face as input results in higher accuracy and faster speed. Furthermore, by using the user's 3D facial image before injury to predict the user's normal skull, it eliminates reliance on the doctor's personal experience, thereby improving accuracy.

[0092] In one or more embodiments of this specification, in such Figure 1 Step S104 involves sampling the 3D face image to obtain a face point cloud, and inputting the face point cloud into a pre-trained prediction model to obtain the skull point cloud output by the prediction model. The accuracy of the skull point cloud output by the prediction model can be improved by iteratively inputting the face point cloud into the prediction model through multiple point cloud samplings. Figure 2 As shown, the specific steps are as follows:

[0093] S200: Determine the current sampling density based on the density of the previous sampling of the three-dimensional face image.

[0094] Specifically, the pre-trained prediction model uses face point clouds acquired from 3D medical images as training samples and skull point clouds acquired from 3D medical images as labeled training samples, and is obtained through multiple iterations of training. Taking the pyramid-shaped cascaded structure of the prediction model as an example, as follows... Figure 3 As shown, each layer of the pyramid contains a complete encoder and decoder structure. This means that the input to each layer of the prediction model's pyramid contains face point clouds, and the output is always a skull point cloud. Furthermore, to improve the accuracy of skull point cloud prediction through multiple iterations, the output skull point cloud of the previous layer can be added to the input of each layer of the pyramid. In other words, the skull point cloud from the previous generation of the prediction model, along with the currently sampled face point cloud, is used as the current input to the prediction model.

[0095] Based on this, multiple samplings can be performed on the 3D face image. Furthermore, since the density of the skull point cloud output by the prediction model is higher than the density of the face point cloud input by the prediction model, the sampling densities of the face point clouds obtained from multiple samplings of the 3D face image will be different. Generally, the sampling density of the previous sampling of the 3D face image is lower than the current sampling density; that is, the density of the face point cloud obtained from the previous sampling of the 3D face image is lower than the density of the current face point cloud obtained from the current sampling. Therefore, it is necessary to increase the sampling density based on the sampling density of the previous sampling to obtain the current sampling density.

[0096] Optionally, the current sampling density can be determined based on the density of the skull point cloud output by the prediction model in the previous step, so that the density of the current face point cloud obtained by sampling the 3D face image based on the current sampling density is the same as the density of the skull point cloud output by the prediction model in the previous step, so that the current face point cloud and the skull point cloud output by the prediction model in the previous step can be input into the prediction model.

[0097] S202: Perform current sampling on the three-dimensional face image according to the current sampling density to obtain the current face point cloud.

[0098] S204: Input the current face point cloud and the skull point cloud previously output by the pre-trained prediction model into the prediction model to obtain the current skull point cloud output by the prediction model. Wherein, the current sampling density is greater than the previous sampling density, and the density of the current skull point cloud is greater than the density of the skull point cloud previously output by the prediction model.

[0099] Each input to the prediction model includes a face point cloud sampled from a 3D face image, and a skull point cloud from the previous output of the prediction model; the output is always a skull point cloud. Through multiple iterations, the predicted skull point cloud becomes more accurate with each layer. The number of layers in the pyramid-shaped prediction model can be flexibly adjusted according to the specific application scenario; this manual does not impose any limitations on this.

[0100] In one or more embodiments of this specification, in a pyramid-shaped prediction model, when there is no previous sampling of the 3D face image, the current sampling density is determined as the initial sampling density. Further, when there is no previous output of a pre-trained prediction model, an initial skull point cloud with the same density as the current face point cloud is obtained and used as the skull point cloud of the previous output of the prediction model. Here, "no previous sampling of the 3D face image" can be considered as the sampling density of the previous sampling of the 3D face image being zero, and the determined current sampling density as the initial sampling density can be preset; this specification does not limit this. Additionally, "no previous output of a pre-trained prediction model" means that the prediction model to be input to the current face point cloud is at the first layer of the pyramid. In this case, an initial skull point cloud selected from the database can be used as the skull point cloud of the previous output of the prediction model, and the density of this initial skull point cloud can be the same as the density of the current face point cloud.

[0101] Taking a pyramid-shaped prediction model with three layers as an example, such as Figure 3 As shown, F is a 3D face image, and face point clouds F1, F2, and F3 are face point clouds obtained by sampling the 3D face image F. The difference between F1, F2, and F3 lies in their density; that is, the sampling densities used to sample F when obtaining F1, F2, and F3 are different. For the first layer, the input of the prediction model is the face point cloud F1 and the initial skull point cloud B0. This is because, for the first layer, there is no previous output of the prediction model. Therefore, the initial skull point cloud B0 is selected from the database storing normal human skull point clouds as the input of the prediction model corresponding to the first layer. The output of the prediction model corresponding to the first layer is the current skull point cloud B1. Therefore, for the second layer, the input of the prediction model is the face point cloud F2 and the skull point cloud B1 from the previous output of the prediction model. This process continues, and the skull point cloud B3 is obtained after three iterations.

[0102] In the embodiments of this specification, the density of the input and output of the prediction model are different, that is, the density of the current face point cloud as input is lower than the density of the current skull point cloud as output.

[0103] It should be noted that, generally, the inputs to the prediction model—the sampled current face point cloud and the previously output skull point cloud—can have the same density. That is, the current face point cloud contains the same number of points as the previously output skull point cloud. The purpose of having a higher current sampling density than the previous sampling density is to progressively increase the density of the skull point cloud output by the prediction model, thereby improving the accuracy of the final skull surface constructed based on the skull point cloud. Generally, a denser skull point cloud can construct a more accurate skull surface. Since the density of the current face point cloud as input is greater than that of the face point cloud previously input to the prediction model, the density of the current skull point cloud is also greater than that of the previously output skull point cloud.

[0104] S206: Determine whether the current skull point cloud meets the preset conditions. If yes, proceed to step S208; otherwise, return to step S200.

[0105] Through multiple iterations, an increasingly dense skull point cloud is gradually obtained. In this step, the current skull point cloud is compared with preset conditions, and the result determines whether the current skull point cloud meets the preset conditions. For example, if the preset condition is that the density of the skull point cloud is greater than a preset density threshold, then if the density of the current skull point cloud is greater than the preset density threshold, it is determined that the current skull point cloud meets the preset conditions; otherwise, it does not. The preset conditions can be determined according to specific application scenarios, such as the number of iterations to obtain the current skull point cloud reaching a preset threshold, or the density of the current skull point cloud being higher than a preset density threshold. This specification does not impose any limitations on these conditions.

[0106] S208: End.

[0107] Based on such Figure 2 The scheme shown uses a pre-trained prediction model to obtain skull point clouds. By using the current face point cloud and the skull point cloud output by the prediction model in the previous step as inputs to the prediction model, and passing through a pyramid-shaped prediction model, the current skull point cloud is output multiple times in a progressively denser manner, thereby improving the accuracy of the skull surface constructed based on the skull point cloud.

[0108] In one or more embodiments of this specification, in such Figure 2 Step S204 involves inputting the current face point cloud and the skull point cloud previously output by the pre-trained prediction model into the prediction model to obtain the current skull point cloud output by the prediction model. This can be implemented in the following ways: Figure 4 As shown:

[0109] S300: Extracts multiple key points from the skull point cloud output by the pre-trained prediction model in the previous iteration.

[0110] In the embodiments of this specification, the skull point cloud output by the previous prediction model and the current face point cloud can be superimposed and merged, and then input together into the prediction model to obtain the current skull point cloud. During this process, the prediction model may be subjected to a large number of input point clouds, resulting in slow prediction speed or decreased accuracy. Therefore, it is possible to... Figure 5 As shown Figure 3 A schematic diagram of a layer in a pyramid-shaped prediction model, which can be represented by the skull point cloud B, the previous output, which serves as the input to the prediction model. i-1 And the current face point cloud F i The point cloud obtained after superposition and merging is clustered to achieve the goal of predicting skull point clouds by region. This method of obtaining the current skull point cloud by fine-grained prediction based on the prediction model can not only improve the prediction speed, but also improve the accuracy of skull point cloud.

[0111] Specifically, by clustering the merged point cloud, the actual effect is that the merged point clouds corresponding to different regions of the user's skull are input separately into the prediction model, and the prediction model outputs the corresponding skull point cloud for each region. Therefore, in this step, multiple key points can be extracted from the skull point cloud previously output by the prediction model, and these key points can be used as reference points to divide the user's skull into multiple regions. The method for extracting key points can be any existing facial landmark detection scheme; this specification does not limit this approach. Figure 5 As shown, by extracting key points from the skull point cloud output by the prediction model in the previous iteration, n key points were obtained. Figure 5 The example in the text shows 5 key points.

[0112] In an optional embodiment of this specification, before extracting multiple keypoints from the previously output skull point cloud of the pre-trained prediction model as shown in this step, the number of keypoints can be determined based on the current sampling density, where the number of keypoints is positively correlated with the current sampling density. That is, the higher the current sampling density, the more keypoints are extracted from the previously output skull point cloud of the prediction model. The purpose is to ensure that the input of the prediction model does not contain an excessively large number of point clouds, thereby improving prediction speed. Furthermore, by dividing the user's skull into increasingly finer-grained segments, the prediction accuracy of the skull point cloud can be improved. Based on this, in such... Figure 4 In step S300, when extracting multiple key points from the skull point cloud previously output by the pre-trained prediction model, key points can be extracted from the skull point cloud based on the number of key points. It is understood that, in cases such as... Figure 3In the pyramid-shaped prediction model shown, for each layer, the number of keypoints can be redefined when extracting keypoints from the previous output of the prediction model. This allows the number of keypoints extracted from each layer to be different. For example, for a pyramid-shaped prediction model... Figure 3 For the first layer (level 1), the number of key points extracted from the initial skull point cloud B0 can be 5. Furthermore, since the current sampling density of the current face point cloud in the second layer (level 2) is greater than that in the first layer (level 1), the number of key points extracted from the skull point cloud B1 in the second layer (level 2) can be 10. And so on; the higher the current sampling density, the more key points are extracted.

[0113] S302: Merge the current face point cloud and the skull point cloud output by the prediction model in the previous step to obtain the point cloud to be processed.

[0114] Specifically, in this step, the current face point cloud and the skull point cloud output by the prediction model are registered and then merged and overlaid to obtain the point cloud to be processed. The point clouds contained in the point cloud to be processed can represent the three-dimensional skull structure of the user's craniofacial region, or they can represent the three-dimensional soft tissue structure of the user's craniofacial region.

[0115] S304: Based on the distance between the point cloud to be processed and each key point, the point cloud to be processed is clustered to obtain each point cloud cluster.

[0116] For each point cloud contained in the point cloud to be processed, the distance between that point cloud and each key point is determined, the nearest target key point is identified, and the point cloud is assigned to the point cloud cluster corresponding to the target key point, thereby achieving clustering of the point cloud to be processed. The distance between the point cloud to be processed and each key point can be determined using any existing distance determination scheme, such as Euclidean distance, Chebyshev distance, etc., and this specification does not limit this approach.

[0117] S306: For each point cloud cluster, input the point cloud to be processed contained in the point cloud cluster into the prediction model to obtain the point cloud corresponding to the point cloud cluster output by the prediction model.

[0118] Furthermore, the point clouds to be processed contained in each point cloud cluster are taken as the point clouds contained in the user's craniofacial region corresponding to that point cloud cluster. The point clouds to be processed contained in that point cloud cluster are input into the prediction model, and the prediction model obtains the skull point cloud corresponding to the user's craniofacial region corresponding to that point cloud cluster. Figure 5 As shown, based on n key points and the point cloud x to be processed... iAfter obtaining n point cloud clusters through clustering, it can be intuitively observed that the craniofacial region of the user, represented by the point cloud to be processed, is divided into n regions. Each point cloud cluster contains x points of the point cloud to be processed. t , where the value of t ranges from [1, n].

[0119] S308: Determine the current skull point cloud based on the point clouds corresponding to each point cloud cluster.

[0120] The point clouds corresponding to each point cloud cluster output by the prediction model are stitched together to obtain the current skull point cloud. For example... Figure 5 The image shown is the current skull point cloud B. i It can be seen that it contains point clouds corresponding to n point cloud clusters.

[0121] In one or more embodiments of this specification, in such Figure 1 Step S104 shows the prediction of the user's skull surface based on the skull point cloud, which can be implemented as follows:

[0122] First, obtain a three-dimensional skull template.

[0123] Then, based on the three-dimensional skull template and the skull point cloud, the predicted skull surface of the user is obtained.

[0124] Specifically, after obtaining the predicted current skull point cloud, a registration method is used to register a selected 3D skull template onto the predicted current skull point cloud, and surface reconstruction is performed to obtain the predicted skull surface. The 3D skull template is then used to establish line segments between point clouds, realizing the transformation from points to a surface (mesh), ultimately resulting in a complete skull surface.

[0125] In an optional embodiment of this specification, in such a way... Figure 1 Before sampling the three-dimensional face image to obtain the face point cloud as shown in step S104, the three-dimensional face image can be registered to further improve the accuracy of the skull point cloud output by the prediction model. Specifically, the three-dimensional face image is registered with a preset three-dimensional medical image to obtain the registered three-dimensional face image.

[0126] The current face point cloud is registered onto a selected 3D medical image using a registration method. The registered face point cloud is then normalized. The registration method can be any existing method, such as Iterative Closest Point (ICP). The purpose of registration is to deform the current face point cloud under the constraints of the preset 3D medical image to eliminate noise and errors. This specification does not limit the specific registration method.

[0127] Based on the registered 3D face images described above, in such cases... Figure 1 When sampling the three-dimensional face image to obtain a face point cloud as shown in step S104, the registered three-dimensional face image can be sampled to obtain a face point cloud.

[0128] In one or more embodiments of this specification, the above-described... Figure 1 , Figure 2 , Figure 3 and Figure 4 The pre-trained prediction model involved can be pre-trained as follows, and as... Figure 6 As shown.

[0129] S400: Acquire a three-dimensional medical image in advance, and obtain a three-dimensional skull image by segmentation from the three-dimensional medical image.

[0130] In practical applications, three-dimensional medical images obtained through medical image scanning equipment may contain human tissues and skulls. By separating the human tissues and skulls using existing methods, a three-dimensional skull image can be obtained from the three-dimensional medical image. The segmentation method can be based on machine learning, manual segmentation, or segmentation based on specified parameters of the three-dimensional medical image (such as CT values). This specification does not limit the specific methods used.

[0131] S402: Determine the sampling density of the face point cloud and the sampling density of the skull point cloud; the sampling density of the skull point cloud is greater than the sampling density of the face point cloud.

[0132] Once the 3D skull image is determined, skull point clouds can be sampled from it and used as annotations for training the prediction model. Face point clouds, used as training samples for the prediction model, can be sampled from 3D medical images. Therefore, the point cloud sampling densities of both need to be determined before sampling.

[0133] Among them, the skull point cloud sampling density is greater than the face point cloud sampling density. Therefore, based on... Figure 3 The pyramid-shaped prediction model shown can achieve progressively denser skull point clouds when predicting skull point clouds, thereby improving the prediction accuracy of skull point clouds.

[0134] S404: The three-dimensional medical image is sampled according to the face point cloud sampling density to obtain a face point cloud as a training sample.

[0135] S406: The three-dimensional skull image is sampled according to the skull point cloud sampling density to obtain the skull point cloud as the annotation of the training sample.

[0136] In the embodiments of this specification, in order to enable the prediction model to learn the ability to predict skull point clouds from face point clouds from training samples and their annotations, face point clouds sampled from three-dimensional medical images are used as training samples, and skull point clouds sampled from three-dimensional skull images are used as annotations for training samples. A supervised model training method is adopted to train the prediction model.

[0137] S408: Train the prediction model based on the training samples and the annotations of the training samples.

[0138] In one or more embodiments of this specification, in such Figure 5 Step S408 shows the training of the prediction model based on the training samples and their annotations, specifically implemented through the following scheme:

[0139] Step 1: Obtain a specified skull point cloud with the same density as the training samples. The specified skull point cloud is obtained based on a prediction model trained on a face point cloud with a density lower than that of the training samples.

[0140] In the embodiments of this specification, when using a prediction model to predict the user's skull point cloud, the following methods can be employed: Figure 3 The pyramid-shaped, hierarchical prediction method shown predicts the user's skull point cloud at each layer based on a prediction model. The input to each prediction model includes the current face point cloud sampled from a 3D face image, and the skull point cloud output by the prediction model in the previous iteration. Therefore, during the training of the prediction model, a specified skull point cloud corresponding to the training samples can also be used as input to participate in the training. The specified skull point cloud is obtained by training a prediction model based on face point clouds with a lower density than the training samples. For example, when training a prediction model with 8000 point clouds, a prediction model trained with 4000 point clouds and their annotations can be introduced. By inputting the face point cloud of 4000 point clouds into the aforementioned trained prediction model, the prediction model outputs a skull point cloud containing 8000 point clouds. This skull point cloud output by the prediction model, containing 8000 point clouds, is then used as the specified skull point cloud, and together with the training samples containing 8000 point clouds, serves as the input to the current prediction model.

[0141] Step 2: Extract multiple key points from the specified skull point cloud, and merge the training sample and the specified skull point cloud to obtain the point cloud to be processed.

[0142] Step 3: Based on each key point, the point cloud to be processed is clustered to obtain each point cloud cluster.

[0143] Step 4: For each point cloud cluster, input the point cloud to be processed contained in the point cloud cluster into the prediction model to be trained, and obtain the predicted skull point cloud corresponding to the point cloud cluster output by the prediction model.

[0144] This is similar to the above. Figure 4 The S300 to S308 shown are similar, and will not be described again here.

[0145] Step 5: Train the prediction model with the goal of minimizing the difference between the predicted skull point cloud corresponding to each point cloud cluster and the annotation of the training sample.

[0146] In this step, since the prediction model provided in the embodiments of this specification is trained by supervised learning, the difference between the predicted skull point cloud corresponding to each point cloud cluster and the annotation of the training sample is determined, and the loss is determined based on the difference. The prediction model is trained with the minimization of the loss as the training objective.

[0147] Figure 7 A schematic diagram of a prediction device provided in this specification specifically includes:

[0148] The face image acquisition module 500 is used to acquire the face image of a user, wherein the face image of the user includes a two-dimensional face image or a depth image of a face;

[0149] The 3D reconstruction module 502 is used to perform 3D face reconstruction based on the user's face image to obtain the user's 3D face image;

[0150] The skull point cloud prediction module 504 is used to sample the three-dimensional face image to obtain a face point cloud, and input the face point cloud into a pre-trained prediction model to obtain the skull point cloud output by the prediction model; the prediction model is pre-trained based on the face point cloud and skull point cloud sampled from the three-dimensional medical image.

[0151] The predicted skull surface determination module 506 is used to obtain the predicted skull surface of the user based on the skull point cloud.

[0152] Optionally, the skull point cloud prediction module 504 is specifically configured to iteratively execute: determining the current sampling density based on the density of the previous sampling of the three-dimensional face image, and performing current sampling on the three-dimensional face image based on the current sampling density to obtain the current face point cloud; inputting the current face point cloud and the skull point cloud previously output by the pre-trained prediction model into the prediction model to obtain the current skull point cloud output by the prediction model, until the current skull point cloud meets preset conditions; wherein, the current sampling density is greater than the density of the previous sampling, and the density of the current skull point cloud is greater than the density of the skull point cloud previously output by the prediction model.

[0153] Optionally, the skull point cloud prediction module 504 is specifically used to: extract multiple key points from the skull point cloud previously output by the pre-trained prediction model; merge the current face point cloud and the skull point cloud previously output by the prediction model to obtain a point cloud to be processed; cluster the point cloud to be processed according to the distance between the point cloud to be processed and each key point to obtain each point cloud cluster; for each point cloud cluster, input the point cloud to be processed contained in the point cloud cluster into the prediction model to obtain the point cloud corresponding to the point cloud cluster output by the prediction model; and determine the current skull point cloud based on the point clouds corresponding to each point cloud cluster.

[0154] Optionally, before extracting multiple key points from the skull point cloud output by the pre-trained prediction model, the skull point cloud prediction module 504 is further configured to determine the number of key points based on the current sampling density, wherein the number of key points is positively correlated with the current sampling density.

[0155] Optionally, the skull point cloud prediction module 504 is specifically used to extract key points from the skull point cloud previously output by the pre-trained prediction model, based on the number of key points.

[0156] Optionally, the predicted skull surface determination module 506 is specifically used to: obtain a three-dimensional skull template; and obtain the user's predicted skull surface based on the three-dimensional skull template and the skull point cloud.

[0157] Optionally, the device further includes:

[0158] The registration module 508 is specifically used to register the three-dimensional face image with a preset three-dimensional medical image to obtain a registered three-dimensional face image.

[0159] Optionally, the skull point cloud prediction module 504 is specifically used to sample the registered three-dimensional face image to obtain a face point cloud.

[0160] Optionally, the device further includes:

[0161] The training module 510 is specifically used to pre-acquire a three-dimensional medical image and segment a three-dimensional skull image from the three-dimensional medical image; determine the sampling density of the face point cloud and the sampling density of the skull point cloud; the sampling density of the skull point cloud is greater than the sampling density of the face point cloud; sample the three-dimensional medical image according to the sampling density of the face point cloud to obtain a face point cloud as a training sample; sample the three-dimensional skull image according to the sampling density of the skull point cloud to obtain a skull point cloud as an annotation of the training sample; and train the prediction model according to the training sample and the annotation of the training sample.

[0162] Optionally, the training module 510 is specifically configured to: acquire a specified skull point cloud with the same density as the training sample, wherein the specified skull point cloud is obtained based on a prediction model trained on a face point cloud with a density lower than that of the training sample; extract multiple key points from the specified skull point cloud and merge the training sample and the specified skull point cloud to obtain a point cloud to be processed; cluster the point cloud to be processed according to each key point to obtain each point cloud cluster; for each point cloud cluster, input the point cloud to be processed contained in the point cloud cluster into the prediction model to be trained to obtain the predicted skull point cloud corresponding to the point cloud cluster output by the prediction model; and train the prediction model with the goal of minimizing the difference between the predicted skull point cloud corresponding to each point cloud cluster and the annotation of the training sample.

[0163] Optionally, the skull point cloud prediction module 504 is specifically used to determine the current sampling density as the initial sampling density when there is no previous sampling of the three-dimensional face image;

[0164] Optionally, before inputting the current face point cloud and the previously output skull point cloud of the pre-trained prediction model into the prediction model to obtain the current skull point cloud output by the prediction model, the skull point cloud prediction module 504 is further configured to, when there is no previously output of the pre-trained prediction model, obtain an initial skull point cloud with the same density as the current face point cloud as the previously output skull point cloud of the prediction model.

[0165] This specification also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 The prediction method shown.

[0166] This instruction manual also provides Figure 8 The diagram shows a schematic structural representation of the electronic device. Figure 8 At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for the business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to achieve the above-mentioned functions. Figure 1 The prediction method is shown. Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. In other words, the execution subject of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0167] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0168] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, ASICs, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0169] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0170] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware.

[0171] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0172] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0173] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0174] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0175] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0176] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0177] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0178] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0179] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0180] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0181] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0182] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A prediction method, characterized in that, include: Acquire a user's facial image, wherein the user's facial image includes a two-dimensional facial image or a depth image of the face; A 3D face reconstruction is performed based on the user's face image to obtain the user's 3D face image; The three-dimensional face image is sampled to obtain a face point cloud, and the face point cloud is input into a pre-trained prediction model to obtain a skull point cloud output by the prediction model. Specifically, the current sampling density is determined based on the density of the previous sampling of the three-dimensional face image, and the three-dimensional face image is sampled according to the current sampling density to obtain the current face point cloud. The current face point cloud and the skull point cloud output by the pre-trained prediction model are input into the prediction model to obtain the current skull point cloud output by the prediction model. This process continues until the current skull point cloud meets preset conditions: the current sampling density is greater than the density of the previous sampling, and the density of the current skull point cloud is greater than the density of the skull point cloud output by the prediction model. Based on the skull point cloud, the predicted skull surface of the user is obtained.

2. The method as described in claim 1, characterized in that, The current face point cloud and the skull point cloud previously output by the pre-trained prediction model are input into the prediction model to obtain the current skull point cloud output by the prediction model, specifically including: Extract multiple key points from the skull point cloud output by the pre-trained prediction model in the previous step. The current face point cloud and the skull point cloud output by the prediction model are merged to obtain the point cloud to be processed. Based on the distance between the point cloud to be processed and each key point, the point cloud to be processed is clustered to obtain each point cloud cluster; For each point cloud cluster, the point cloud to be processed contained in the point cloud cluster is input into the prediction model to obtain the point cloud corresponding to the point cloud cluster output by the prediction model. The current skull point cloud is determined based on the point clouds corresponding to each point cloud cluster.

3. The method as described in claim 2, characterized in that, Before extracting multiple keypoints from the skull point cloud previously output by the pre-trained prediction model, the method further includes: The number of key points is determined based on the current sampling density, and the number of key points is positively correlated with the current sampling density; Multiple key points are extracted from the skull point cloud output by the pre-trained prediction model, specifically including: Based on the number of key points, key points are extracted from the skull point cloud output by the pre-trained prediction model in the previous iteration.

4. The method as described in claim 1, characterized in that, Based on the skull point cloud, the predicted skull surface of the user is obtained, specifically including: Obtain a 3D skull template; Based on the three-dimensional skull template and the skull point cloud, the predicted skull surface of the user is obtained.

5. The method as described in claim 1, characterized in that, Before sampling the 3D face image to obtain a face point cloud, the method further includes: The three-dimensional face image is registered with a preset three-dimensional medical image to obtain a registered three-dimensional face image. The process of sampling the three-dimensional face image to obtain a face point cloud specifically includes: The registered 3D face image is sampled to obtain a face point cloud.

6. The method according to any one of claims 1 to 5, characterized in that, Pre-training the prediction model specifically includes: A three-dimensional medical image is acquired in advance, and a three-dimensional skull image is obtained by segmentation from the three-dimensional medical image; Determine the sampling density of the face point cloud and the sampling density of the skull point cloud; wherein the sampling density of the skull point cloud is greater than the sampling density of the face point cloud. The three-dimensional medical image is sampled according to the face point cloud sampling density to obtain the face point cloud as a training sample; The three-dimensional skull image is sampled according to the skull point cloud sampling density to obtain the skull point cloud as the annotation of the training sample; The prediction model is trained based on the training samples and their annotations.

7. The method as described in claim 6, characterized in that, The prediction model is trained based on the training samples and their annotations, specifically including: Obtain a specified skull point cloud with the same density as the training sample, wherein the specified skull point cloud is obtained based on a prediction model trained on a face point cloud with a density lower than that of the training sample. Multiple key points are extracted from the specified skull point cloud, and the training sample and the specified skull point cloud are merged to obtain the point cloud to be processed. Based on each key point, the point cloud to be processed is clustered to obtain each point cloud cluster; For each point cloud cluster, the point cloud to be processed contained in the point cloud cluster is input into the prediction model to be trained, and the predicted skull point cloud corresponding to the point cloud cluster is output by the prediction model. The prediction model is trained with the goal of minimizing the difference between the predicted skull point cloud corresponding to each point cloud cluster and the annotation of the training sample.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1 to 7.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method described in any one of claims 1 to 7.

Images