A method, device, medium and equipment for constructing facial bone structure
Through deep learning methods, the facial bone structure is extracted, which solves the problem of insufficient measurement accuracy of facial bone structure in the prior art, and achieves more accurate temporal orbital marginal positioning and ophthalmic disease diagnosis.
Patent Information
- Application Number
- CN202411506756.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-28
- Publication Date
- 2024-12-31
- Estimated Expiration
- 2044-10-28
AI Technical Summary
The prior art cannot accurately measure facial osteopathy, especially in the diagnosis of endocrine-related eye diseases, and the temporal orbital margin cannot be accurately positioned, resulting in insufficient measurement accuracy.
By obtaining a point cloud data set containing facial surface skin information and bone structure information, performing intermediate value filtering and sharpness enhancement, making a difference, combining CT generator and point cloud generator, point cloud and CT are trained using deep learning methods Data, the total cost function is calculated to extract facial bone structure.
It improves the measurement accuracy of facial bone structure, provides more accurate temporal orbital marginal positioning, and enhances the reliability of ophthalmic disease diagnosis.
Smart Images
Figure CN119048688B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of clinical optical imaging, and in particular to a method, device, medium and equipment for constructing facial bone structure. Background Art
[0002] Reconstructing three-dimensional facial bone structure is the basis for the diagnosis and treatment of clinical ophthalmic diseases. For example, in the diagnosis of endocrine-related eye diseases, the diagnosis of eyeball exophthalmia needs to be based on the vertical distance from the temporal orbital margin to the corneal vertex in the facial bone structure. Usually, a transparent ruler or an exophthalmosometer is used, which is manually measured by the doctor on the patient's body surface, but it is impossible to accurately locate the temporal orbital margin.
[0003] It can be seen from the above description that the facial bone structure cannot be accurately measured at present. How to improve the accuracy of the facial bone structure is a technical problem that technical personnel in this field urgently need to solve. Summary of the invention
[0004] In order to overcome the disadvantage of poor accuracy in the existing facial bone structure construction, the present invention provides a method, device, medium and equipment for constructing facial bone structure.
[0005] In order to achieve the above object, according to a first aspect of the present invention, an embodiment of the present invention provides a method for constructing a facial bone structure, the method comprising the following steps:
[0006] Acquire a first point cloud data set and a second point cloud data set, wherein the first point cloud data set includes facial surface skin information and subcutaneous tissue information, and the second point cloud data set includes facial surface skin information, subcutaneous tissue information, and bone structure information;
[0007] The second point cloud data set is subtracted from the first point cloud data set after median filtering and sharpness enhancement to obtain optical point cloud data;
[0008] The optical point cloud data is input into the first CT generator to obtain the first CT generated data; the first CT generated data and the CT measurement data are both input into the CT discriminator. If the CT discriminator successfully distinguishes the first CT generated data from the CT measurement data, the first CT generator is updated; otherwise, the CT discriminator is updated; wherein the CT discriminator is a 3-layer fully convolutional network structure,
[0009] The first CT generated data is input into the first point cloud generator to obtain the first point cloud generated data, wherein the first point cloud generated data and the optical point cloud data are used to calculate the point cloud adversarial cost function and the point cloud cycle consistency cost function;
[0010] The CT measurement data is input into the second point cloud generator to obtain the second point cloud generation data; the second point cloud generation data and the optical point cloud data are both input into the point cloud discriminator, and if the point cloud discriminator successfully distinguishes the second point cloud generation data from the optical point cloud data, the second point cloud generator is updated, otherwise, the point cloud discriminator is updated; wherein the point cloud discriminator is a 5-layer full convolutional network structure;
[0011] The second point cloud generated data is input into the second CT generator to obtain second CT generated data, wherein the second CT generated data and the CT measurement data are used to calculate the CT adversarial cost function and the CT cycle consistency cost function;
[0012] The point cloud adversarial cost function, the point cloud cycle consistency cost function, the CT adversarial cost function and the CT cycle consistency cost function constitute a total cost function;
[0013] According to the total cost function, a solver is used to calculate until convergence to obtain the facial bone structure.
[0014] Optionally, the second point cloud data set is subtracted from the first point cloud data set after median filtering and sharpness enhancement to obtain optical point cloud data, including:
[0015] Perform median filtering on the first point cloud data set using a 3×3×3 or 5×5×5 window;
[0016] The data after the intermediate value filtering is convolved using a preset matrix, wherein the preset matrix is as follows: [
[0018] [[ 0, 0, 0], [ 0, -1, 0], [ 0, 0, 0]],
[0019] [[ 0, -1, 0], [-1, 8, -1], [ 0, -1, 0]],
[0020] [[ 0, 0, 0], [ 0, -1, 0], [ 0, 0, 0]] ]
[0022] The second point cloud data set is subtracted from the first point cloud data set after convolution to obtain optical point cloud data.
[0023] Optionally, the optical point cloud data is input into a first CT generator to obtain first CT generated data, including:
[0024] The first CT generator downsamples the optical point cloud data five times using a 3×3×3 convolution with a step size of 2, and symmetrically upsamples the optical point cloud data using a 3×3×3 transposed convolution, and finally uses a linear rectification function as an activation function to obtain the first CT generated data;
[0025] The second point cloud generation data is input into a second CT generator to obtain second CT generation data, including:
[0026] The second CT generator downsamples the second point cloud generation data 5 times using a 3×3×3 convolution with a stride of 2, and symmetrically upsamples using a 3×3×3 transposed convolution, and finally uses a linear rectification function as the activation function to obtain the second CT generation data.
[0027] Optionally, the point cloud adversarial cost function, the point cloud cycle consistency cost function, the CT adversarial cost function and the CT cycle consistency cost function constitute a total cost function, and the formula is as follows:
[0028] ,
[0029] Where L represents the total cost function, represents the point cloud adversarial cost function, represents the CT adversarial cost function, represents the point cloud cycle consistency cost function, represents the CT cycle consistency cost function, and is a constant between 0 and 10.
[0030] According to a second aspect of the present invention, an embodiment of the present invention further provides a facial bone structure construction device, comprising:
[0031] An acquisition module, configured to acquire a first point cloud data set and a second point cloud data set, wherein the first point cloud data set includes facial surface skin information and subcutaneous tissue information, and the second point cloud data set includes facial surface skin information, subcutaneous tissue information, and bone structure information;
[0032] A difference module, used for performing difference between the second point cloud data set and the first point cloud data set after median filtering and sharpness enhancement to obtain optical point cloud data;
[0033] A first updating module is used for inputting optical point cloud data into a first CT generator to obtain first CT generated data; both the first CT generated data and the CT measurement data are input into a CT discriminator, and if the CT discriminator successfully distinguishes the first CT generated data from the CT measurement data, the first CT generator is updated, otherwise, the CT discriminator is updated; wherein the CT discriminator is a 3-layer fully convolutional network structure;
[0034] A first cost function calculation module, used for inputting first CT generated data into a first point cloud generator to obtain first point cloud generated data, wherein the first point cloud generated data and the optical point cloud data are used to calculate a point cloud adversarial cost function and a point cloud cycle consistency cost function;
[0035] The second updating module is used for inputting CT measurement data into the second point cloud generator to obtain second point cloud generation data; the second point cloud generation data and the optical point cloud data are both input into the point cloud discriminator, and if the point cloud discriminator successfully distinguishes the second point cloud generation data from the optical point cloud data, the second point cloud generator is updated, otherwise, the point cloud discriminator is updated; wherein the point cloud discriminator is a 5-layer full convolutional network structure;
[0036] A second cost function calculation module, used for inputting second point cloud generated data into a second CT generator to obtain second CT generated data, wherein the second CT generated data and CT measurement data are used to calculate a CT adversarial cost function and a CT cycle consistency cost function;
[0037] A total cost function module, used for combining the point cloud adversarial cost function, the point cloud cycle consistency cost function, the CT adversarial cost function and the CT cycle consistency cost function into a total cost function;
[0038] The solution module is used to calculate the facial bone structure using a solver until convergence according to the total cost function.
[0039] Optionally, the difference module is further used for:
[0040] Perform median filtering on the first point cloud data set using a 3×3×3 or 5×5×5 window;
[0041] The data after the intermediate value filtering is convolved using a preset matrix, wherein the preset matrix is as follows: [
[0043] [[ 0, 0, 0], [ 0, -1, 0], [ 0, 0, 0]],
[0044] [[ 0, -1, 0], [-1, 8, -1], [ 0, -1, 0]],
[0045] [[ 0, 0, 0], [ 0, -1, 0], [ 0, 0, 0]] ]
[0047] The second point cloud data set is subtracted from the first point cloud data set after convolution to obtain optical point cloud data.
[0048] Optionally, the first updating module is further used for:
[0049] The first CT generator downsamples the optical point cloud data five times using a 3×3×3 convolution with a step size of 2, and symmetrically upsamples the optical point cloud data using a 3×3×3 transposed convolution, and finally uses a linear rectification function as an activation function to obtain the first CT generated data;
[0050] The second update module is also used for:
[0051] The second CT generator downsamples the second point cloud generation data 5 times using a 3×3×3 convolution with a stride of 2, and symmetrically upsamples using a 3×3×3 transposed convolution, and finally uses a linear rectification function as the activation function to obtain the second CT generation data.
[0052] Optionally, the total cost function module combines the point cloud adversarial cost function, the point cloud cycle consistency cost function, the CT adversarial cost function and the CT cycle consistency cost function into a total cost function, and the formula is as follows:
[0053] ,
[0054] Where L represents the total cost function, represents the point cloud adversarial cost function, represents the CT adversarial cost function, represents the point cloud cycle consistency cost function, represents the CT cycle consistency cost function, and is a constant between 0 and 10.
[0055] According to the third aspect of the present invention, an embodiment of the present invention further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method for constructing facial bone structure in any of the above embodiments are implemented.
[0056] According to a fourth aspect of the present invention, an embodiment of the present invention further provides a storage medium, wherein the storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the steps of the method for constructing facial bone structure in any of the above embodiments.
[0057] As described above, a method, apparatus, medium and device for facial bone structure provided by an embodiment of the present invention have the following beneficial effects: by acquiring a first point cloud data set and a second point cloud data set, wherein the first point cloud data set includes facial surface skin information and subcutaneous tissue information, and the second point cloud data set includes facial surface skin information, subcutaneous tissue information and bone information; the second point cloud data set is subtracted from the first point cloud data set after median filtering and sharpness enhancement to obtain optical point cloud data; the optical point cloud data is input into a first CT generator to obtain first CT generated data; the first CT generated data and the CT measurement data are both input into a CT discriminator, if the CT discriminator successfully distinguishes the first CT generated data from the CT measurement data, the first CT generator is updated, otherwise, the CT discriminator is updated; wherein the CT discriminator is a 3-layer fully convolutional network structure; the first CT generated data is input into the first point cloud generator to obtain the first point cloud generated data The first point cloud generated data and the optical point cloud data are used to calculate the point cloud adversarial cost function and the point cloud cycle consistency cost function; the CT measurement data is input into the second point cloud generator to obtain the second point cloud generated data; the second point cloud generated data and the optical point cloud data are both input into the point cloud discriminator, if the point cloud discriminator successfully distinguishes the second point cloud generated data from the optical point cloud data, the second point cloud generator is updated, otherwise, the point cloud discriminator is updated; wherein the point cloud discriminator is a 5-layer fully convolutional network structure; the second point cloud generated data is input into the second CT generator to obtain the second CT generated data, the second CT generated data and the CT measurement data are used to calculate the CT adversarial cost function and the CT cycle consistency cost function; the point cloud adversarial cost function, the point cloud cycle consistency cost function, the CT adversarial cost function and the CT cycle consistency cost function constitute the total cost function; according to the total cost function, the solver is used to calculate until convergence to obtain the facial bone structure. The present invention extracts deep information describing the facial bone structure, and then uses a deep learning method based on the extracted bone structure information to train the facial bone structure from optical point cloud data and CT measurement data, thereby effectively improving the accuracy of the facial bone structure. BRIEF DESCRIPTION OF THE DRAWINGS
[0058] Figure 1 It is a schematic flow chart of a method for constructing facial bone structure provided by an embodiment of the present invention;
[0059] Figure 2 It is a schematic diagram of a process of subtracting a point cloud data set provided by an embodiment of the present invention;
[0060] Figure 3 It is a schematic diagram of point cloud data set difference calculation provided by an embodiment of the present invention;
[0061] Figure 4 It is a schematic diagram of a training process provided by an embodiment of the present invention;
[0062] Figure 5 It is a schematic diagram of the structure of a facial bone structure construction device provided by an embodiment of the present invention;
[0063] Figure 6 It is a schematic diagram of the hardware structure of an electronic device for executing the facial bone structure construction method provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0064] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0065] See also Figures 1 to 6 It should be noted that the illustrations provided in this embodiment are only used to illustrate the basic concept of the present invention in a schematic manner, and the illustrations only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.
[0066] See also Figure 1 , is a flow chart of a method for constructing facial bone structure provided by an embodiment of the present invention, such as Figure 1 As shown, the embodiment of the present invention shows the process of constructing facial bone structure.
[0067] Step S101: Acquire a first point cloud data set and a second point cloud data set, wherein the first point cloud data set includes facial surface skin information and subcutaneous tissue information, and the second point cloud data set includes facial skin information, subcutaneous tissue information, and bone information.
[0068] In specific implementation, near-infrared laser radar can be used, which has poor tissue penetration ability, but can obtain facial surface skin information and subcutaneous tissue information, that is, the first point cloud data set in this application. Further, combined with the use of mid-infrared laser radar, which has strong tissue penetration ability and its signal contains facial bone structure information, that is, the second point cloud data set, the second point cloud data set includes facial surface skin information, subcutaneous tissue information and facial bone structure information. Combining the two can remove the influence of the facial surface skin and subcutaneous tissue, thereby extracting deep facial tissue information.
[0069] Of course, the above-mentioned first point cloud data set and second point cloud data are sets of three-dimensional coordinates of the facial surface skin, subcutaneous tissue and bone structure. The specific method of obtaining them is not limited to the use of near-infrared laser radar and mid-infrared laser radar, but may also include other medical scanning methods, which are not limited in this application.
[0070] Step S102: The second point cloud data set is subtracted from the first point cloud data set after median filtering and sharpness enhancement to obtain optical point cloud data.
[0071] See also Figure 2 , which is a schematic diagram of a process of performing subtraction of a point cloud data set provided by an embodiment of the present invention, and also see Figure 3 , is a schematic diagram of point cloud data set difference calculation provided by an embodiment of the present invention. As shown in the figure, the difference process includes the following steps:
[0072] Step S1021: Use a 3×3×3 or 5×5×5 window to perform median filtering on the first point cloud data set.
[0073] In an exemplary embodiment, a window of any size can be used to perform median filtering on the first point cloud data set. In the present application, a 3×3×3 or 5×5×5 window is preferably used for median filtering to remove signal noise and obtain better quality data information. The specific method of median filtering will not be repeated in this application.
[0074] Step S1022: Use a preset matrix to convolve the data after the intermediate value filtering.
[0075] In an exemplary embodiment, the preset matrix is a three-dimensional matrix, that is, Figure 3 The image sharpening matrix shown in is as follows: [
[0077] [[ 0, 0, 0], [ 0, -1, 0], [ 0, 0, 0]],
[0078] [[ 0, -1, 0], [-1, 8, -1], [ 0, -1, 0]],
[0079] [[ 0, 0, 0], [ 0, -1, 0], [ 0, 0, 0]] ]
[0081] Through the convolution process, the sharpness of the first point cloud data set after the median filtering can be enhanced.
[0082] Step S1023: Subtract the second point cloud data set from the convolved first point cloud data set to obtain optical point cloud data.
[0083] Through the above steps, the first point cloud dataset has completed median filtering (noise removal) and sharpness enhancement. The second point cloud dataset is further used to perform subtraction on the first point cloud dataset after convolution, thereby removing the facial surface skin information and subcutaneous tissue information and retaining the bone structure information.
[0084] See also Figure 3 As shown in the figure, the present application performs median filtering on the first point cloud data set (window size is 3x3x3 or 5x5x5), and performs sharpness enhancement on the filtered data (convolution of the 3x3 matrix in the above embodiment). The processed data is subtracted from the second point cloud data set to obtain optical point cloud data, which is one of the basic data for constructing facial bone structure.
[0085] Step S103: the optical point cloud data is input into the first CT generator to obtain the first CT generated data; the first CT generated data and the CT measurement data are both input into the CT discriminator, if the CT discriminator successfully distinguishes the first CT generated data from the CT measurement data, the first CT generator is updated, otherwise, the CT discriminator is updated; wherein the CT discriminator is a 3-layer fully convolutional network structure.
[0086] See also Figure 4 , is a training process diagram provided by an embodiment of the present invention. In order to clearly describe the facial bone structure construction method of the present application, the subsequent training steps will be described in detail in conjunction with this figure.
[0087] The optical point cloud data is the total input of the neural network. First, the optical point cloud data is input into the trained modal transfer generation neural network to generate CT-like bone structures. That is, correspondingly, the optical point cloud data obtained in the above steps is input into the first CT generator to obtain the first CT generated data, which is obtained from the optical point cloud data and has a CT-like bone structure.
[0088] In a specific implementation, the first CT generator downsamples the optical point cloud data five times using a 3×3×3 convolution with a step size of 2, and symmetrically upsamples using a 3×3×3 transposed convolution, and finally uses a linear rectification function as an activation function to obtain the first CT generated data.
[0089] Moreover, in an exemplary embodiment, the first CT generator has a Unet structure with long-range jump connections, and the specific activation function form and neural network structure are not repeated in this embodiment.
[0090] Further, the generated first CT generated data and CT measurement data are input into the CT discriminator. Among them, the CT measurement data is the three-dimensional bone structure data obtained by clinical CT scanning, which is the gold standard (the goal to be achieved) or label (Label) used in neural network training. The CT discriminator uses a three-layer fully convolutional network structure to classify the same modality data (generated and measured) as true or false. If the CT discriminator can accurately distinguish the first CT generated data and the CT measurement data, the first CT generator is updated; otherwise, if the CT discriminator cannot accurately distinguish the two, the CT discriminator is updated. The specific method of updating the CT discriminator and the first CT generator can be achieved by adjusting the corresponding parameters, which is not limited in this application.
[0091] Moreover, the gold standard (the goal to be achieved) or label (Label) used in the neural network training of this embodiment is not limited to CT test data, but can also be other three-dimensional imaging modality data including MRI (English: Magnetic Resonance Imaging, Chinese: Magnetic Resonance Imaging) data, etc., so as to meet different training needs.
[0092] Step S104: the first CT generated data is input into the first point cloud generator to obtain the first point cloud generated data, wherein the first point cloud generated data and the optical point cloud data are used to calculate the point cloud adversarial cost function and the point cloud cycle consistency cost function.
[0093] The first CT generated data is also input into the first point cloud generator to generate the first point cloud generated data. The first point cloud generator has a similar structure to the first CT generator in the above embodiment and will not be described in detail in this embodiment.
[0094] The first point cloud generation data and the optical point cloud data are combined to calculate the point cloud adversarial cost function in the regularization term of the cost function. And the point cloud cycle consistency cost function These two functions will form a total cost function in subsequent steps to update the neural network. The calculation of the above two cost functions using the first point cloud data and the optical point cloud data is a commonly used calculation method in the art and will not be repeated in this application.
[0095] Step S105: The CT measurement data is input into the second point cloud generator to obtain the second point cloud generation data; the second point cloud generation data and the optical point cloud data are both input into the point cloud discriminator. If the point cloud discriminator successfully distinguishes the second point cloud generation data from the optical point cloud data, the second point cloud generator is updated; otherwise, the point cloud discriminator is updated; wherein the point cloud discriminator is a 5-layer fully convolutional network structure.
[0096] Similarly, the CT measurement data enters the second point cloud generator to generate the second point cloud generation data, the second point cloud generation data enters the point cloud discriminator, and the optical point cloud data also enters the point cloud discriminator. According to the discrimination result, if the point cloud discriminator can accurately distinguish the two, the second point cloud generator is updated. If it cannot accurately distinguish the two, the point cloud discriminator is updated.
[0097] The second point cloud generator and the point cloud discriminator have similar structures to the generator and the discriminator described in the above embodiment. Please refer to the description of the above embodiment and will not be repeated here.
[0098] Step S106: the second point cloud generated data is input into the second CT generator to obtain second CT generated data, and the second CT generated data and the CT measurement data are used to calculate the CT adversarial cost function and the CT cycle consistency cost function.
[0099] The second point cloud generated data is passed through a second CT generator to generate second CT generated data. Specifically, the second CT generator downsamples the second point cloud generated data 5 times using a 3×3×3 convolution with a step size of 2, and symmetrically upsamples the second point cloud generated data using a 3×3×3 transposed convolution, and finally uses a linear rectification function as an activation function to obtain the second CT generated data.
[0100] The second CT generated data and CT measurement data are used to calculate the CT adversarial cost function in the regularization term of the cost function and CT cycle consistency cost function .
[0101] Step S107: The point cloud adversarial cost function, the point cloud cycle consistency cost function, the CT adversarial cost function and the CT cycle consistency cost function constitute a total cost function.
[0102] The point cloud adversarial cost function, the point cloud cycle consistency cost function, the CT adversarial cost function and the CT cycle consistency cost function constitute the total cost function, and the formula is as follows:
[0103] ,
[0104] Where L represents the total cost function, represents the point cloud adversarial cost function, represents the CT adversarial cost function, represents the point cloud cycle consistency cost function, represents the CT cycle consistency cost function, and is a constant between 0 and 10.
[0105] It should be noted that in the specific implementation, the point cloud adversarial cost function And CT adversarial cost function , you can use the cross entropy loss function or the mean square loss function. Point cloud cycle consistency cost function and CT cycle consistency cost function The L1 loss function or the L2 loss function can be used. Moreover, according to actual training needs, the convolution kernel of the UNet in the first CT generator, the second CT generator, the first point cloud generator, and the second point cloud generator can also be 5x5x5, the number of downsampling can be 3-10 times, and the number of upsampling should also be the same number symmetrically.
[0106] Step S108: According to the total cost function, a solver is used to calculate until convergence to obtain the facial bone structure.
[0107] According to the results obtained in the above steps, use and as well as and , calculate the total cost function L, and update the entire network according to the gradient obtained by the solver and the set learning rate.
[0108] In specific implementation, the solver can be the Adam solver, which is a tool used in deep learning training. The learning rate is set to ϵ (a constant between 0.0001 and 0.01), which is independent of the deep learning network framework. It combines the two optimization algorithms, AdaGrad and RMSProp, and comprehensively considers the first moment estimation (First Moment Estimation, i.e. the mean of the gradient) and the second moment estimation (Second Moment Estimation, i.e. the uncentered variance of the gradient) to calculate the update step size of the statistical gradient descent algorithm at each iteration.
[0109] In this way, after obtaining the optical point cloud data, the embodiment of the present invention simultaneously obtains the CT measurement data obtained by actual measurement, and performs deep learning training based on the optical point cloud data and the CT measurement data, thereby obtaining the final facial bone structure.
[0110] It can be seen from the description of the above embodiments that an embodiment of the present invention provides a method for constructing a facial bone structure, by acquiring a first point cloud data set and a second point cloud data set, wherein the first point cloud data set includes facial surface skin information and subcutaneous tissue information, and the second point cloud data set includes facial surface skin information, subcutaneous tissue information and bone information; the second point cloud data set is subtracted from the first point cloud data set after median filtering and sharpness enhancement to obtain optical point cloud data; the optical point cloud data is input into a first CT generator to obtain first CT generated data; the first CT generated data and the CT measurement data are both input into a CT discriminator, if the CT discriminator successfully distinguishes the first CT generated data from the CT measurement data, the first CT generator is updated, otherwise, the CT discriminator is updated; wherein the CT discriminator is a 3-layer fully convolutional network structure, the first CT generated data is input into the first point cloud generator to obtain the first point cloud generated data ... The point cloud generation data and the optical point cloud data are used to calculate the point cloud adversarial cost function and the point cloud cycle consistency cost function; the CT measurement data are input into the second point cloud generator to obtain the second point cloud generation data; the second point cloud generation data and the optical point cloud data are both input into the point cloud discriminator, if the point cloud discriminator successfully distinguishes the second point cloud generation data from the optical point cloud data, the second point cloud generator is updated, otherwise, the point cloud discriminator is updated; wherein the point cloud discriminator is a 5-layer fully convolutional network structure; the second point cloud generation data is input into the second CT generator to obtain the second CT generation data, the second CT generation data and the CT measurement data are used to calculate the CT adversarial cost function and the CT cycle consistency cost function; the point cloud adversarial cost function, the point cloud cycle consistency cost function, the CT adversarial cost function and the CT cycle consistency cost function constitute a total cost function; according to the total cost function, a solver is used to calculate until convergence to obtain the facial bone structure. The present invention extracts deep information describing facial bone structure, and on this basis, combined with facial CT measurement data, uses a cycle-consistent adversarial generative network to perform modal transfer learning, realizes the generation of CT-like three-dimensional imaging data from three-dimensional optical point cloud data, and completes the contactless imaging and reconstruction of three-dimensional facial bone structure, thereby having high accuracy. Moreover, the embodiment of the present invention combines cycle consistency with the adversarial generative network for adversarial generative training, solves the problem that three-dimensional optical point cloud data and three-dimensional CT data cannot correspond at the voxel level, provides a basic tool for realizing contactless positioning of the temporal orbital rim bone structure, and also provides a convenient and reliable imaging tool for the diagnosis of orbital tumors, skeletal dysplasia, facial trauma, etc.
[0111] Through the description of the above method embodiments, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk.
[0112] An embodiment of the present invention provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions can execute the method for estimating the state of charge of a rechargeable battery in any of the above method embodiments.
[0113] Corresponding to the embodiment of the method for constructing facial bone structure provided by the present invention, the present invention also provides a device for constructing facial bone structure.
[0114] See also Figure 5 , is a schematic diagram of a facial bone structure construction device provided by an embodiment of the present invention. As shown in the figure, the device includes:
[0115] An acquisition module 11 is used to acquire a first point cloud data set and a second point cloud data set, wherein the first point cloud data set includes facial surface skin information and subcutaneous tissue information, and the second point cloud data set includes facial surface skin information, subcutaneous tissue information, and bone structure information;
[0116] A difference module 12 is used for performing a difference between the second point cloud data set and the first point cloud data set after the median filtering and the sharpness enhancement to obtain optical point cloud data;
[0117] The first updating module 13 is used for inputting the optical point cloud data into the first CT generator to obtain the first CT generated data; the first CT generated data and the CT measurement data are both input into the CT discriminator, if the CT discriminator successfully distinguishes the first CT generated data from the CT measurement data, the first CT generator is updated, otherwise, the CT discriminator is updated; wherein the CT discriminator is a 3-layer full convolutional network structure;
[0118] A first cost function calculation module 14, configured to input first CT generated data into a first point cloud generator to obtain first point cloud generated data, wherein the first point cloud generated data and optical point cloud data are used to calculate a point cloud adversarial cost function and a point cloud cycle consistency cost function;
[0119] The second updating module 15 is used for inputting the CT measurement data into the second point cloud generator to obtain the second point cloud generated data; the second point cloud generated data and the optical point cloud data are both input into the point cloud discriminator, if the point cloud discriminator successfully distinguishes the second point cloud generated data from the optical point cloud data, the second point cloud generator is updated, otherwise, the point cloud discriminator is updated; wherein the point cloud discriminator is a 5-layer full convolutional network structure;
[0120] A second cost function calculation module 16, configured to input second point cloud generated data into a second CT generator to obtain second CT generated data, wherein the second CT generated data and CT measurement data are used to calculate a CT adversarial cost function and a CT cycle consistency cost function;
[0121] A total cost function module 17, used for combining the point cloud adversarial cost function, the point cloud cycle consistency cost function, the CT adversarial cost function and the CT cycle consistency cost function into a total cost function;
[0122] The solving module 18 is used to calculate the facial bone structure using a solver according to the total cost function until convergence.
[0123] Optionally, the difference module 12 is further used for:
[0124] Perform median filtering on the first point cloud data set using a 3×3×3 or 5×5×5 window;
[0125] The data after the intermediate value filtering is convolved using a preset matrix, wherein the preset matrix is a three-dimensional matrix, specifically as follows: [
[0127] [[ 0, 0, 0], [ 0, -1, 0], [ 0, 0, 0]],
[0128] [[ 0, -1, 0], [-1, 8, -1], [ 0, -1, 0]],
[0129] [[ 0, 0, 0], [ 0, -1, 0], [ 0, 0, 0]] ]
[0131] The second point cloud data set is subtracted from the first point cloud data set after convolution to obtain optical point cloud data.
[0132] Optionally, the first updating module 13 is further used for:
[0133] The first CT generator downsamples the optical point cloud data five times using a 3×3×3 convolution with a step size of 2, and symmetrically upsamples the optical point cloud data using a 3×3×3 transposed convolution, and finally uses a linear rectification function as an activation function to obtain the first CT generated data;
[0134] The second updating module 15 is also used for:
[0135] The second CT generator downsamples the second point cloud generation data 5 times using a 3×3×3 convolution with a stride of 2, and symmetrically upsamples using a 3×3×3 transposed convolution, and finally uses a linear rectification function as the activation function to obtain the second CT generation data.
[0136] Optionally, the total cost function module combines the point cloud adversarial cost function, the point cloud cycle consistency cost function, the CT adversarial cost function and the CT cycle consistency cost function into a total cost function, and the formula is as follows:
[0137] ,
[0138] Where L represents the total cost function, represents the point cloud adversarial cost function, represents the CT adversarial cost function, represents the point cloud cycle consistency cost function, represents the CT cycle consistency cost function, and is a constant between 0 and 10.
[0139] Figure 6 is a schematic diagram of the hardware structure of an electronic device for executing the facial bone structure construction method provided by an embodiment of the present invention, such as Figure 6 As shown, the device includes:
[0140] One or more processors 610 and memory 620, Figure 6 A processor 610 is taken as an example.
[0141] The device for executing the facial bone structure construction method may further include: an input device 630 and an output device 640 .
[0142] The processor 610, the memory 620, the input device 630 and the output device 640 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.
[0143] The memory 620 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the method for constructing facial bone structure in the embodiment of the present invention (for example, the attached Figure 5 The processor 610 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 620, that is, the method for constructing the facial bone structure of the above method embodiment is implemented.
[0144] The memory 620 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created by the use of a processing device constructed according to the facial bone structure, etc. In addition, the memory 620 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 620 may optionally include a memory remotely arranged relative to the processor 610, and these remote memories may be connected to the processing device constructed by the facial bone structure via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0145] The input device 630 can receive input digital or character information and generate key signal input related to user settings and function control of the processing device for facial bone structure construction. The output device 640 can include display devices such as display screens.
[0146] The one or more modules are stored in the memory 620, and when executed by the one or more processors 610, the method for constructing the facial bone structure in any of the above method embodiments is executed.
[0147] The above product can execute the method provided by the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the method provided by the embodiment of the present invention.
[0148] The electronic device of the embodiment of the present invention exists in various forms, including but not limited to:
[0149] (1) Mobile communication devices: These devices are characterized by their mobile communication functions and their main purpose is to provide voice and data communications. These terminals include: smart phones (such as iPhone), multimedia phones, functional phones, and low-end phones.
[0150] (2) Ultra-mobile personal computer devices: These devices fall into the category of personal computers, have computing and processing capabilities, and generally also have mobile Internet access features. These terminals include: PDA, MID and UMPC devices, such as iPad.
[0151] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players (such as iPods), handheld game consoles, e-books, smart toys, and portable car navigation devices.
[0152] (4) Server: A device that provides computing services. The server consists of a processor, hard disk, memory, system bus, etc. The server is similar to a general computer architecture, but because it needs to provide highly reliable services, it has higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.
[0153] (5) Other electronic devices with data interaction functions.
[0154] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0155] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0156] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0157] The above description is only a specific embodiment of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but should conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for constructing facial bone structure, characterized in that: include: Acquire a first point cloud data set and a second point cloud data set, wherein the first point cloud data set includes facial surface skin information and subcutaneous tissue information, and the second point cloud data set includes facial surface skin information, subcutaneous tissue information, and bone structure information; The second point cloud data set is subtracted from the first point cloud data set after median filtering and sharpness enhancement to obtain optical point cloud data; The optical point cloud data is input into the first CT generator to obtain the first CT generated data; the first CT generated data and the CT measurement data are both input into the CT discriminator, if the CT discriminator successfully distinguishes the first CT generated data from the CT measurement data, the first CT generator is updated, otherwise, the CT discriminator is updated; wherein the CT discriminator is a 3-layer full convolutional network structure; The first CT generated data is input into the first point cloud generator to obtain the first point cloud generated data, wherein the first point cloud generated data and the optical point cloud data are used to calculate the point cloud adversarial cost function and the point cloud cycle consistency cost function; The CT measurement data is input into the second point cloud generator to obtain the second point cloud generation data; the second point cloud generation data and the optical point cloud data are both input into the point cloud discriminator, and if the point cloud discriminator successfully distinguishes the second point cloud generation data from the optical point cloud data, the second point cloud generator is updated, otherwise, the point cloud discriminator is updated; wherein the point cloud discriminator is a 5-layer full convolutional network structure; The second point cloud generated data is input into the second CT generator to obtain second CT generated data, wherein the second CT generated data and the CT measurement data are used to calculate the CT adversarial cost function and the CT cycle consistency cost function; The point cloud adversarial cost function, the point cloud cycle consistency cost function, the CT adversarial cost function and the CT cycle consistency cost function constitute a total cost function; According to the total cost function, a solver is used to calculate until convergence to obtain the facial bone structure.
2. The method for constructing facial bone structure according to claim 1, characterized in that: The second point cloud data set is subtracted from the first point cloud data set after median filtering and sharpness enhancement to obtain optical point cloud data, including: Perform median filtering on the first point cloud data set using a 3×3×3 or 5×5×5 window; The data after the intermediate value filtering is convolved using a preset matrix, wherein the preset matrix is as follows: [ [[ 0, 0, 0], [ 0, -1, 0], [ 0, 0, 0]], [[ 0, -1, 0], [-1, 8, -1], [ 0, -1, 0]], [[ 0, 0, 0], [ 0, -1, 0], [ 0, 0, 0]] ] The second point cloud data set is subtracted from the first point cloud data set after convolution to obtain optical point cloud data.
3. The method for constructing facial bone structure according to claim 1, characterized in that: The optical point cloud data is input into a first CT generator to obtain first CT generated data, including: The first CT generator downsamples the optical point cloud data five times using a 3×3×3 convolution with a step size of 2, and symmetrically upsamples the optical point cloud data using a 3×3×3 transposed convolution, and finally uses a linear rectification function as an activation function to obtain the first CT generated data; The second point cloud generation data is input into a second CT generator to obtain second CT generation data, including: The second CT generator downsamples the second point cloud generation data 5 times using a 3×3×3 convolution with a stride of 2, and symmetrically upsamples using a 3×3×3 transposed convolution, and finally uses a linear rectification function as the activation function to obtain the second CT generation data.
4. The method for constructing facial bone structure according to claim 1, characterized in that: The point cloud adversarial cost function, the point cloud cycle consistency cost function, the CT adversarial cost function and the CT cycle consistency cost function constitute the total cost function, and the formula is as follows: ， Where L represents the total cost function, represents the point cloud adversarial cost function, represents the CT adversarial cost function, represents the point cloud cycle consistency cost function, represents the CT cycle consistency cost function, and is a constant between 0 and 10.
5. A facial bone structure construction device, characterized in that: include: An acquisition module, configured to acquire a first point cloud data set and a second point cloud data set, wherein the first point cloud data set includes facial surface skin information and subcutaneous tissue information, and the second point cloud data set includes facial surface skin information, subcutaneous tissue information, and bone structure information; A difference module, used for performing difference between the second point cloud data set and the first point cloud data set after median filtering and sharpness enhancement to obtain optical point cloud data; A first updating module is used for inputting optical point cloud data into a first CT generator to obtain first CT generated data; both the first CT generated data and the CT measurement data are input into a CT discriminator, and if the CT discriminator successfully distinguishes the first CT generated data from the CT measurement data, the first CT generator is updated, otherwise, the CT discriminator is updated; wherein the CT discriminator is a 3-layer fully convolutional network structure; A first cost function calculation module, used for inputting first CT generated data into a first point cloud generator to obtain first point cloud generated data, wherein the first point cloud generated data and the optical point cloud data are used to calculate a point cloud adversarial cost function and a point cloud cycle consistency cost function; The second updating module is used for inputting CT measurement data into the second point cloud generator to obtain second point cloud generation data; the second point cloud generation data and the optical point cloud data are both input into the point cloud discriminator, and if the point cloud discriminator successfully distinguishes the second point cloud generation data from the optical point cloud data, the second point cloud generator is updated, otherwise, the point cloud discriminator is updated; wherein the point cloud discriminator is a 5-layer full convolutional network structure; A second cost function calculation module, used for inputting second point cloud generated data into a second CT generator to obtain second CT generated data, wherein the second CT generated data and CT measurement data are used to calculate a CT adversarial cost function and a CT cycle consistency cost function; A total cost function module, used for combining the point cloud adversarial cost function, the point cloud cycle consistency cost function, the CT adversarial cost function and the CT cycle consistency cost function into a total cost function; The solution module is used to calculate the facial bone structure using a solver until convergence according to the total cost function.
6. The facial bone structure constructing device according to claim 5, characterized in that: The difference module is also used for: Perform median filtering on the first point cloud data set using a 3×3×3 or 5×5×5 window; The data after the intermediate value filtering is convolved using a preset matrix, wherein the preset matrix is as follows: [ [[ 0, 0, 0], [ 0, -1, 0], [ 0, 0, 0]], [[ 0, -1, 0], [-1, 8, -1], [ 0, -1, 0]], [[ 0, 0, 0], [ 0, -1, 0], [ 0, 0, 0]] ] The second point cloud data set is subtracted from the first point cloud data set after convolution to obtain optical point cloud data.
7. The facial bone structure constructing device according to claim 5, characterized in that: The first update module is also used for: The first CT generator downsamples the optical point cloud data five times using a 3×3×3 convolution with a step size of 2, and symmetrically upsamples the optical point cloud data using a 3×3×3 transposed convolution, and finally uses a linear rectification function as an activation function to obtain the first CT generated data; The second update module is also used for: The second CT generator downsamples the second point cloud generation data 5 times using a 3×3×3 convolution with a stride of 2, and symmetrically upsamples using a 3×3×3 transposed convolution, and finally uses a linear rectification function as the activation function to obtain the second CT generation data.
8. The facial bone structure constructing device according to claim 5, characterized in that: The total cost function module combines the point cloud adversarial cost function, the point cloud cycle consistency cost function, the CT adversarial cost function and the CT cycle consistency cost function into a total cost function, and the formula is as follows: ， Where L represents the total cost function, represents the point cloud adversarial cost function, represents the CT adversarial cost function, represents the point cloud cycle consistency cost function, represents the CT cycle consistency cost function, and is a constant between 0 and 10.
9. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the steps of the method for constructing a facial bone structure as described in any one of claims 1 to 4.
10. An electronic device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the method for constructing a facial bone structure according to any one of claims 1 to 4 are implemented.