Facial deformation prediction method and device, terminal and storage medium

Abstract

The invention provides a face deformation prediction method and device, a terminal and a storage medium. The method for predicting facial deformation comprises the following steps: scanning a face through cone beam computed tomography (CBCT) to obtain an image of a maxillofacial tissue structure; based on the image, acquiring first position information of the bone tissue and the motion tracking plate as a whole; second position information and third position information of the motion tracking plate and the artificial texture moving along with the mandibular motion are obtained through a three-dimensional visual system; obtaining a conversion coefficient between the first position information and the second position information; acquiring fourth position information of surface nodes of the bone tissue and the motion tracking plate as a whole; training the neural network model to obtain a trained neural network model; and based on the movement design of the bone block position, obtaining the predicted movement information of the facial artificial texture feature points. The method can be used for predicting the facial deformation of the operation, and is good in real-time performance and high in precision.

Description

Technical Field

[0001] The present disclosure relates to the field of information technology, and in particular to a method and device, a terminal and a storage medium for predicting facial deformation. Background Art

[0002] Skeletal maxillofacial deformity is a disease caused by abnormal growth and development of the jaw, which leads to abnormal jaw volume and shape, thus affecting the occlusal function and aesthetic appearance. Orthognathic surgery can treat occlusal disorders, improve facial contours, and correct deformities by changing the position of the upper and lower jaws. In recent years, more and more patients with skeletal maxillofacial deformities are willing and even eager to undergo orthognathic surgery, and the treatment demands have also shifted from "normal occlusion" to "gaining beauty"; good surgical appearance results are based on good surgical design plans, and are also an important reference for patients to choose doctors. Preoperative facial prediction can meet the demands of both doctors and patients, leverage the increase in supply and demand, and is of great significance for promoting the development of disciplines, promoting a virtuous cycle of "appearance" related industries, and balancing medical resources.

[0003] Ideal facial shape prediction needs to meet the following requirements: First, high real-time performance. When the doctor moves the bones, the program calculates the facial shape prediction effect within seconds. Otherwise, the doctor-patient communication will be interrupted by the prediction process, which will affect the doctor's and patient's trust in the entire technical system. Second, high precision. Accurate prediction is the basis for doctors to formulate surgical plans. Large prediction deviations not only mislead doctors to formulate plans, but also cause inappropriate expectations of patients and even cause medical disputes. Third, details. The simulation of simple profile changes can no longer meet the requirements of modern orthognathic patients. It needs to be refined to the prediction of aesthetic structures such as nasolabial angle, chin-labial groove, and face-neck angle. The facial shape prediction functions of current commercial software cannot meet the above requirements, cannot be used as the basis for formulating surgical plans, and cannot be used for doctor-patient communication.

[0004] Facial shape prediction is a typical cross-disciplinary research field involving orthognathic surgery, image processing, and biomechanics. Physical modeling methods that simulate elastic deformation have always been a hot topic in the field of facial shape prediction research. The mainstream methods include mass-spring model, mass-tensor model, and finite element model. The mass-spring model has a simple structure, fast operation speed, and good real-time performance, but poor simulation and low accuracy. The finite element model has a complex structure and good simulation, but due to the large amount of data, high hardware requirements, and long operation time, it has not been able to achieve clinical application. The mass-tensor model is improved on the basis of the mass-spring model, and multivariate parameters are introduced to express facial biomechanical characteristics. The real-time performance and profile prediction accuracy are clinically acceptable, but the prediction accuracy of the midline structure is poor. In addition to the "modeling" method, the factors that affect the accuracy of deformation prediction also include "elastic characteristics", such as elastic modulus and Poisson's ratio. In the past, there were few studies on "elastic characteristics", and the elastic modulus of known objects was used to approximate them. Moreover, the few elastic characteristic parameters of human tissues are mostly obtained by testing in vitro specimens. It should be said that the current research solutions for elastic parameters are "non-personalized", "non-specific", and "non-physiological". However, the elastic characteristics of facial soft tissue vary greatly from person to person, and the elastic characteristics of different anatomical divisions and layers are also different. The elastic properties of ex vivo tissues are even more different from those in physiological states. There are currently no reports on measuring personalized elastic parameters of facial soft tissues in physiological states, so the current methods are not very accurate in predicting facial deformation. Summary of the invention

[0005] To solve the existing problems, the present disclosure provides a method and device, a terminal and a storage medium for facial deformation prediction.

[0006] The present disclosure adopts the following technical solutions.

[0007] An embodiment of the present disclosure provides a method for predicting facial deformation, the method comprising: fixing a motion tracking plate on the mandibular teeth, scanning the face through cone beam computed tomography (CBCT), and acquiring an image of the maxillofacial tissue structure; based on the image, extracting soft tissue, bone tissue, and the motion tracking plate, and acquiring first position information of the bone tissue and the motion tracking plate as a whole; making an artificial texture on the face, and acquiring second position information and third position information of the motion tracking plate and the artificial texture as the mandibular moves, respectively, through a three-dimensional vision system; based on the first position information and the second position information information, and obtain a conversion coefficient between the first position information and the second position information; based on the first position information and the second position information, obtain the fourth position information of the surface nodes of the bone tissue and the motion tracking board as a whole; construct a neural network model, take the fourth position information as input, take the third position information as output, train the neural network model, and obtain a trained neural network model; based on the movement design of the bone block position, obtain the position information of the surface nodes of the bone block after the movement, as the input of the trained neural network model, and obtain the predicted motion information of the facial artificial texture feature points, thereby realizing the prediction of facial deformation.

[0008] Another embodiment of the present disclosure provides a device for facial deformation prediction, the processing device comprising: an image acquisition module, configured to fix a motion tracking plate on the mandibular teeth, scan the face through cone beam computed tomography (CBCT), and acquire an image of the maxillofacial tissue structure; a position information acquisition module, configured to extract soft tissue, bone tissue and the motion tracking plate based on the image, and acquire first position information of the bone tissue and the motion tracking plate as a whole; create artificial texture on the face, and acquire second position information and third position information of the motion tracking plate and the artificial texture as the mandibular moves respectively through a three-dimensional visual system; a conversion coefficient acquisition module, configured to acquire, based on the first position information and the second position information, The conversion coefficient between the first position information and the second position information; the position information acquisition module is also configured to obtain the fourth position information of the surface nodes of the bone tissue and the motion tracking board as a whole based on the first position information and the second position information; the neural network construction and training module is configured to construct a neural network model, take the fourth position information as input, take the third position information as output, train the neural network model, and obtain a trained neural network model; the prediction module is configured to obtain the position information of the surface nodes of the moved bone blocks based on the movement design of the bone block position, as the input of the trained neural network model, and obtain the predicted motion information of the facial artificial texture feature points, thereby realizing the prediction of facial deformation.

[0009] In some embodiments, the present disclosure provides a terminal, comprising: at least one memory and at least one processor; wherein the memory is used to store program code, and the processor is used to call the program code stored in the memory to execute the above-mentioned method for facial deformation prediction.

[0010] In some embodiments, the present disclosure provides a storage medium for storing program code, wherein the program code is used to execute the above-mentioned method for facial deformation prediction.

[0011] The method disclosed in the present invention can be applied to individualized and specific facial deformation prediction for surgery for individuals in physiological states, with good real-time performance and high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0013] Figure 1 A schematic diagram showing components of a system according to some embodiments.

[0014] Figure 2 A schematic diagram of a motion tracking pad is shown in accordance with some embodiments.

[0015] Figure 3 A schematic diagram of an artificial texture according to some embodiments is shown.

[0016] Figure 4 The figure is a flowchart of a method for facial deformation prediction according to an embodiment of the present disclosure.

[0017] Figure 5 is a schematic flowchart of a method for facial deformation prediction according to an embodiment of the present disclosure.

[0018] Figure 6 A schematic diagram showing bone tissue and soft tissue reconstruction based on CT images according to some embodiments.

[0019] Figure 7 It is a partial module of the apparatus for facial deformation prediction according to another embodiment of the present disclosure.

[0020] Figure 8 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0022] It should be understood that the various steps described in the method embodiments of the present disclosure can be performed in sequence and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0023] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0024] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0025] It should be noted that the modification of “one” mentioned in the present disclosure is illustrative rather than restrictive, and those skilled in the art should understand that it should be understood as “one or more” unless otherwise clearly indicated in the context.

[0026] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0027] The present invention establishes an artificial intelligence model that conforms to the deformation characteristics of facial soft tissues based on the personalized elastic characteristics of patients under actual measurements of physiological conditions, and is expected to achieve the goal of real-time, accurate, and detailed facial shape prediction. The human mandible is non-rigidly connected to the skull, and can achieve forward, backward, and lateral movements within a certain range, causing changes in facial contours, which is very similar to the situation in which bone movement in orthognathic surgery produces contour changes. Mandibular movement creates natural conditions for tracking and recording facial soft tissue deformation under physiological conditions, and then analyzing its elastic characteristics. If the mandibular movement-tissue deformation process can be continuously tracked, the relative position relationship of soft and hard tissues at each moment can be recorded, and the actual postoperative soft tissue final morphology can be incorporated, the elastic characteristic parameters of personalized facial shape prediction can be performed through artificial intelligence.

[0028] like Figure 1 As shown, the system of the present disclosure is mainly composed of a three-dimensional vision system, a motion tracking board and artificial texture. The three-dimensional vision system is composed of multiple cameras. By calibrating the relative positions between the cameras, when any two cameras simultaneously shoot the same target, the three-dimensional information of the target can be calculated. Therefore, it can be used to obtain the movement of the landmarks or feature points of the motion tracking board, and then obtain the movement of the entire mandible. The motion tracking board is as shown in FIG. Figure 2 As shown in FIG. 1 , the surface of the tracking plate is provided with four asymmetric cylinders as feature points for tracking and positioning by a three-dimensional vision system. When in use, the tracking plate is fixed to the surface of the lower teeth. Figure 3 As shown, the pattern is in the shape of a grid, and there are circular marks at the intersections of the grids, which serve as a reference for the visual system to extract feature points. There are numbers and letters in each grid to facilitate human eye observation. The artificial texture is printed on the surface of the water transfer paper film using a laser printer. After applying it to the face, it is soaked with water to transfer the pattern to the surface of the face.

[0029] Figure 4 and Figure 5A flowchart of a method for facial deformation prediction according to an embodiment of the present disclosure is provided. The method for facial deformation prediction according to the present disclosure may include step S101, fixing a motion tracking plate on the mandibular teeth, scanning the face by cone beam computed tomography (CBCT), and acquiring an image of the maxillofacial tissue structure. In some embodiments, when taking CBCT, the motion tracking plate is bonded to the mandibular dentition by resin so that the two become a whole, that is, when the mandibular dentition moves, the motion tracking plate generates synchronous movement. The patient's face is scanned by CBCT to acquire a CT image sequence of the maxillofacial tissue structure.

[0030] In some embodiments, the method of the present disclosure may further include step S102, extracting soft tissue, bone tissue, and the motion tracking plate based on the CT image, and obtaining first position information of the bone tissue and the motion tracking plate as a whole. Figure 6 As shown, with the help of three-dimensional image reconstruction technology commonly used in the field, bone tissue, soft tissue and motion tracking plate are respectively extracted through, for example, threshold segmentation. In some embodiments, the motion tracking plate used to track the rigid body motion of bone tissue is regarded as a whole with the bone tissue to export its stereolithography (STL) format file, which stores the surface triangular patch information of the motion tracking plate and the lower dentition as a whole, that is, the surface is discretized into triangular patches, and the file stores the vertex spatial position of each triangle and the normal vector of the triangle. Therefore, in some embodiments, the first position information is stored in a stereolithography (STL) format file, the first position information includes the surface triangular patch information, and the file stores the vertex spatial position of each triangle and the normal vector of the triangle. In some embodiments, steps S101 and S102 correspond to Figure 5 Soft tissue and bone segmentation in

[0031] In some embodiments, the method of the present disclosure may further include step S103, creating an artificial texture on the face, and obtaining second position information and third position information of the motion tracking board and the artificial texture respectively moving with the movement of the mandible through a three-dimensional visual system. In some embodiments, the tracking of facial deformation and the tracking of bone tissue movement adopt a three-dimensional visual tracking method, such as Figure 1 As shown, water transfer printing was used to Figure 3The pattern of the artificial texture shown is transferred to the face for marking, producing an artificial texture that is clearly distinguishable from the natural skin texture. In some embodiments, the motion tracking board is fixed to the mandible and fixed to the mandibular teeth by glue or other means. Under the doctor's instructions, the patient performs mandibular movements such as biting and protrusion, which in turn drives the movement of the motion tracking board and the movement of the artificial texture feature points. Multiple cameras of the three-dimensional vision system simultaneously shoot the motion tracking board and the area on the face surface where the artificial texture is printed. Based on the circular features of the motion tracking board and the circular features of the artificial texture, N groups of artificial texture feature point sets Q1, Q2, Q3, ..., Q N , where each point set contains the spatial positions of n artificial texture feature points. Taking Q1 as an example, Q1 = {q1, q2, q3, …, q n}, where q1 represents the spatial position of one of the feature points of the artificial texture feature, and the corresponding point set F1, F2, F3, ..., F N , where each point set contains the positions of the centers of the four cylindrical surfaces. Take F1 as an example, that is, F1 = {f1, f2, f3, f4}, where f1 represents the spatial position of one of the centers of the cylindrical surface of the motion tracking board. In some embodiments, as described above, the three-dimensional vision system includes a plurality of cameras arranged at different positions; the surface of the motion tracking board is provided with a plurality of (for example, 4) asymmetric cylinders as feature points for tracking and positioning by the three-dimensional vision system; the pattern of the artificial texture is in the form of a grid, and circular marks are provided at the intersection of the grids as a reference for the three-dimensional vision system to extract feature points, wherein the artificial texture is printed onto the surface of a water transfer paper film using a laser printer, and after being applied to the face, it is soaked with water to transfer the pattern to the surface of the face. In some embodiments, step S103 corresponds to Figure 5 Facial deformation and bone motion tracking in .

[0032] In some embodiments, the method of the present disclosure may further include step S104, obtaining a conversion coefficient between the first position information and the second position information based on the first position information and the second position information. In some embodiments, the second position information includes position information of feature points of an asymmetric cylinder on the surface of the motion tracking board. In some embodiments, due to the feature point set Q1, Q2, Q3, ..., Q N , the positions of the motion tracking board F1, F2, F3, ..., F N The three-dimensional data of anatomical structures come from different image acquisition systems and have different coordinate systems. Therefore, it is first necessary to convert the bone tissue surface node coordinates in the CBCT coordinate system into the three-dimensional visual coordinate system.

[0033] In some embodiments, a point set F consisting of the coordinates of the four center points of the cylindrical surface in the STL format file of the motion tracking board and the bone tissue as a whole is extracted, and the coordinates of the four center points of the cylindrical surface are stored in F, that is, F = {f1, f2, f3, f4}. The coordinate conversion uses the four center points of the cylindrical surface for matching, and the following linear equation is established:

[0034] F1=H1*F

[0035] H1 represents a linear transformation, and {f1,f2,f3,f4} represents a matrix composed of the spatial coordinates of the four center points. The least squares method can be used to solve the problem. N Solve H1, H2, H3, ..., H N .

[0036] In some embodiments, the method of the present disclosure may further include step S105, based on the first position information and the second position information, obtaining fourth position information of the surface nodes of the bone tissue and the motion tracking plate as a whole. In some embodiments, the third position information includes the position information of the feature points of the circular mark of the artificial texture. In some embodiments, the position composition point set G of all surface nodes in the STL format file of the motion tracking plate and the bone tissue as a whole is extracted, and the surface node point sets G1, G2, G3, ..., G during the mandibular movement are calculated using the following formula N .

[0037] G1=H1*G;G2=H2*G;…G N =H N *G.

[0038] In some embodiments, steps S104 and S105 correspond to Figure 5 Data conversion and organization in .

[0039] In some embodiments, the method of the present disclosure may further include step S106, constructing a neural network model, taking the fourth position information as input, taking the third position information as output, training the neural network model, and obtaining a trained neural network model. In some embodiments, the neural network model is constructed, and the input is the surface node point set G1, G2, G3, ..., G during the mandibular movement. N , the corresponding output is the feature point set Q1, Q2, Q3, ..., Q N , based on the reverse error propagation principle or the minimum loss function principle, the neural network model is trained to obtain a trained neural network model. In some embodiments, step S106 corresponds to Figure 5 The training neural network model in .

[0040] In some embodiments, the method of the present disclosure may further include step S107, based on the movement design of the bone block position, obtaining the position information of the surface nodes of the moved bone block as the input of the trained neural network model, and obtaining the motion information of the predicted facial artificial texture feature points, thereby realizing the prediction of facial deformation. In some embodiments, according to the movement of the bone block designed by the surgery, the position of the bone block is moved, and the surface node point set G of the moved bone block is obtained. N+1 , as the input of the neural network model, the predicted movement of the facial artificial texture feature points is obtained, and the facial soft tissue surface is reconstructed according to the predicted movement of the feature points, thereby realizing the prediction of facial deformation. In some embodiments, step S107 corresponds to Figure 5 Predicting facial deformation from bone motion input.

[0041] By adopting the method disclosed in the present invention, facial deformation prediction for surgery can be performed on an individual basis and specifically for individuals in physiological states, with good real-time performance and high accuracy.

[0042] The embodiment of the present disclosure also provides a device 400 for facial deformation prediction. Figure 7The apparatus 400 for predicting facial deformation according to some embodiments is shown. The apparatus 400 for predicting facial deformation includes an image acquisition module 401, a position information acquisition module 402, a conversion coefficient acquisition module 403, a neural network construction and training module 404, and a prediction module 405. In some embodiments, the image acquisition module 401 is configured to fix the motion tracking plate on the mandibular teeth, scan the face through cone beam computed tomography (CBCT), and obtain an image of the maxillofacial tissue structure. In some embodiments, the position information acquisition module 402 is configured to extract soft tissue, bone tissue, and motion tracking plate based on the image, and obtain the first position information of the bone tissue and the motion tracking plate as a whole; make artificial texture on the face, and obtain the second position information and third position information of the motion tracking plate and the artificial texture moving with the mandibular movement respectively through a three-dimensional visual system. In some embodiments, the conversion coefficient acquisition module 403 is configured to obtain the conversion coefficient between the first position information and the second position information based on the first position information and the second position information. In some embodiments, the position information acquisition module is also configured to obtain the fourth position information of the surface nodes of the bone tissue and the motion tracking plate as a whole based on the first position information and the second position information. In some embodiments, the neural network construction and training module 404 is configured to construct a neural network model, take the fourth position information as input, take the third position information as output, train the neural network model, and obtain a trained neural network model. In some embodiments, the prediction module 405 is configured to obtain the position information of the surface nodes of the moved bone blocks based on the movement design of the bone block position as the input of the trained neural network model, and obtain the predicted motion information of the facial artificial texture feature points, thereby realizing the prediction of facial deformation.

[0043] It should be understood that the contents described about the method for facial deformation prediction are also applicable to the apparatus 400 for facial deformation prediction herein, and for the sake of simplicity, a detailed description is not given here.

[0044] In some embodiments, the three-dimensional vision system includes a plurality of cameras arranged at different positions. In some embodiments, a plurality of asymmetric cylinders are arranged on the surface of the motion tracking board as feature points for tracking and positioning of the three-dimensional vision system. In some embodiments, the pattern of the artificial texture is in the form of a grid, and circular marks are arranged at the intersection of the grids as a reference for the three-dimensional vision system to extract feature points, wherein the artificial texture is printed on the surface of a water transfer paper film by a laser printer, and after being applied to the face, it is soaked with water to transfer the pattern to the surface of the face. In some embodiments, the first position information is stored in a file in a stereolithography (STL) format, and the first position information includes surface triangle patch information, and the file stores the spatial position of the vertices of each triangle and the normal vector of the triangle. In some embodiments, the second position information includes the position information of the feature points of the asymmetric cylinders on the surface of the motion tracking board. In some embodiments, the third position information includes the position information of the feature points of the circular marks of the artificial texture.

[0045] In addition, the present disclosure also provides a terminal, including: at least one memory and at least one processor; wherein the memory is used to store program code, and the processor is used to call the program code stored in the memory to execute the above-mentioned method for facial deformation prediction.

[0046] In addition, the present disclosure also provides a computer storage medium, which stores program code, and the program code is used to execute the above-mentioned method for facial deformation prediction.

[0047] The above describes the method and device for facial deformation prediction of the present disclosure based on the embodiments and application examples. In addition, the present disclosure also provides a terminal and a storage medium, which are described below.

[0048] Reference below Figure 8 , which shows a schematic diagram of the structure of an electronic device (such as a terminal device or a server) 500 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0049] like Figure 8As shown, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0050] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 8 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0051] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0052] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0053] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0054] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0055] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method of the present disclosure.

[0056] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0057] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0058] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0059] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0060] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0061] According to one or more embodiments of the present disclosure, a method for predicting facial deformation is provided, the method comprising: fixing a motion tracking plate on the mandibular teeth, scanning the face by cone beam computed tomography (CBCT), and acquiring an image of the maxillofacial tissue structure; based on the image, extracting soft tissue, bone tissue, and the motion tracking plate, and acquiring first position information of the bone tissue and the motion tracking plate as a whole; making an artificial texture on the face, and acquiring second position information and third position information of the motion tracking plate and the artificial texture as the mandibular moves, respectively, through a three-dimensional vision system; based on the first position information and the third position information The invention relates to a method for detecting a facial deformation of a human face, wherein the first position information of the human face and the second position information are used to detect the facial deformation of the human face. The method comprises the following steps: first, a first position information of the human face and the second position information are used to detect the facial deformation of the human face. Second, the conversion coefficient between the first position information and the second position information is obtained; based on the first position information and the second position information, the fourth position information of the surface nodes of the bone tissue and the motion tracking board as a whole is obtained; a neural network model is constructed, and the fourth position information is used as input and the third position information is used as output to train the neural network model to obtain a trained neural network model; based on the movement design of the bone block position, the position information of the surface nodes of the bone block after the movement is obtained as input of the trained neural network model to obtain the motion information of the predicted facial artificial texture feature points, thereby realizing the prediction of facial deformation.

[0062] According to one or more embodiments of the present disclosure, the three-dimensional vision system includes a plurality of cameras arranged at different positions.

[0063] According to one or more embodiments of the present disclosure, a plurality of asymmetric cylinders are provided on the surface of the motion tracking board as feature points for tracking and positioning of the three-dimensional vision system.

[0064] According to one or more embodiments of the present disclosure, the pattern of the artificial texture is in a grid shape, and there are circular marks at the intersections of the grids, which serve as a reference for the three-dimensional vision system to extract feature points, wherein the artificial texture is printed onto the surface of a water transfer paper film using a laser printer, and after being applied to a human face, it is soaked with water to transfer the pattern to the surface of the human face.

[0065] According to one or more embodiments of the present disclosure, the first position information is stored in a file in a stereolithography (STL) format, the first position information includes surface triangle patch information, and the file stores the vertex spatial position of each triangle and the normal vector of the triangle.

[0066] According to one or more embodiments of the present disclosure, the second position information includes position information of feature points of an asymmetric cylinder on the surface of the motion tracking board.

[0067] According to one or more embodiments of the present disclosure, the third position information includes position information of feature points of the circular mark of the artificial texture.

[0068] According to one or more embodiments of the present disclosure, there is provided a device for predicting facial deformation, the device comprising: an image acquisition module, configured to fix a motion tracking plate on the mandibular teeth, scan the face through cone beam computed tomography (CBCT), and obtain an image of the maxillofacial tissue structure; a position information acquisition module, configured to extract soft tissue, bone tissue and the motion tracking plate based on the image, and obtain first position information of the bone tissue and the motion tracking plate as a whole; make an artificial texture on the face, and obtain second position information and third position information of the motion tracking plate and the artificial texture as the mandibular moves respectively through a three-dimensional visual system; a conversion coefficient acquisition module, configured to obtain second position information and third position information of the motion tracking plate and the artificial texture based on the first position information and the second position information; The position information acquisition module is further configured to obtain the fourth position information of the surface nodes of the bone tissue and the motion tracking board as a whole based on the first position information and the second position information; the neural network construction and training module is configured to construct a neural network model, take the fourth position information as input, take the third position information as output, train the neural network model, and obtain a trained neural network model; the prediction module is configured to obtain the position information of the surface nodes of the bone blocks after the movement based on the movement design of the bone block position, as the input of the trained neural network model, and obtain the predicted motion information of the facial artificial texture feature points, thereby realizing the prediction of facial deformation.

[0069] According to one or more embodiments of the present disclosure, a terminal is provided, comprising: at least one memory and at least one processor; wherein the at least one memory is used to store program code, and the at least one processor is used to call the program code stored in the at least one memory to execute any one of the methods described above.

[0070] According to one or more embodiments of the present disclosure, a storage medium is provided, wherein the storage medium is used to store program code, and the program code is used to execute the above method.

[0071] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0072] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0073] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A method for predicting facial deformation, characterized in that: The method for facial deformation prediction comprises: The motion tracking plate was fixed on the mandibular teeth, and the face was scanned by cone beam computed tomography (CBCT) to obtain images of maxillofacial tissue structures; Extracting soft tissue, bone tissue and the motion tracking board based on the image, and acquiring first position information of the bone tissue and the motion tracking board as a whole; Creating an artificial texture on the face, and acquiring, through a three-dimensional visual system, second position information and third position information of the motion tracking board and the artificial texture respectively moving along with the movement of the mandible; Based on the first position information and the second position information, obtaining a conversion coefficient between the first position information and the second position information; Based on the first position information and the second position information, acquiring fourth position information of the surface nodes of the bone tissue and the motion tracking board as a whole; Constructing a neural network model, taking the fourth position information as input and the third position information as output, training the neural network model to obtain a trained neural network model; Based on the movement design of the bone block position, the position information of the surface nodes of the moved bone block is obtained as the input of the trained neural network model to obtain the motion information of the predicted facial artificial texture feature points, thereby realizing the prediction of facial deformation.

2. The method for facial deformation prediction according to claim 1, characterized in that: The three-dimensional vision system includes a plurality of cameras arranged at different positions.

3. The method for facial deformation prediction according to claim 1, characterized in that: The surface of the motion tracking board is provided with a plurality of asymmetric cylinders as feature points for tracking and positioning of the three-dimensional vision system.

4. The method for facial deformation prediction according to claim 1, characterized in that: The pattern of the artificial texture is in a grid shape, and there are circular marks at the intersections of the grids, which serve as a reference for the three-dimensional visual system to extract feature points. The artificial texture is printed on the surface of a water transfer paper film using a laser printer, and after being applied to the face, it is soaked with water to transfer the pattern to the surface of the face.

5. The method for facial deformation prediction according to claim 1, characterized in that: The first position information is stored in a file in a stereolithography (STL) format, the first position information includes surface triangle patch information, and the file stores the spatial position of the vertices of each triangle and the normal vector of the triangle.

6. The method for facial deformation prediction according to claim 3, characterized in that: The second position information includes position information of feature points of an asymmetric cylinder on the surface of the motion tracking board.

7. The method for facial deformation prediction according to claim 4, characterized in that: The third position information includes position information of feature points of the circular mark of the artificial texture.

8. A device for predicting facial deformation, characterized in that: The device for facial deformation prediction comprises: an image acquisition module, configured to fix the motion tracking plate on the mandibular teeth, scan the face through cone beam computed tomography (CBCT), and acquire an image of the maxillofacial tissue structure; A position information acquisition module is configured to extract soft tissue, bone tissue and the motion tracking plate based on the image, and obtain first position information of the bone tissue and the motion tracking plate as a whole; create an artificial texture on the face, and obtain second position information and third position information of the motion tracking plate and the artificial texture respectively moving with the movement of the mandible through a three-dimensional visual system; a conversion coefficient acquisition module, configured to acquire a conversion coefficient between the first position information and the second position information based on the first position information and the second position information; The position information acquisition module is further configured to acquire fourth position information of the surface nodes of the bone tissue and the motion tracking board as a whole based on the first position information and the second position information; A neural network construction and training module, configured to construct a neural network model, take the fourth position information as input, take the third position information as output, train the neural network model, and obtain a trained neural network model; The prediction module is configured to obtain the position information of the surface nodes of the moved bones based on the movement design of the bone block, and use it as the input of the trained neural network model to obtain the motion information of the predicted facial artificial texture feature points, thereby realizing the prediction of facial deformation.

9. A terminal, comprising: at least one memory and at least one processor; The at least one memory is used to store program codes, and the at least one processor is used to call the program codes stored in the at least one memory to execute the method for facial deformation prediction according to any one of claims 1 to 7.

10. A storage medium, the storage medium being used to store program code, the program code being used to execute the method for facial deformation prediction according to any one of claims 1 to 7.

Images