Temporomandibular joint deflection determination method and system based on deep learning

Abstract

The invention provides a temporal-mandibular joint deflection determination method and system based on deep learning, and the method comprises the steps: obtaining a multi-view two-dimensional image and surface scanning data of a user; performing three-dimensional reconstruction on the multi-view two-dimensional image of the user by adopting a preset multi-view photo reconstruction algorithm, and determining a three-dimensional face model of the user; adopting a preset point cloud feature extraction network to determine a three-dimensional face model of the user and key feature points of the face scanning data; inputting the three-dimensional face model of the user and the key feature points of the face scanning data into a preset multi-modal image registration algorithm, and determining multi-modal data of the same three-dimensional space; inputting the multi-modal data into a preset deep neural network for diagnosis, and determining a diagnosis result of the user; and according to the multi-modal data, determining a facial index of temporal-mandibular joint deflection corresponding to the user. According to the method and the device, early screening of temporomandibular joint deflection is realized by adopting the two-dimensional image, the data acquisition cost is reduced, and the efficiency and the precision of medical diagnosis are improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of three-dimensional medical image processing, and in particular, to a method and system for determining temporomandibular joint deviation based on deep learning. Background Art

[0002] Deep learning technology originated from artificial neural networks. This technology analyzes practical problems by simulating the information transmission and decision-making processes of human brain neurons, and is a new branch of artificial intelligence technology. This technology can analyze the underlying features of different aspects of things and then perform combined learning through certain rules to analyze high-dimensional feature information, and is widely used in the fields of computer vision, natural language processing, intelligent speech systems, pattern recognition, and intelligent medical diagnosis. Deep learning technology can continuously improve the ability to solve practical problems by establishing an appropriate number of neurons and a multi-layer structure, selecting the input and output that simulates the human brain to solve problems, and choosing appropriate learning objectives and optimization algorithms.

[0003] Three-dimensional face reconstruction technology means reconstructing the corresponding three-dimensional facial model through one or more two-dimensional images. The reconstructed three-dimensional facial model is represented in the form of coordinates, color textures, normal vectors, etc., and can be used in the fields of face recognition, animated movies, security and privacy, and medical processing. Three-dimensional data has richer information in the medical diagnosis scenario, so the three-dimensional face technology with higher accuracy has high application value and prospects in the intelligent medical diagnosis scenario.

[0004] In the field of intelligent medical diagnosis, it mainly processes medical images through deep learning technology for auxiliary diagnosis, realizes functions such as organ segmentation, specific lesion detection, and three-dimensional reconstruction and fusion of tissue calibration, provides diagnostic references for doctors to improve the efficiency and accuracy of diagnosis. At the same time, it can provide more convenient services for patients, improve the efficiency of doctor-patient communication, and enable both doctors and patients to more conveniently track the development of the disease, and is also convenient for doctors to formulate more appropriate subsequent treatment plans. Summary of the Invention

[0005] Aiming at the deficiencies in the prior art, the purpose of the present disclosure is to provide a method and system for determining temporomandibular joint deviation based on deep learning.

[0006] To achieve the above purpose, according to one aspect of the present disclosure, there is provided a method for determining temporomandibular joint deviation based on deep learning, including:

[0007] Obtain multi-view two-dimensional images and face scan data of a user;

[0008] Perform three-dimensional reconstruction on the multi-view two-dimensional images of the user by using a preset multi-view photo reconstruction algorithm to determine the three-dimensional facial model of the user;

[0009] Input the 3D facial model of the user and the face scan data into a preset point cloud feature extraction network respectively to determine the key feature points of the 3D facial model of the user and the key feature points of the face scan data;

[0010] Input the key feature points of the 3D facial model of the user and the key feature points of the face scan data into a preset multimodal image registration algorithm to determine multimodal data in the same 3D space;

[0011] Input the multimodal data into a preset deep neural network for diagnosis to determine the diagnosis result of the user;

[0012] Determine the facial index of temporomandibular joint deviation corresponding to the user according to the multimodal data.

[0013] Optionally, the multi-view 2D images of the user include a front view image, a left oblique view image, and a right oblique view image.

[0014] Optionally, the method of using a preset multi-view photo reconstruction algorithm to perform 3D reconstruction on the multi-view 2D images of the user to determine the 3D facial model of the user includes:

[0015] Use the preset multi-view photo reconstruction algorithm to perform 3D reconstruction on the front view image, the left oblique view image, and the right oblique view image on a preset standard face model to determine the 3D facial model of the user.

[0016] Optionally, the key feature points include the feature points of the left inner canthus, the right inner canthus, the center of the eyebrows, the root of the nose, the bottom of the columella nasi, the mental point, the left alar nose, the right alar nose, the left tragus, and the right tragus.

[0017] Optionally, the step of inputting the 3D facial model of the user and the face scan data into a preset point cloud feature extraction network respectively to determine the key feature points of the 3D facial model of the user and the key feature points of the face scan data includes:

[0018] Use the preset point cloud feature extraction network to perform point cloud decentralization processing and noise point removal processing on the 3D facial model of the user and the face scan data in sequence to determine the preprocessed 3D facial model of the user and the preprocessed face scan data;

[0019] Perform key feature point extraction processing on the preprocessed 3D facial model of the user and the preprocessed face scan data to determine the key feature points of the 3D facial model of the user and the key feature points of the face scan data.

[0020] Optionally, the method further includes:

[0021] Perform segmentation processing on the multi-modal data to determine the mandibular region data;

[0022] Use a preset downsampling algorithm to perform data augmentation processing on the mandibular region data to determine the data-augmented mandibular region data;

[0023] Input the data-augmented mandibular region data into a preset machine learning algorithm for verification to determine the verification result.

[0024] Optionally, the inputting the multi-modal data into a preset deep neural network for diagnosis to determine the diagnosis result of the user includes:

[0025] Input the data-augmented mandibular region data into the preset deep neural network to determine the diagnosis result of the user.

[0026] Optionally, the facial indicators of the temporomandibular joint deviation corresponding to the user include the mandibular deviation angle, the maxillary retrusion angle, and the mandibular protrusion angle.

[0027] Optionally, the determining the facial indicators of the temporomandibular joint deviation corresponding to the user according to the multi-modal data includes:

[0028] Use the key feature points of the facial data to correct the key feature points of the three-dimensional facial model of the user to determine the target key feature points;

[0029] Calculate the mandibular deviation angle, the maxillary retrusion angle, and the mandibular protrusion angle of the user according to the target key feature points.

[0030] According to the second aspect of the present disclosure, there is provided a system for determining temporomandibular joint deviation based on deep learning, including:

[0031] A data acquisition module for acquiring multi-view two-dimensional images and face scan data of a user;

[0032] A three-dimensional reconstruction module for performing three-dimensional reconstruction on the multi-view two-dimensional images of the user by using a preset multi-view photo reconstruction algorithm to determine the three-dimensional facial model of the user;

[0033] A key feature point extraction module for inputting the three-dimensional facial model of the user and the face scan data into a preset point cloud feature extraction network respectively to determine the key feature points of the three-dimensional facial model of the user and the key feature points of the face scan data;

[0034] A multi-modal registration module for inputting the key feature points of the three-dimensional facial model of the user and the key feature points of the face scan data into a preset multi-modal image registration algorithm to determine multi-modal data in the same three-dimensional space;

[0035] A qualitative diagnosis module for inputting the multi-modal data into a preset deep neural network for diagnosis to determine the diagnosis result of the user;

[0036] A quantitative index analysis module for determining the facial index of the temporomandibular joint deviation corresponding to the user according to the multi-modal data.

[0037] According to a third aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of the method provided in the first aspect of the present disclosure are implemented.

[0038] According to a fourth aspect of the present disclosure, there is provided an electronic device, including:

[0039] A memory on which a computer program is stored;

[0040] A processor for executing the computer program in the memory to implement the steps of the method provided in the first aspect of the present disclosure.

[0041] Compared with the prior art, the embodiments of the present disclosure have at least one of the following beneficial effects:

[0042] Through the above technical solutions, multi-view two-dimensional images of the user are collected to three-dimensionally reconstruct the three-dimensional facial model of the user, and the face scan data of the user is used to assist in improving the accuracy of the three-dimensional facial model. Based on a preset deep learning network, the diagnosis of temporomandibular joint deviation is realized using multi-view two-dimensional images, without relying on complex test results of magnetic resonance or X-ray projection, which can be used for early screening, can also assist medical diagnosis, reduce the data acquisition cost, and improve the efficiency and accuracy of medical diagnosis; the preset point cloud feature extraction network can dynamically modify key feature points according to the scene requirements, and has wide applicability and portability. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] By reading the following detailed description of the non-limiting embodiments with reference to the accompanying drawings, other features, objects, and advantages of the present disclosure will become more apparent:

[0044] Figure 1 is a schematic flowchart of a method for determining temporomandibular joint deviation based on deep learning shown in an exemplary embodiment.

[0045] Figure 2 is a schematic diagram of a data processing process of a method for determining temporomandibular joint deviation based on deep learning shown in an exemplary embodiment.

[0046] Figure 3 is a schematic diagram of a data processing process of a qualitative diagnosis shown in an exemplary embodiment.

[0047] Figure 4 It is a schematic diagram of a data processing process for quantitative index analysis shown according to an exemplary embodiment.

[0048] Figure 5 It is a block diagram of a temporomandibular joint deviation determination system based on deep learning shown according to an exemplary embodiment. Detailed implementation manners

[0049] The present disclosure will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present disclosure, but do not limit the present disclosure in any form. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present disclosure, several modifications and improvements can be made. These all belong to the protection scope of the present disclosure.

[0050] Figure 1 It is a schematic flowchart of a method for determining temporomandibular joint deviation based on deep learning shown according to an exemplary embodiment. Figure 2 It is a schematic diagram of a data processing process for a method for determining temporomandibular joint deviation based on deep learning shown according to an exemplary embodiment.

[0051] As Figure 1 、 Figure 2 shown, the present disclosure provides a method for determining temporomandibular joint deviation based on deep learning, including S11 to S16.

[0052] S11, obtaining multi-view two-dimensional images and face scan data of a user.

[0053] Among them, the multi-view two-dimensional images of the user do not need to have static features, and non-static images of the user at different times and different postures can be used to reduce the image acquisition requirements and image acquisition costs.

[0054] In the present disclosure, an image acquisition device can be used to acquire the front-view photo, left oblique-view photo, and right oblique-view photo of the user as the multi-view two-dimensional images of the user.

[0055] If required by the scenario, the multi-view two-dimensional images of the user can also be replaced with magnetic resonance (MRI) images and computed tomography (CT) images.

[0056] S12, performing three-dimensional reconstruction on the multi-view two-dimensional images of the user by using a preset multi-view photo reconstruction algorithm to determine the three-dimensional facial model of the user.

[0057] S13, respectively inputting the three-dimensional facial model and the face scan data of the user into a preset point cloud feature extraction network to determine the key feature points of the three-dimensional facial model and the key feature points of the face scan data.

[0058] Among them, the point cloud feature extraction network is used to extract key feature points and high-dimensional global point cloud features. In this step, it is used to extract key feature points with anatomical significance from the three-dimensional facial model and face scan data for different modalities of medical data.

[0059] S14. Input the key feature points of the user's three-dimensional facial model and the key feature points of the face scan data into a preset multi-modal image registration algorithm to determine multi-modal data in the same three-dimensional space.

[0060] S15. Input the multi-modal data into a preset deep neural network for diagnosis to determine the user's diagnosis result.

[0061] Among them, the user's diagnosis result includes the result of whether the user has temporomandibular joint deviation.

[0062] S16. Determine the facial indexes of the user's temporomandibular joint deviation according to the multi-modal data.

[0063] Among them, the facial indexes of the user's temporomandibular joint deviation may include the mandibular deviation angle, the maxillary retrusion angle, and the mandibular protrusion angle.

[0064] Through the above technical solution, multi-view two-dimensional images of the user are collected to three-dimensionally reconstruct the user's three-dimensional facial model, and the user's face scan data is used to assist in improving the accuracy of the three-dimensional facial model. Based on a preset deep learning network, the diagnosis of temporomandibular joint deviation is realized using multi-view two-dimensional images, without relying on complex test results of magnetic resonance or X-ray projection, which can be used for early screening, and can also assist in medical diagnosis, reduce the data acquisition cost, and improve the efficiency and accuracy of medical diagnosis; the preset point cloud feature extraction network can dynamically modify key feature points according to scene requirements, and has wide applicability and portability.

[0065] In a possible embodiment, in S12, a preset multi-view photo reconstruction algorithm is used to three-dimensionally reconstruct the user's multi-view two-dimensional images to determine the user's three-dimensional facial model, which may include:

[0066] Use a preset multi-view photo reconstruction algorithm to perform three-dimensional reconstruction of the front view image, the left oblique view image, and the right oblique view image on a preset standard human face model to determine the user's three-dimensional facial model.

[0067] Among them, by using a preset multi-view photo reconstruction algorithm to analyze the parameters of the standard human face error and the appearance consistency error of the multi-view two-dimensional images of the same user on a preset standard human face model, a personalized three-dimensional facial model of the user is reconstructed.

[0068] In a possible embodiment, the preset point cloud feature extraction network can also preprocess the initial data, and the preprocessing includes point cloud decentralization processing and noise point removal processing to remove noise points with large errors.

[0069] In a possible embodiment, S13, input the user's three-dimensional facial model and face scan data into the preset point cloud feature extraction network respectively, and determine the key feature points of the user's three-dimensional facial model and the key feature points of the face scan data, including S131 to S132.

[0070] S131, use the preset point cloud feature extraction network to perform point cloud decentralization processing and noise point removal processing on the user's three-dimensional facial model and face scan data in sequence, and determine the preprocessed user's three-dimensional facial model and the preprocessed face scan data.

[0071] S132, perform key feature point extraction processing on the preprocessed user's three-dimensional facial model and the preprocessed face scan data, and determine the key feature points of the user's three-dimensional facial model and the key feature points of the face scan data.

[0072] In the present disclosure, the key feature points with anatomical significance pre-manually marked include the feature points of the left inner canthus, right inner canthus, glabella, nasion, subnasale, mental point, left alar, right alar, left tragus and right tragus.

[0073] Use the preset point cloud feature extraction network to extract the feature points of the left inner canthus, right inner canthus, glabella, nasion, subnasale, mental point, left alar, right alar, left tragus and right tragus of the preprocessed user's three-dimensional facial model and the preprocessed face scan data.

[0074] A method for determining temporomandibular joint deviation based on deep learning provided by the present disclosure can also pre-manually mark the types of key feature points according to specific application scenario requirements, and use the preset point cloud feature extraction network to extract the key feature points corresponding to the types of key feature points, which has wide applicability and portability.

[0075] In a possible embodiment, S14, input the key feature points of the user's three-dimensional facial model and the key feature points of the face scan data into the preset multimodal image registration algorithm to determine the multimodal data in the same three-dimensional space, which can include S141 to S142.

[0076] S141, use the preset multimodal image registration algorithm to determine the rotation matrix and / or translation matrix of the key feature points of the user's three-dimensional facial model and the key feature points of the face scan data.

[0077] Among them, the key feature points of the user's three-dimensional facial model and the key feature points of the face scan data are multimodal data with different initial positions and scales.

[0078] S142. Rotate and / or translate the key feature points of the user's three-dimensional facial model and the key feature points of the face scan data to a unified three-dimensional space reference system according to the rotation matrix and / or translation matrix, and determine multi-modal data in the same three-dimensional space.

[0079] Registering the key feature points of the user's three-dimensional facial model and the key feature points of the face scan data in the same three-dimensional space can meet the data requirements for subsequent qualitative diagnosis and quantitative index analysis.

[0080] Figure 3 It is a schematic diagram of a data processing process for qualitative diagnosis shown according to an exemplary embodiment.

[0081] As Figure 3 shown, in a possible embodiment, a method for determining temporomandibular joint deviation based on deep learning may further include S17 to S19.

[0082] S17. Perform segmentation processing on the multi-modal data to determine the mandibular region data.

[0083] Segment the mandibular region data from the facial region constructed by the three-dimensional key feature points for qualitative diagnosis of temporomandibular joint deviation and prevent the influence of other facial regions.

[0084] S18. Use a preset downsampling algorithm to perform data augmentation processing on the mandibular region data to determine the mandibular region data after data augmentation.

[0085] Among them, the preset downsampling algorithm may include a random sampling algorithm, a farthest point sampling algorithm, a voxel grid sampling algorithm, and a Lloyd algorithm.

[0086] As an example, sequentially use a random sampling algorithm, a farthest point sampling algorithm, a voxel grid sampling algorithm, and a Lloyd algorithm to perform data augmentation on the mandibular region data of each user's three-dimensional facial model, expand the database, and provide comprehensive point cloud information.

[0087] Based on the limitation of computing power, use a preset downsampling algorithm to perform data augmentation processing on the mandibular region data, amplify the mandibular region data, and improve the diagnostic accuracy of the preset deep neural network.

[0088] Since the data processed by the deep neural network is high-level features and has no interpretability, therefore, first input the mandibular region data after data augmentation into a preset machine learning algorithm to verify the feasibility and distinguishability, and then input it into a preset deep neural network for intelligent diagnosis.

[0089] S19. Input the data of the mandibular region after data augmentation into a preset machine learning algorithm for verification to determine the verification result.

[0090] Among them, the preset machine learning algorithm is a traditional machine learning algorithm, and support vector machine and random forest algorithm can be adopted.

[0091] The verification process is also the preliminary diagnosis process of temporomandibular joint deviation, and the verification result includes the feasibility and distinguishability of the determination of temporomandibular joint deviation.

[0092] Steps S17 to S19 of the present disclosure are executed after step S14.

[0093] In a possible embodiment, S15. Input the multi-modal data into a preset deep neural network for diagnosis to determine the diagnosis result of the user, which may include:

[0094] Input the data of the mandibular region after data augmentation into a preset deep neural network to determine the diagnosis result of the user.

[0095] In the present disclosure, the preset deep neural network is PointNet++, and PointNet++ can realize the diagnosis and classification of diseases after being trained with a specific binary classification dataset.

[0096] The preset deep neural network can also adopt PointNet and DGCNN.

[0097] In this step, the preset deep neural network gradually realizes the diagnosis and determination of temporomandibular joint deviation from local features to global features for the data of the mandibular region segmented from the three-dimensional facial model to determine whether the user has temporomandibular joint deviation.

[0098] The preset deep neural network can be pre-trained and learned with a database of the mandibular region after data augmentation to learn the diagnosis of temporomandibular joint deviation.

[0099] Figure 4 It is a schematic diagram of the data processing process of a quantitative index analysis shown according to an exemplary embodiment.

[0100] In a possible embodiment, the facial indexes of the user corresponding to temporomandibular joint deviation include mandibular deviation angle, maxillary posterior retraction angle, and mandibular anterior protrusion angle.

[0101] As Figure 4 shown, S16. Determine the facial indexes of the user corresponding to temporomandibular joint deviation according to the multi-modal data, including S161 to S162.

[0102] S161. Use the key feature points of the facial data to correct the key feature points of the user's three-dimensional facial model, and determine the target key feature points.

[0103] Among them, errors may occur during the reconstruction of the user's three-dimensional facial model. Use the face scan data as a standard to assist in correcting the key feature points of the user's three-dimensional facial model, improve the accuracy of the key feature points, and determine the target key feature points.

[0104] S162. Calculate the mandibular deviation angle, maxillary retrusion angle, and mandibular protrusion angle of the user according to the target key feature points.

[0105] Among them, the three-dimensional coordinate information of the target key points can be obtained, and the mandibular deviation angle, maxillary retrusion angle, and mandibular protrusion angle of the user can be calculated according to the three-dimensional coordinate information of the target key feature points, so as to determine the facial indexes of the corresponding temporomandibular joint deviation of the user, enhance the medical staff's and the user's understanding of the case, improve the communication efficiency between the medical staff and the user, and assist the medical staff in formulating a targeted diagnosis and treatment plan.

[0106] A method for determining temporomandibular joint deviation based on deep learning provided by the present disclosure uses a three-dimensional facial model reconstructed from the user's multi-view two-dimensional images to diagnose and determine occult temporomandibular joint deviation, realizing early screening of temporomandibular joint deviation, and can also improve the diagnosis efficiency and accuracy of medical staff; it can also dynamically modify the types of key feature points marked by the preset point cloud feature extraction network according to the target disease diagnosis scenario, so as to be better applied to other medical diagnosis fields.

[0107] The method for diagnosing and determining occult temporomandibular joint deviation by reconstructing a three-dimensional facial model using multi-view two-dimensional images and using deep neural network learning provided by the present disclosure is faster and more accurate than the traditional diagnosis method, and can reduce the data acquisition cost and improve the diagnosis efficiency and accuracy of medical staff.

[0108] Figure 5 It is a block diagram of a system for determining temporomandibular joint deviation based on deep learning shown according to an exemplary embodiment.

[0109] Based on the same concept, the present disclosure also provides a system for determining temporomandibular joint deviation based on deep learning, as Figure 5 shown, including: a data acquisition module 110, a three-dimensional reconstruction module 120, a key feature point extraction module 130, a multi-modal registration module 140, a qualitative diagnosis module 150, and a quantitative index analysis module 160.

[0110] The data acquisition module 110 is used to obtain the multi-view two-dimensional images and face scan data of the user;

[0111] A three-dimensional reconstruction module 120, configured to perform three-dimensional reconstruction on multi-view two-dimensional images of a user by using a preset multi-view photo reconstruction algorithm to determine a three-dimensional facial model of the user;

[0112] A key feature point extraction module 130, configured to input the three-dimensional facial model of the user and the face scan data into a preset point cloud feature extraction network respectively to determine the key feature points of the three-dimensional facial model of the user and the key feature points of the face scan data;

[0113] A multi-modal registration module 140, configured to input the key feature points of the three-dimensional facial model of the user and the key feature points of the face scan data into a preset multi-modal image registration algorithm to determine multi-modal data in the same three-dimensional space;

[0114] A qualitative diagnosis module 150, configured to input the multi-modal data into a preset deep neural network for diagnosis to determine a diagnosis result of the user;

[0115] A quantitative index analysis module 160, configured to determine facial indexes of the user corresponding to temporomandibular joint deviation according to the multi-modal data.

[0116] Through the above technical solutions, multi-view two-dimensional images of a user are collected to three-dimensionally reconstruct the three-dimensional facial model of the user, and the face scan data of the user is used to assist in improving the accuracy of the three-dimensional facial model. Based on a preset deep learning network, the diagnosis of temporomandibular joint deviation is realized by using multi-view two-dimensional images, without relying on complex test results of magnetic resonance or X-ray projection, which can be used for early screening, and can also assist medical diagnosis, reduce the data acquisition cost, and improve the efficiency and accuracy of medical diagnosis; the preset point cloud feature extraction network can dynamically modify key feature points according to scene requirements, and has wide applicability and portability.

[0117] Regarding the embodiments of the above system, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated herein.

[0118] Based on the same concept as above, in another embodiment of the present disclosure, an electronic device is further provided, including a memory, a processor, and a computer program stored on the memory and capable of running on the processor. When the processor executes the program, it is used to execute a method for determining temporomandibular joint deviation based on deep learning.

[0119] Optionally, a memory for storing programs; the memory may include volatile memory (e.g., random-access memory, such as static random-access memory (SRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM), etc.); the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs and functional modules for implementing the above methods), computer instructions, etc. The above computer programs, computer instructions, etc. can be stored in partitions in one or more memories. And the above computer programs, computer instructions, data, etc. can be called by the processor.

[0120] The above computer programs, computer instructions, etc. can be stored in partitions in one or more memories. And the above computer programs, computer instructions, data, etc. can be called by the processor.

[0121] A processor for executing the computer programs stored in the memory to implement each step in the method related to the above embodiments. For details, reference can be made to the relevant descriptions in the foregoing method embodiments.

[0122] The processor and the memory can be of independent structures or integrated structures. When the processor and the memory are of independent structures, the memory and the processor can be coupled and connected through a bus.

[0123] In the embodiments of the present disclosure, a non-transitory computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by the processor, the steps of a method for determining temporomandibular joint deviation based on deep learning in any of the above embodiments are implemented.

[0124] Those skilled in the art should understand that the embodiments of the present disclosure can be provided as a method, a system, or a computer program product. Therefore, the present disclosure can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present disclosure can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program codes.

[0125] This disclosure is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the disclosure. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processors of general-purpose computers, special-purpose computers, embedded processors, or other programmable data processing devices to produce a machine, such that the instructions executed by the processors of the computer or other programmable data processing devices produce means for implementing the functions specified in one or more flows Figure 1 one or more flows and / or blocks Figure 1 or means for implementing the functions specified in one or more blocks.

[0126] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufactured article including instruction means for implementing the functions specified in one or more flows Figure 1 one or more flows and / or blocks Figure 1 or means for implementing the functions specified in one or more blocks.

[0127] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operational steps are performed on the computer or other programmable device to produce a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more flows Figure 1 one or more flows and / or blocks Figure 1 or means for implementing the functions specified in one or more blocks.

[0128] Although the preferred embodiments of the disclosure have been described, additional changes and modifications can be made to these embodiments by those skilled in the art once they learn of the basic creative concept. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications that fall within the scope of the disclosure.

[0129] Obviously, those skilled in the art can make various changes and variations to the disclosure without departing from the spirit and scope of the disclosure. Thus, if these modifications and variations of the disclosure fall within the scope of the claims of the disclosure and their equivalent technologies, the disclosure is also intended to include these changes and variations.

Claims

1. A method for determining temporomandibular joint deviation based on deep learning, characterized in that: include: Obtain multi-view 2D images and face scan data of the user; Using a preset multi-view photo reconstruction algorithm to perform three-dimensional reconstruction on the multi-view two-dimensional image of the user to determine a three-dimensional facial model of the user; Inputting the user's three-dimensional facial model and the facial scan data into a preset point cloud feature extraction network respectively, and determining key feature points of the user's three-dimensional facial model and key feature points of the facial scan data; Inputting key feature points of the three-dimensional facial model of the user and key feature points of the face scan data into a preset multimodal image registration algorithm to determine multimodal data in the same three-dimensional space; Inputting the multimodal data into a preset deep neural network for diagnosis to determine a diagnosis result for the user; A facial index of the temporomandibular joint deviation corresponding to the user is determined based on the multimodal data.

2. The method according to claim 1, characterized in that The multi-view two-dimensional image of the user includes a front view image, a left oblique view image and a right oblique view image; The method of using a preset multi-view photo reconstruction algorithm to perform three-dimensional reconstruction on the multi-view two-dimensional image of the user to determine the three-dimensional facial model of the user includes: The preset multi-view photo reconstruction algorithm is used to perform three-dimensional reconstruction on the frontal view image, the left oblique view image and the right oblique view image on a preset standard face model to determine the three-dimensional facial model of the user.

3. The method according to claim 1, characterized in that The key feature points include the feature points of the left inner canthus, the right inner canthus, the center of the eyebrows, the root of the nose, the base of the columella, the chin point, the left nose wing, the right nose wing, the left tragus and the right tragus.

4. The method according to claim 1, characterized in that: The step of inputting the three-dimensional facial model of the user and the facial scan data into a preset point cloud feature extraction network to determine key feature points of the three-dimensional facial model of the user and key feature points of the facial scan data comprises: Using the preset point cloud feature extraction network to sequentially perform point cloud decentralization processing and noise point removal processing on the three-dimensional facial model of the user and the face scan data, to determine a pre-processed three-dimensional facial model of the user and the pre-processed face scan data; Key feature point extraction processing is performed on the preprocessed three-dimensional facial model of the user and the preprocessed face scan data to determine the key feature points of the three-dimensional facial model of the user and the key feature points of the face scan data.

5. The method according to claim 1, characterized in that The method further comprises: Segmenting the multimodal data to determine mandibular area data; Performing data enhancement processing on the mandibular region data by using a preset downsampling algorithm to determine the mandibular region data after data enhancement; The data-enhanced mandibular area data is input into a preset machine learning algorithm for verification to determine a verification result.

6. The method according to claim 5, characterized in that The step of inputting the multimodal data into a preset deep neural network for diagnosis to determine the diagnosis result of the user includes: The data-enhanced mandibular area data is input into the preset deep neural network to determine the diagnosis result of the user.

7. The method according to claim 1, characterized in that The facial indicators of temporomandibular joint deviation corresponding to the user include mandibular deviation angle, maxillary face retraction angle, and mandibular face protrusion angle; Determining the facial index of the temporomandibular joint deviation corresponding to the user according to the multimodal data includes: Correcting the key feature points of the three-dimensional facial model of the user using the key feature points of the facial data to determine target key feature points; The mandibular deviation angle, the maxillary facial retraction angle, and the mandibular protrusion angle of the user are calculated based on the target key feature points.

8. A temporomandibular joint deviation determination system based on deep learning, characterized in that: include: A data acquisition module, used to obtain multi-view two-dimensional images and face scan data of the user; A three-dimensional reconstruction module, used to perform three-dimensional reconstruction on the multi-view two-dimensional image of the user using a preset multi-view photo reconstruction algorithm to determine a three-dimensional facial model of the user; A key feature point extraction module, used to input the three-dimensional facial model of the user and the face scan data into a preset point cloud feature extraction network, and determine the key feature points of the three-dimensional facial model of the user and the key feature points of the face scan data; A multimodal registration module, used for inputting key feature points of the three-dimensional facial model of the user and key feature points of the face scan data into a preset multimodal image registration algorithm to determine multimodal data in the same three-dimensional space; A qualitative diagnosis module, used to input the multimodal data into a preset deep neural network for diagnosis, and determine the diagnosis result of the user; The quantitative index analysis module is used to determine the facial index of the temporomandibular joint deviation corresponding to the user according to the multimodal data.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 7.

Images