Temporomandibular joint motion function state recognition method and device based on multi-modal deep learning and electronic equipment

Abstract

The invention relates to a temporal-mandibular joint motion function state recognition method and device based on multi-modal deep learning and electronic equipment. The method comprises the following steps: acquiring a multi-modal data sample from a pre-constructed temporal-mandibular joint motion function state database, and preprocessing the multi-modal data sample; respectively training and optimizing a pre-trained image segmentation model and a pre-trained image classification model according to data categories based on the preprocessed multi-modal data sample; fusing the trained image segmentation model and the image classification model by using a back-end fusion method to obtain a multi-modal deep learning model, and training and optimizing the multi-modal deep learning model by using the preprocessed multi-modal data sample; multi-modal data to be recognized are obtained and input into the trained multi-modal deep learning model, and a corresponding temporomandibular joint motion function state recognition result is obtained. Compared with the prior art, the method has the advantages that the temporal-mandibular joint motion function state recognition efficiency and accuracy are improved, and the like.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence deep learning, and in particular to a method, device and electronic equipment for identifying the motion function state of a temporomandibular joint based on multimodal deep learning. Background Art

[0002] Temporomandibular Disorder (TMD) is a common disease caused by displacement of the temporomandibular joint disc or functional or organic changes in its surrounding ligaments, muscles and condyles. Temporomandibular joint disorder (TMD) is one of the common diseases of the oral and maxillofacial region, which is prone to occur in young and middle-aged people, with the highest prevalence and consultation rate in the 20-30 age group. TMD does not refer to a single disease, but a general term for a group of diseases with the same or similar clinical symptoms whose etiology is not yet fully understood. Patients often show symptoms such as pain in the joint area, tenderness in the masticatory muscles, limited mouth opening, joint snapping, headache and ear pain. According to statistics, the global prevalence of TMD is 28%, which is the main cause of non-odontogenic oral and facial pain. The clinical manifestations of TMD include abnormal mandibular movement, pain, snapping and murmur, headache, ear disease, etc. Therefore, for the problem of temporomandibular joint disease caused by joint disc displacement, the prediction of abnormal mandibular movement status plays an important role in assisting the diagnosis of TMD.

[0003] At present, magnetic resonance imaging (MRI) of the temporomandibular joint can clearly display the soft tissue structure of the temporomandibular joint. Displacement of the articular disc includes: when the mandible is in the intercusp position (closed mouth position), the articular disc is displaced anteriorly and inferiorly relative to the anterior slope of the condyle (the posterior boundary of the articular disc is located before the 11:30 hour hand position), and the anteriorly displaced articular disc is usually accompanied by varying degrees of deformation. If the anterior displacement of the disc is reducible, the middle zone of the articular disc is located between the condylar head and the articular tubercle at the maximum opening position; if the anterior displacement of the disc is irreducible, the posterior zone of the articular disc is still located in front of the condylar head at the maximum opening position. However, MRI technology has certain limitations and may lead to misdiagnosis due to the existence of false images. For example, in patients with reducible anterior displacement of the disc, if the degree of opening does not reach the position where the articular disc can be reduced during arthrography or MRI examination, it will also be misdiagnosed as "irreducible anterior displacement of the disc". Sometimes, the imaging findings of the temporomandibular joint are inconsistent with the clinical manifestations: some patients have severe symptoms but no severe abnormalities or only slight abnormalities in imaging examinations; while some other patients have severe degenerative changes in imaging examinations, such as osteophyte formation, cystic changes, bone sclerosis, and obvious anterior displacement and deformation of the articular disc, but the clinical manifestations are mild or even without obvious clinical symptoms. Because many diseases can have similar or even identical imaging manifestations to TMD, and because imaging diagnosis is complex and often lacks specific diagnostic indications, it is far from enough to estimate and judge the abnormal state of mandibular movement based solely on medical imaging examinations. It must be closely combined with clinical conditions and other auxiliary technologies.

[0004] Mandibular movement analysis is another important reference method. Patients with temporomandibular joint diseases will have abnormal temporomandibular joint movement. At present, orthodontists in clinical practice analyze mandibular movement through clinical examinations. This clinical examination is highly subjective and difficult for junior doctors to master. The electronic face bow technology can accurately reflect the trajectory of mandibular movement to a certain extent. The occlusal relationship module of the electronic face bow measures the central relationship position. Patients only need to follow the system instructions to open and close their mouths and perform Gothic movements to quickly and accurately find the central relationship position. The electronic position analysis of the condyle can target the occlusal position of the temporal joint and effectively verify the treatment of the bite plate. The measurement process only takes 40 seconds to obtain the tracing of the movement trajectory of the condyle and the incisal edge, which assists doctors in analysis and diagnosis. In addition, the electronic face bow has a compact design and a simple operation process. You can wear the face bow with a simple finger operation, which solves the poor experience brought to patients by the complicated operation of the traditional face bow.

[0005] In addition to the above examination methods, CT can reconstruct and obtain images of any layer of the temporomandibular joint. CT images have good spatial resolution and can display the bony structure of the joint well. Some cone beam CT (CBCT) devices can also clearly display the condylar compact bone plate and the trabecular bone structure of the medullary cavity. Therefore, CT can show bone changes in any part of the condyle and articular tubercle. Although imaging examinations are the main auxiliary examination method for temporomandibular joint disorders, they should still be combined with clinical examinations to jointly judge the status of the temporomandibular joint.

[0006] Multimodal Deep Learning (MMDL) is an important research direction that has emerged in recent years with the development of deep learning. Its main purpose is to use deep learning methods to process and understand information from multiple sources. Different modal data have different ways of expressing things and different perspectives on things, so there is information redundancy and complementarity. If the information from multiple sources can be processed reasonably, more abundant feature information can be obtained, which is very helpful for auxiliary diagnosis. Compared with single-modal deep learning, the characteristic of multimodal deep learning is that the network input layer needs to have the ability to accept different modal data inputs at the same time, and in the training process of multimodal deep learning models, it is necessary not only to fully extract and utilize the useful information within each modality, but also to simultaneously explore the complementarity between the modal data and perform effective information fusion to achieve better performance than the single-modal model. Therefore, in the process of applying deep learning technology to medical research, strengthening the application of multimodal deep learning technology in clinical medicine is the current development trend.

[0007] In summary, when predicting the motor function state of the temporomandibular joint, the more comprehensive the factors considered, such as clinical conditions, motor status, medical imaging manifestations, related laboratory tests, etc., the more accurate the prediction results will be. However, the current application of related technologies is mostly based on data from a single source, such as relying solely on magnetic resonance imaging of the temporomandibular joint, or relying solely on CBCT data of the temporomandibular joint. It is rare to be able to conduct a comprehensive analysis based on multiple imaging examinations of the temporomandibular joint, resulting in low efficiency and accuracy in identifying the motor function state of the temporomandibular joint. Therefore, how to further improve the efficiency and accuracy of identifying the motor function state of the temporomandibular joint has become a problem that needs to be solved in this field. Summary of the invention

[0008] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a method, device and electronic device for identifying the functional state of the temporomandibular joint movement based on multimodal deep learning. Through the orthodontic big database, an orthodontic mandibular movement analysis method is established. Combined with artificial intelligence technology, the learning cost can be reduced, and the temporomandibular joint magnetic resonance imaging, temporomandibular joint CBCT images, and mandibular movement trajectory data can be integrated to more quickly and accurately predict the state of the temporomandibular joint from multiple dimensions, thereby training an analysis system with an analysis level close to that of experts to assist clinical diagnosis and treatment.

[0009] The purpose of the present invention can be achieved by the following technical solutions:

[0010] According to a first aspect of the present invention, a method for identifying the motion function state of the temporomandibular joint based on multimodal deep learning is provided, comprising the following steps: obtaining and preprocessing multimodal data samples, the multimodal data samples comprising temporomandibular joint image data and temporomandibular joint electronic face bow data, the multimodal data samples being obtained from a pre-constructed temporomandibular joint motion function state database; based on the preprocessed multimodal data samples, respectively training and optimizing the pre-trained image segmentation model and the image classification model according to the data category; utilizing a back-end fusion method to fuse the trained image segmentation model and the image classification model to obtain a multimodal deep learning model, and utilizing the preprocessed multimodal data samples to train and optimize the multimodal deep learning model; obtaining the multimodal data to be identified and inputting it into the trained multimodal deep learning model to obtain the corresponding recognition result of the motion function state of the temporomandibular joint.

[0011] As a preferred technical solution, the specific process of the preprocessing includes: denoising, labeling, classifying and normalizing the multimodal data samples.

[0012] As a preferred technical solution, a pre-trained image segmentation model and an image classification model are trained and optimized according to data categories. The specific process includes: using pre-processed temporomandibular joint image data to train and optimize the pre-trained image segmentation model, wherein the temporomandibular joint image data includes temporomandibular joint magnetic resonance imaging data and temporomandibular joint cone beam CT image data; and, using pre-processed temporomandibular joint electronic face bow data to train and optimize the pre-trained image classification model.

[0013] As a preferred technical solution, the preprocessing includes labeling processing, and the training process of the multimodal deep learning model specifically includes: using the labeled multimodal data samples as training samples; dividing the training samples into a training set and a validation set; based on preset initial model parameters, using the sample data in the training set and the gradient descent algorithm to iteratively train the multimodal deep learning model; and using the sample data in the validation set to determine the optimal parameters of the multimodal deep learning model.

[0014] As an optimal technical solution, a back-end fusion method is used to fuse the trained image segmentation model and the image classification model to obtain a multimodal deep learning model. The specific process includes: building a framework of the multimodal deep learning model, including at least one fusion layer and one optimization layer; connecting the trained feature extraction layers of the image segmentation model and the image classification model to the fusion layer, and the connection method adopts one of weighted average, fully connected layer, and attention mechanism; adding the optimization layer after the fusion layer, and the optimization layer adopts one of fully connected layer, normalization layer, and activation function.

[0015] As a preferred technical solution, the temporomandibular joint image data to be identified is obtained and input into a trained multimodal deep learning model to obtain the corresponding temporomandibular joint movement function state recognition result. The specific process includes: preprocessing the input multimodal data to be identified; based on the preprocessed temporomandibular joint image data, using the trained image segmentation model to extract the image features of the key areas and structures of the temporomandibular joint; based on the preprocessed electronic face bow data, using the trained image classification model to extract the motion trajectory classification features of the temporomandibular joint; inputting the image features and the motion trajectory classification features into the fusion layer, and outputting the corresponding preliminary fusion features; inputting the preliminary fusion features into the optimization layer, and outputting the final fusion features.

[0016] As a preferred technical solution, the multimodal data sample also includes clinical examination text data.

[0017] As a preferred technical solution, the method for identifying the functional state of the temporomandibular joint movement also includes the following steps: training and optimizing the pre-trained natural language processing model based on the pre-processed clinical examination text data; utilizing a back-end fusion method to fuse the trained natural language processing model, image segmentation model, and image classification model to obtain a multimodal deep learning model, and performing training and optimization.

[0018] According to a second aspect of the present invention, there is provided a device for identifying the movement function state of a temporomandibular joint based on multimodal deep learning, comprising a data acquisition and preprocessing module, a single-modal model training module, a multimodal model fusion and training module, and a function state identification module, wherein the data acquisition and preprocessing module is used to acquire and preprocess multimodal data samples, wherein the multimodal data samples comprise temporomandibular joint image data and temporomandibular joint electronic face bow data, and wherein the multimodal data samples are acquired from a pre-constructed temporomandibular joint movement function state database;

[0019] The single-modal model training module is used to train and optimize the pre-trained image segmentation model and image classification model according to the data category based on the pre-processed multi-modal data samples; the multi-modal model fusion and training module is used to use the back-end fusion method to fuse the trained image segmentation model and image classification model to obtain a multi-modal deep learning model, and use the pre-processed multi-modal data samples to train and optimize the multi-modal deep learning model; the functional state recognition module is used to obtain the multi-modal data to be recognized and input it into the trained multi-modal deep learning model to obtain the corresponding temporomandibular joint movement functional state recognition result.

[0020] According to a third aspect of the present invention, there is provided an electronic device, comprising a memory, a processor, and a program stored in the memory, wherein the processor implements the method when executing the program.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] 1. The method for identifying the motion function state of the temporomandibular joint provided by the present invention collects the patient's temporomandibular joint image data and the temporomandibular joint electronic face bow data. The temporomandibular joint image data includes the temporomandibular joint MRI image data and the temporomandibular joint CBCT image data, which respectively provide detailed structural information of the temporomandibular joint soft tissue and bone structure information. The temporomandibular joint electronic face bow data is used to reflect the motion trajectory of the temporomandibular joint. On this basis, the corresponding single-modal deep neural network model is first constructed, trained and optimized according to the data type, and then multiple single-modal models are fused to construct, train and optimize the multi-modal deep learning model, and then the recognition result of the motion function state of the temporomandibular joint is predicted according to the multi-modal data to be identified. The method takes into account a variety of data, can provide multi-perspective and more comprehensive information for the multi-modal deep learning model, make up for the shortcomings of single-modal data, and effectively improve the accuracy of the recognition of the motion function state of the temporomandibular joint. At the same time, the multi-modal deep learning model can process the input multi-modal data in real time, automatically extract the corresponding features and fuse them, reduce manual intervention, and significantly improve the efficiency and convenience of clinical application;

[0023] 2. The method for identifying the functional state of the temporomandibular joint movement provided by the present invention, when training a single-modality deep learning model, respectively uses the pre-processed temporomandibular joint image data to train and optimize the pre-trained image segmentation model, and uses the pre-processed temporomandibular joint electronic face bow data to train and optimize the pre-trained image classification model, so as to ensure that each single-modality model has optimal parameters in its specific field (such as soft tissue, bone structure, and motion trajectory), so that the data features of each modality can be fully utilized, and the multi-modal deep model comprehensively considers the features of each modality when performing prediction and recognition in the end, so that the recognition result is more reliable and accurate;

[0024] 3. The multimodal data samples collected by the method for identifying the motor function state of the temporomandibular joint provided by the present invention may include clinical examination text data, including detailed information such as diagnosis records, patient's medical history, symptom description, treatment plan, etc., which can further enrich data diversity and information richness, improve the robustness and generalization ability of the multimodal deep learning model, and thus improve the accuracy of identifying the motor function state of the temporomandibular joint, and provide a more reliable and detailed reference;

[0025] 4. The temporomandibular joint movement function state identification device and electronic device provided by the present invention are loaded with the method provided by the present invention, have a simple structure, low cost, high computing efficiency, are easy to be integrated into clinical medical equipment for operation, improve the prediction efficiency of the temporomandibular joint state, and reduce the user's learning cost, which is convenient for helping areas with poor medical levels to improve the level of intelligent clinical diagnosis and treatment. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A schematic flow chart of the method is provided for the present invention. DETAILED DESCRIPTION

[0027] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0028] Example

[0029] like Figure 1As shown, the present embodiment provides a method for identifying the motion function state of the temporomandibular joint based on multimodal deep learning. The method first obtains and preprocesses multimodal data samples, the multimodal data samples include temporomandibular joint image data and temporomandibular joint electronic face bow data, and the multimodal data samples are obtained from a pre-constructed temporomandibular joint motion function state database; secondly, based on the preprocessed multimodal data samples, the pre-trained image segmentation model and the image classification model are trained and optimized according to the data category; thirdly, the trained image segmentation model and the image classification model are fused by the back-end fusion method to obtain a multimodal deep learning model, and the preprocessed multimodal data samples are used to train and optimize the multimodal deep learning model; finally, the multimodal data to be identified is obtained and input into the trained multimodal deep learning model to obtain the corresponding temporomandibular joint motion function state recognition result. Specifically:

[0030] (1) Data collection and preprocessing, database establishment

[0031] Collect imaging data, including TMJ MRI, TMJ CBCT and other imaging types, and collect the patient's electronic facebow data. Preprocess and annotate the imaging data, such as image reconstruction, image segmentation, and marking key structures. During the data collection process, strictly abide by relevant ethical standards and laws and regulations to protect patient privacy and data security. Obtain informed consent from patients and conduct ethical review and approval when necessary.

[0032] Preprocessing of various collected image data and electronic face bow data, including noise reduction, normalization, labeling and classification, can also be done by establishing unified standards for data labeling in advance, and then building a complete database of temporomandibular joint movement function status. In the database, electronic face bow data can reflect the mandibular movement trajectory of orthodontic temporomandibular joints with different symptoms; changes in image information of temporomandibular joint disc displacement include: normal mandibular movement with left reducible disc anterior displacement, right reducible disc anterior displacement, left irreducible disc anterior displacement, right irreducible disc anterior displacement, bilateral irreducible disc anterior displacement, bilateral reducible disc anterior displacement, etc.

[0033] Preprocessing is performed on TMJ MRI, TMJ CBCT, and TMJ electronic facebow data. For multi-layer 3D data, key images are extracted to segment the TMJ region of interest so that the model can better focus on the lesion area. In addition, the image size can be adjusted and the grayscale of the image data can be standardized to ensure the consistency of the input model.

[0034] (3) Model selection and adjustment

[0035] For data of different modalities, select the appropriate deep learning model according to the data category: for temporomandibular joint MRI images and temporomandibular joint CBCT images, consider using an algorithm model based on image segmentation; for electronic facebow data, consider using an algorithm model based on image classification.

[0036] According to different types of models and corresponding input data, pre-trained models are loaded separately, and the models are fine-tuned to make the data converge quickly.

[0037] (4) Single modal model training

[0038] The method provided in this embodiment adopts a distributed training strategy to train the multimodal deep learning model, that is, first train the single-modal deep learning model separately, and then train the multimodal deep learning model.

[0039] The training of the unimodal deep learning model uses a transfer learning strategy. On the basis of the model being pre-trained with other data sets, the model is trained using the training set samples in the database of temporomandibular joint movement function status, and the performance evaluation and parameter tuning are performed through the validation set. During training, the multimodal data samples that have been annotated are first used as training samples, and the training samples are divided into a training set and a validation set; then, the model that has been pre-trained on the ImageNet data set is loaded, and then the model is fine-tuned, and the adjusted model is trained using the training set sample data to learn to identify different types of temporomandibular joint movement function states. The performance of the model is monitored during the training process, and adjustments and optimizations are made as needed to improve the accuracy of the model. After the training is completed, the valid feature information is stored in each unimodal model. The trained unimodal model is verified on the sample data of the validation set to obtain the model with the highest prediction accuracy, and it is exported for backup. Specifically:

[0040] The pre-processed temporomandibular joint MRI image data and CBCT images in the training set are used to train and optimize the pre-trained image segmentation model. The image segmentation model uses one of Recurrent neural networks (RNNs), MDRNNs (Multi-Dimensional Recurrent Neural Networks), FCN (Fully Convolutional Network), and SetNet. The trained image segmentation model can output the segmented image features. In the subsequent testing process using the validation set, the input MRI image features and CBCT image features can be segmented to identify the key areas and structures of the temporomandibular joint.

[0041] The pre-processed temporomandibular joint electronic face bow data in the training set was used to train and optimize the pre-trained image classification model. The image classification model used one of EfficientNetb3, ResNet, DenseNet, and 3D-ResNet. The model was modified by modifying the fully connected layer of the pre-trained neural network, and the hyperparameters of the neural network model were optimized using the gradient descent algorithm. The trained image classification model can identify the category corresponding to the electronic face bow image features. In the subsequent testing process using the validation set, the presence of temporomandibular joint movement abnormalities can be determined by identifying the image category. The image feature screening of temporomandibular joint movement abnormalities includes: classifying the temporomandibular joint movement function status according to different types of mandibular movement using the input mandibular electronic face bow data; and inputting each image block in the image block sequence into the corresponding trained mandibular movement classification model to determine whether the input electronic face bow data has the corresponding temporomandibular joint movement function status. The process of acquiring the image block sequence includes: inputting the mandibular electronic facebow data into the trained image classification model to generate the corresponding electronic facebow motion image and locate the fault range of each anatomical area; respectively extracting the image block sequence of each mandibular functional state layer by layer from the located different electronic facebow motion data.

[0042] (5) Multimodal model training

[0043] The best performing single-modal models on the validation set are extracted respectively, and the previously trained single-modal models are loaded into the multimodal deep learning model using the backend fusion method. Finally, the parameters of the entire multimodal deep learning model are fine-tuned, and the model is trained multiple times until the multimodal deep learning model converges well. Specifically:

[0044] The trained single-modal models are fused using the back-end fusion method to obtain a multimodal deep learning model. The specific process includes: building a framework of the multimodal deep learning model, including at least one fusion layer and one optimization layer; connecting the feature extraction layer of the trained image segmentation model and image classification model to the fusion layer, and the connection method adopts one of weighted average, fully connected layer, and attention mechanism; adding an optimization layer after the fusion layer, and the optimization layer adopts one of fully connected layer, normalization layer, and activation function.

[0045] The training process of the multimodal deep learning model specifically includes: first, using the labeled multimodal data samples as training samples; second, dividing the training samples into a training set and a validation set; then, based on the preset initial model parameters, using the sample data in the training set and the gradient descent algorithm to iteratively train the multimodal deep learning model to optimize the model's hyperparameters; finally, using the sample data in the validation set to determine the optimal parameters of the multimodal deep learning model, and ultimately obtaining the model with the highest recognition accuracy, and exporting the model for future use.

[0046] The process of functional state identification of input data using the exported model is as follows:

[0047] a. Obtain the temporomandibular joint MRI, temporomandibular joint CBCT and electronic face bow data to be identified and input them into the trained multimodal deep learning model;

[0048] b. Preprocessing the input multimodal data to be identified;

[0049] c. Input the preprocessed temporomandibular joint MRI data and temporomandibular joint CBCT data into the trained image segmentation model to extract the image features of the key areas and structures of the temporomandibular joint;

[0050] d. Input the preprocessed electronic face bow data into the trained image classification model to extract the motion trajectory classification features of the temporomandibular joint;

[0051] e. Input the image features and motion trajectory classification features into the fusion layer and output the corresponding preliminary fusion features;

[0052] f. Input the preliminary fusion features into the optimization layer and output the final fusion features, that is, the final recognition result of the temporomandibular joint motion function state.

[0053] (6) Model evaluation and validation

[0054] Use the test group data to evaluate and verify the trained model to assess its generalization ability and performance on unseen data, analyze the model's prediction results, and use appropriate evaluation indicators to evaluate the model's accuracy, recall, precision, F1 score, etc. Evaluate the model's performance in identifying different types of temporomandibular joint movement function states.

[0055] (7) Model deployment and application

[0056] The trained model is deployed in a clinical setting to assist doctors in image diagnosis of temporomandibular joint disorders.

[0057] Monitor the effectiveness of the model in actual clinical applications and continuously optimize and improve it based on feedback.

[0058] (8) Documentation and Communication

[0059] Record the details of data processing, model training, and evaluation throughout the process for later review and reproduction.

[0060] Design reasonable experimental plans and processes according to research objectives and questions, including controlled experiments, randomized controlled experiments, etc. Control variables and interference factors in the experiment to ensure the credibility and reliability of the experimental results.

[0061] In some other embodiments, during the data acquisition and preprocessing and database establishment stages, clinical examination text data can be collected simultaneously, and the clinical examination text data can be subjected to standardized text processing and standardized vector processing. On this basis, the method for identifying the functional state of the temporomandibular joint movement further includes the following steps: based on the preprocessed clinical examination text data, training and optimizing the pre-trained natural language processing model; when obtaining a multimodal model using a back-end fusion method, the trained natural language processing model, image segmentation model and image classification model are integrated to obtain a multimodal deep learning model, and training and optimization are performed.

[0062] Furthermore, the present embodiment also provides a device for identifying the functional state of the movement of the temporomandibular joint based on multimodal deep learning, comprising a data acquisition and preprocessing module, a single-modal model training module, a multimodal model fusion and training module, and a functional state recognition module. The data acquisition and preprocessing module is used to acquire and preprocess multimodal data samples, the multimodal data samples comprising temporomandibular joint image data and temporomandibular joint electronic face bow data, and the multimodal data samples are acquired from a pre-constructed temporomandibular joint movement functional state database; the single-modal model training module is used to train and optimize the pre-trained image segmentation model and image classification model according to the data category based on the pre-processed multimodal data samples; the multimodal model fusion and training module is used to use the back-end fusion method to fuse the trained image segmentation model and image classification model to obtain a multimodal deep learning model, and use the pre-processed multimodal data samples to train and optimize the multimodal deep learning model; the functional state recognition module is used to acquire the multimodal data to be recognized and input the trained multimodal deep learning model to obtain the corresponding recognition result of the functional state of the movement of the temporomandibular joint. The various modules of the device are used to implement the various steps of the aforementioned identification method, and the specific implementation process will not be repeated here.

[0063] Furthermore, this embodiment also provides an electronic device, including a memory, a processor, and a program stored in the memory. When the processor executes the program, each step of the aforementioned identification method is implemented, and the specific implementation process is not repeated here.

[0064] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A method for identifying the functional state of temporomandibular joint movement based on multimodal deep learning, characterized in that: The following steps are involved: Acquiring and preprocessing a multimodal data sample, wherein the multimodal data sample includes temporomandibular joint image data and temporomandibular joint electronic face bow data, and the multimodal data sample is acquired from a pre-constructed temporomandibular joint motion function state database; Based on the preprocessed multimodal data samples, the pre-trained image segmentation model and image classification model are trained and optimized according to the data categories; Using a back-end fusion method, the trained image segmentation model and the image classification model are fused to obtain a multimodal deep learning model, and the preprocessed multimodal data samples are used to train and optimize the multimodal deep learning model; The multimodal data to be identified is obtained and input into the trained multimodal deep learning model to obtain the corresponding temporomandibular joint motion function state recognition results.

2. The method for identifying the motion function state of the temporomandibular joint based on multimodal deep learning according to claim 1, characterized in that: The specific process of the preprocessing includes: denoising, labeling, classifying and normalizing the multimodal data samples.

3. The method for identifying the motion function state of the temporomandibular joint based on multimodal deep learning according to claim 1, characterized in that: The pre-trained image segmentation model and image classification model are trained and optimized according to the data category. The specific process includes: Using the preprocessed temporomandibular joint image data to train and optimize the pretrained image segmentation model, the temporomandibular joint image data includes temporomandibular joint nuclear magnetic resonance image data and temporomandibular joint cone beam CT image data; and, The pre-processed temporomandibular joint electronic facebow data was used to train and optimize the pre-trained image classification model.

4. The method for identifying the motion function state of the temporomandibular joint based on multimodal deep learning according to claim 1, characterized in that: The preprocessing includes labeling processing, and the training process of the multimodal deep learning model specifically includes: Use the annotated multimodal data samples as training samples; Dividing the training samples into a training set and a validation set; Based on the preset initial model parameters, the multimodal deep learning model is iteratively trained using the sample data in the training set and a gradient descent algorithm; The optimal parameters of the multimodal deep learning model are determined using the sample data in the validation set.

5. The method for identifying the motion function state of the temporomandibular joint based on multimodal deep learning according to claim 1, characterized in that: Using the backend fusion method, the trained image segmentation model and image classification model are fused to obtain a multimodal deep learning model. The specific process includes: Building a framework of the multimodal deep learning model, including at least one fusion layer and one optimization layer; Connecting the trained feature extraction layers of the image segmentation model and the image classification model to the fusion layer, wherein the connection method is one of weighted average, fully connected layer, and attention mechanism; The optimization layer is added after the fusion layer, and the optimization layer adopts one of a fully connected layer, a normalization layer, and an activation function.

6. The method for identifying the motion function state of the temporomandibular joint based on multimodal deep learning according to claim 5, characterized in that: The image data of the temporomandibular joint to be identified is obtained and input into the trained multimodal deep learning model to obtain the corresponding temporomandibular joint motion function state recognition results. The specific process includes: Preprocessing the input multimodal data to be identified; Based on the preprocessed temporomandibular joint image data, the trained image segmentation model is used to extract the image features of the key areas and structures of the temporomandibular joint; Based on the preprocessed electronic facebow data, the trained image classification model is used to extract the motion trajectory classification features of the temporomandibular joint; Inputting the image features and the motion trajectory classification features into the fusion layer, and outputting corresponding preliminary fusion features; The preliminary fusion features are input into the optimization layer, and the final fusion features are output.

7. The method for identifying the motion function state of the temporomandibular joint based on multimodal deep learning according to claim 1, characterized in that: The multimodal data sample also includes clinical examination text data.

8. The method for identifying the motion function state of the temporomandibular joint based on multimodal deep learning according to claim 7, characterized in that: The method for identifying the movement function state of the temporomandibular joint also includes the following steps: Based on the pre-processed clinical examination text data, train and optimize the pre-trained natural language processing model; Using the back-end fusion method, the trained natural language processing model, image segmentation model and image classification model are integrated to obtain a multimodal deep learning model, which is then trained and optimized.

9. A device for identifying the motion function state of the temporomandibular joint based on multimodal deep learning, characterized in that: It includes data acquisition and preprocessing module, single-modal model training module, multi-modal model fusion and training module, and functional state recognition module. The data acquisition and preprocessing module is used to acquire and preprocess multimodal data samples, wherein the multimodal data samples include temporomandibular joint image data and temporomandibular joint electronic face bow data, and the multimodal data samples are acquired from a pre-constructed temporomandibular joint motion function state database; The single-modal model training module is used to train and optimize the pre-trained image segmentation model and the image classification model according to the data category based on the pre-processed multi-modal data samples; The multimodal model fusion and training module is used to use a back-end fusion method to fuse the trained image segmentation model and image classification model to obtain a multimodal deep learning model, and use the pre-processed multimodal data samples to train and optimize the multimodal deep learning model; The functional state recognition module is used to obtain multimodal data to be recognized and input the trained multimodal deep learning model to obtain the corresponding temporomandibular joint movement functional state recognition results.

10. An electronic device comprising a memory, a processor, and a program stored in the memory, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

Images