Intelligent auxiliary evaluation method for temporomandibular joint based on multi-modal fusion

Abstract

The application discloses a temporomandibular joint intelligent auxiliary evaluation method based on multi-modal fusion, comprising the following steps: step 1: data acquisition and preprocessing; step 2: feature extraction; step 3: feature fusion and analysis; 3.1) constructing a vector field: constructing a vector field A(x) of joint motion based on motion tracking data F; 3.2) applying a comprehensive algorithm formula for in-depth analysis: fusing the results of steps 2 and 3, applying a comprehensive algorithm formula for in-depth analysis, obtaining a comprehensive feature description reflecting the functional state of the joint; the comprehensive algorithm formula is as follows: step 4: intelligent evaluation and diagnosis; step 5: model deployment and user interaction.

Description

Technical Field

[0001] This invention relates to the field of intelligent auxiliary assessment technology for the temporomandibular joint, and more specifically, to an intelligent auxiliary assessment method for the temporomandibular joint based on multimodal fusion. Background Technology

[0002] In the field of temporomandibular joint (TMJ) functional status assessment, traditional techniques primarily rely on medical imaging methods such as computed tomography (CT), magnetic resonance imaging (MRI), and X-ray imaging. These techniques can provide static views of the joint structure, but they have limitations in capturing dynamic movements and subtle changes in the joint. Furthermore, these methods typically cannot provide a comprehensive assessment of the joint's functional status, particularly regarding its biomechanical properties and microstructural changes.

[0003] On the other hand, the analysis of joint motion typically relies on physical examinations and patients' subjective descriptions, methods that lack precision and repeatability. While some advanced motion capture techniques have been introduced, they tend to be independent of imaging methods and lack the ability to fuse and synthesize data.

[0004] In summary, existing technologies have the following drawbacks:

[0005] 1. Limited dynamic analysis capabilities: Existing technologies mainly provide static images of joints, making it difficult to capture the dynamic changes of joints during actual movement.

[0006] 2. Lack of comprehensive assessment: Traditional methods often fail to take into account the joint's anatomical structure, biomechanical properties, and functional status, resulting in incomplete assessment results.

[0007] 3. Insufficient data fusion: In existing technologies, different types of data (such as image data and motion capture data) are usually processed separately, lacking effective data fusion and comprehensive analysis.

[0008] 4. Limitations of personalized diagnosis: Traditional assessment methods lack the ability to customize diagnoses based on individual differences, and cannot provide personalized treatment recommendations for different patients.

[0009] 5. Operational complexity and cost issues: Some advanced imaging techniques are complex to operate and expensive, which limits their popularization and application in clinical practice.

[0010] In conclusion, existing technologies have significant limitations in assessing the functional status of the temporomandibular joint, and more advanced technologies are needed to overcome these shortcomings and provide more accurate, comprehensive, and personalized assessment results. Summary of the Invention

[0011] 1. Technical problems to be solved

[0012] To address the problems existing in the prior art, the purpose of this invention is to provide an intelligent assisted assessment method for the temporomandibular joint based on multimodal fusion.

[0013] 2. Technical Solution

[0014] To solve the above problems, the present invention adopts the following technical solution:

[0015] A multimodal fusion-based intelligent assisted assessment method for the temporomandibular joint includes the following steps:

[0016] Step 1: Data Acquisition and Preprocessing

[0017] 1.1) Data Acquisition: Collect temporomandibular joint motion tracking data F, CBCT three-dimensional image I1, and MRI three-dimensional image I2;

[0018] 1.2) Data preprocessing: Perform necessary preprocessing on the collected data, including noise reduction, standardization, and formatting, to facilitate subsequent analysis;

[0019] Step 2: Feature Extraction

[0020] 2.1) Application of convolutional neural networks: The UNet network is used to segment the joint anatomical structures of I1 and I2 to obtain segmentation Masks M1 and M2;

[0021] 2.2) Feature Vector Extraction: Extract key feature vectors of the joints from M1 and M2.

[0022] Step 3: Feature Fusion and Analysis

[0023] 3.1) Constructing the vector field: Constructing the vector field A(x) of joint motion based on motion tracking data F;

[0024] 3.2) Deep analysis using comprehensive algorithm formulas: The results of steps 2 and 3 are combined, and comprehensive algorithm formulas are applied. A deep analysis is performed to obtain a comprehensive feature description that reflects the functional state of the joint; the comprehensive algorithm formula is as follows:

[0025]

[0026] Here, vector field A(x) represents the vector field of joint motion, where x is the three-dimensional position vector of the joint; divergence The divergence of the joint motion vector field is used to analyze the spatial variation of joint motion; Gaussian function. Used to simulate the effect of specific points inside a joint, where x i This indicates the location of a specific marker point, where λ is an adjustment parameter and α is an adjustment parameter. iIt is the attenuation parameter; the logic function. Used to process image data, among which σ represents the feature vector of the image data. j These are adjustment parameters; the differential geometric function Γ(x) is used to describe the curvature and shape changes of the joint surface;

[0027] Step 4: Intelligent Assessment and Diagnosis

[0028] 4.1) Building a deep learning model: Build an end-to-end deep learning model;

[0029] 4.2) Training and Evaluation: The deep learning model is trained using the output of step 3 as input data, and the model is evaluated and optimized.

[0030] 4.3) Output joint status score: The model outputs a functional status score of the temporomandibular joint;

[0031] Step 5: Model Deployment and User Interaction

[0032] 5.1) Model Deployment: Deploy the optimized model to cloud servers or edge computing devices;

[0033] 5.2) Develop user interface: Develop mobile and web applications to display the visualization results of joint movement and intelligent assessment and diagnostic reports.

[0034] The synergy between Step 1 and Step 2: Data acquisition and preprocessing (Step 1) provides high-quality input data, which is crucial for the success of feature extraction (Step 2); denoising, standardization, and formatting ensure the consistency and reliability of the data, providing optimized input for the convolutional neural network, thereby improving the accuracy of feature extraction.

[0035] The complementarity between Step 2 and Step 3: The segmentation mask and key feature vectors of the joint anatomy obtained in feature extraction (Step 2) provide the necessary input for feature fusion and analysis (Step 3); this detailed feature information enables the comprehensive algorithm formula to analyze the functional state of the joint more accurately.

[0036] The mutual reinforcement of steps 3 and 4: The comprehensive feature description generated by the comprehensive algorithm formula F(x) in feature fusion and analysis (step 3) provides a deep and comprehensive data view for intelligent assessment and diagnosis (step 4); this enables the deep learning model to learn and predict the functional state of the temporomandibular joint more effectively.

[0037] The connection between steps 4 and 5: After training and evaluation (step 4) to optimize the model, it is deployed to cloud servers or edge computing devices in model deployment and user interaction (step 5). This allows the evaluation results to be presented to end users through a user-friendly interface, improving the model's usability and acceptability; and enabling end users to intuitively understand the evaluation results. Model deployment ensures the accessibility and real-time nature of the evaluation.

[0038] In step 4, the deep learning model is represented as follows:

[0039]

[0040] Where, input x,t: x is a spatial position vector, representing the position of the temporomandibular joint in space; t is time, representing the moment of evaluation; From step 3, it contains feature information about the temporomandibular joint; weight matrix W k Each W k It is a matrix used to transform input features at the k-th layer; bias vector b k : Each b k It is a vector used to add bias at the k-th layer; the weight coefficient α k : Each α k is a scalar used to adjust the output contribution of the k-th layer; the number of layers K: the number of layers in the network, which determines the depth of the model; the activation function tanh is the hyperbolic tangent function, used to introduce nonlinearity; softmax is the normalized exponential function, used in the output layer to convert the output into a probability distribution;

[0041] The specific calculation process is as follows:

[0042] Feature transformation: For each layer k, use the weight matrix W k and bias vector b k Transform input features;

[0043] Nonlinear activation: Applying the tanh function to increase the nonlinearity of the model;

[0044] Weighted summation: using weighting coefficient α k Weight the outputs of each layer, then sum them.

[0045] Normalized output: The output is normalized using the softmax function to obtain the functional status score of the temporomandibular joint.

[0046] Specifically, step 1: data acquisition and preprocessing

[0047] 1.1) Data Acquisition

[0048] Temporomandibular joint motion tracking data F: High-density marker data of joint motion are collected. This data is obtained through a motion capture system and contains spatial and temporal information of joint motion.

[0049] CBCT 3D Images I1: Using cone-beam computed tomography (CBCT) technology to acquire 3D structural images of joints, providing information on the bone structure of the joints;

[0050] MRI 3D Image I2: Images of the soft tissue structure of a joint are obtained through magnetic resonance imaging (MRI) technology, providing information about the soft tissue of the joint;

[0051] 1.2) Data Preprocessing

[0052] Denoising: Using filtering algorithms to remove noise from an image;

[0053] The following algorithm formula is used: Where G is a Gaussian filter, σ is the standard deviation, and x and y are pixel positions;

[0054] Standardization: Standardizing image data to a certain range or distribution to reduce differences between different devices, using the following algorithm formula: Where I′ is the standardized image, I is the original image, and μ and σ are the mean and standard deviation of the image, respectively;

[0055] Formatting: Converting data into a format suitable for subsequent processing, and converting images to a specific resolution and color space; using the following formula: I″=resize(I′,w,h), where I″ is the formatted image, and w and h are the width and height of the target resolution, making it suitable for subsequent analysis and processing;

[0056] The data will be converted into a unified digital format. The preprocessed data will be used to construct a three-dimensional model of the temporomandibular joint, analyze the joint's motion characteristics, and serve as input for a deep learning model.

[0057] In 2.1), the basic components of the UNet network structure include two parts: the shrinking path and the expanding path;

[0058] The shrinkage path consists of multiple convolutional and pooling layers, which gradually reduce the spatial dimension of the image and increase the number of feature channels.

[0059] The extended path gradually restores the spatial dimension of the image through upsampling and convolution operations, while reducing the number of feature channels;

[0060] Input layer: Receives preprocessed images I1 and I2;

[0061] Convolutional layers: Multiple convolutional layers are used to extract features from an image; each convolutional layer includes a convolution operation, an activation function, and batch normalization;

[0062] Pooling layer: Apply max pooling operation to reduce the spatial dimensionality of the image;

[0063] Upsampling layer: Uses transpose convolution to restore the spatial dimensions of the image;

[0064] Skip connections combine feature maps from the contracted path with those from the expanded path to recover local information of the image.

[0065] Output layer: Generates segmented Masks M1 and M2, corresponding to the joint anatomy in CBCT and MRI images;

[0066] Each of the convolutional layers contains 64, 128, 256, 512, and 1024 convolutional kernels, with a kernel size typically of 3x3;

[0067] The pooling layer uses 2x2 max pooling;

[0068] The upsampling layer uses a 2x2 transposed convolution kernel for upsampling;

[0069] The training process is as follows:

[0070] Loss function: Use the cross-entropy loss function;

[0071] Optimizer: Use Adam or SGD to adjust network weights;

[0072] Training data: Supervised learning using labeled medical images;

[0073] Number of iterations: Adjusted based on dataset size and model performance.

[0074] In 2.2), the following steps are involved.

[0075] Edge detection:

[0076] Applying the Sobel operator:

[0077] Texture analysis:

[0078] Calculate the gradient magnitude:

[0079] GLCM calculation: Analyze the spatial relationships between pixels in an image; select a specific direction and distance, traverse the image, and for each pixel, calculate its co-occurrence relationship with its neighboring pixels at the specified direction and distance;

[0080] Extracting texture features:

[0081] Contrast: Contrast = ∑ i,j (ij) 2 Ci,j ;

[0082] Uniformity:

[0083] Entropy: Entropy = -∑ i,j C i,j log(C i,j );

[0084] Shape descriptor extraction, application of edge detection algorithms to identify contours in images, and calculation of geometric features:

[0085] Perimeter: The total length of the outline.

[0086] Area: Calculates the total number of pixels inside the outline;

[0087] Shape factor:

[0088] The vector field A(x) defines a vector at each point x, representing the displacement or velocity at that point. The specific steps are as follows:

[0089] Data preprocessing: Converting motion tracking data F into a format suitable for analysis;

[0090] Calculate the displacement vector: For each time point, calculate the displacement of each part of the joint;

[0091] Constructing a vector field: Mapping displacement vectors to their corresponding positions in space to form a vector field; the vector field A(x) is represented as a numerical array or matrix; it is used to analyze the motion characteristics of joints and provides a foundation for subsequent feature fusion.

[0092] The calculation process of the comprehensive algorithm formula is as follows:

[0093] Vector field calculation: First, calculate the vector field A(x) of the joint motion;

[0094] Divergence calculation: Calculate the divergence of a vector field to analyze the spatial changes in joint motion;

[0095] Application of Gaussian function: Applying the Gaussian function to simulate the influence of specific points inside a joint;

[0096] Logical function processing: Processing image data and extracting features;

[0097] Differential geometric analysis: analyzing the curvature and shape changes of joint surfaces;

[0098] Overall score: The above calculation results are combined to obtain a score of joint function status.

[0099] 5.1) The specific steps for model deployment are as follows:

[0100] Model optimization: Compress and optimize the trained model to adapt to the resource constraints of the deployment environment;

[0101] Model conversion: Converting the model into a format suitable for a specific deployment environment.

[0102] Deployment: Deploy the model to cloud servers or edge computing devices;

[0103] Performance monitoring: Monitor the performance and stability of the model to ensure its effective operation in the production environment;

[0104] 5.2) The specific steps for developing the user interface are as follows:

[0105] Interface Design: Design an intuitive and user-friendly interface to display the visualization results of joint movements;

[0106] Functionality Implementation: Develop a functional module for displaying diagnostic reports and scores;

[0107] Interaction design: Implementing functions for user interaction with the system;

[0108] Mobile and Web Development: Develop applications using the appropriate technology stack.

[0109] The hyperbolic tangent function tanh(x) is defined as:

[0110]

[0111] Where e is the base of the natural logarithm; the tanh function is used to process the intermediate layer output in the deep learning model, increasing the model's ability to handle complex joint data; in neural networks, nonlinear functions such as tanh can help the network learn complex data patterns; the output range of the tanh function is [-1, 1], which helps to normalize the data and keep it stable during training.

[0112] The softmax function for a vector z is defined as follows:

[0113]

[0114] Among them, z i Let be the i-th element in vector z, and K be the length of the vector. The softmax function converts the output into a probability distribution, where each element's value is between 0 and 1, and the sum is 1. In multi-class classification problems, the softmax function is often used in the output layer to provide probability predictions for each class. In this case, the softmax function is used in the model's output layer to convert the deep learning model's output into a probability distribution of temporomandibular joint functional status, thus providing doctors with intuitive diagnostic information.

[0115] 3. Beneficial effects

[0116] Compared with the prior art, the advantages of this invention are:

[0117] 1. Multimodal data analysis: Combining motion tracking data, CBCT and MRI images, the algorithm can provide a multi-dimensional assessment of joint function, helping doctors understand the health status of joints from different perspectives; combining kinematic, imaging and geometric multi-dimensional analysis provides a more comprehensive assessment of joint functional status; through multimodal data fusion and advanced feature extraction technology, it can more comprehensively understand the state of joints, thereby improving the accuracy of diagnosis.

[0118] 2. Dynamic assessment and real-time feedback: Supports real-time analysis of joint movement, providing doctors and patients with immediate assessment results and feedback; the model is deployed to cloud servers or edge computing devices, enabling assessments to be conducted in real time and providing immediate feedback to doctors and patients.

[0119] 3. Personalized medical decision support: By adjusting parameters, highly customized assessments can be performed for different patients' specific conditions; the algorithm's high degree of customization makes it a powerful tool for personalized medical decision-making, especially when dealing with complex or rare cases.

[0120] 4. Data fusion: It can effectively fuse data from different sources (such as motion tracking, CBCT, MRI) to improve the accuracy of the assessment.

[0121] 5. Enhanced user experience: The user-friendly interface makes it easy for non-professional users to understand complex diagnostic information, improving the efficiency and quality of doctor-patient communication.

[0122] 6. Facilitating telemedicine services: Cloud deployment enables remote access to the system, promoting the development of telemedicine services, especially in resource-limited areas.

[0123] 7. Improved Efficiency of Medical Resource Utilization: Automated and intelligent diagnostic processes reduce the workload of doctors and improve the efficiency of medical resource utilization. Through the close collaboration and complementarity of these steps, this invention not only improves the accuracy and efficiency of temporomandibular joint functional status assessment, but also provides patients and doctors with a more convenient and intuitive diagnostic experience, while promoting the innovation and development of medical technology. Attached Figure Description

[0124] Figure 1 The flowchart shows the intelligent assisted assessment method for the temporomandibular joint based on multimodal fusion according to the present invention.

[0125] Figure 2 This is a schematic diagram illustrating the calculation process of the comprehensive algorithm formula of this invention.

[0126] Figure 3 This is a schematic diagram illustrating the computational process of the deep learning model of this invention. Detailed Implementation

[0127] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0128] In the description of this invention, it should be noted that the terms "upper," "lower," "inner," "outer," and "top / bottom" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0129] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installed," "equipped with," "sleeved / connected," and "connected" should be interpreted broadly. For example, "connected" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be a connection within two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0130] Example 1

[0131] Please refer to Figure 1-3 A multimodal fusion-based intelligent assisted assessment method for the temporomandibular joint includes the following steps:

[0132] Step 1: Data Acquisition and Preprocessing

[0133] 1.1) Data Acquisition: Collect temporomandibular joint motion tracking data F, CBCT three-dimensional image I1, and MRI three-dimensional image I2;

[0134] 1.2) Data preprocessing: Perform necessary preprocessing on the collected data, including noise reduction, standardization, and formatting, to facilitate subsequent analysis;

[0135] Step 2: Feature Extraction

[0136] 2.1) Application of convolutional neural network: Use UNet network to segment the joint anatomical structure of I1 and I2 to obtain segmentation masks M1 and M2;

[0137] 2.2) Feature Vector Extraction: Extract key feature vectors of the joints from M1 and M2.

[0138] Step 3: Feature Fusion and Analysis

[0139] 3.1) Constructing the vector field: Constructing the vector field A(x) of joint motion based on motion tracking data F;

[0140] 3.2) Deep analysis using comprehensive algorithm formulas: The results of steps 2 and 3 are combined, and comprehensive algorithm formulas are applied. A deep analysis is performed to obtain a comprehensive feature description that reflects the functional state of the joint; the comprehensive algorithm formula is as follows:

[0141]

[0142] Here, vector field A(x) represents the vector field of joint motion, where x is the three-dimensional position vector of the joint; divergence The divergence of the joint motion vector field is used to analyze the spatial variation of joint motion; Gaussian function. Used to simulate the effect of specific points inside a joint, where x i This indicates the location of a specific marker point, where λ is an adjustment parameter and α is an adjustment parameter. i It is the attenuation parameter; the logic function. Used to process image data, among which σ represents the feature vector of the image data. j These are adjustment parameters; the differential geometric function Γ(x) is used to describe the curvature and shape changes of the joint surface;

[0143] pass The divergence term of the vector field allows for in-depth analysis of the biomechanical characteristics of joint movement, such as pressure distribution and fluid dynamics within the joint. In assessing temporomandibular joint disorder symptoms, this algorithm can help identify abnormal pressure points within the joint, thereby guiding treatment planning. Through Gaussian functions... In this application, the algorithm can perform personalized analysis on specific areas of a joint; for joint pain problems in specific patients, the algorithm can accurately locate the source of pain and provide targeted treatment suggestions; logical functions This enables the algorithm to effectively process and integrate complex imaging data; when processing multimodal imaging data, such as combining CBCT and MRI data, the algorithm can provide a comprehensive perspective to help doctors understand the condition of the joint more fully; the Γ(x) term provides a detailed description of the curvature and shape changes of the joint surface; when assessing joint degeneration or injury, the algorithm can provide detailed information on joint morphological changes, which is helpful for early diagnosis and treatment.

[0144] Multimodal data analysis: By combining motion tracking data, CBCT and MRI images, the algorithm can provide a multi-dimensional assessment of joint function, helping doctors understand the health status of joints from different perspectives; by combining kinematic, imaging and geometric multi-dimensional analysis, it provides a more comprehensive assessment of joint function status.

[0145] Dynamic assessment and real-time feedback: The algorithm supports real-time analysis of joint movement, providing doctors and patients with immediate assessment results and feedback.

[0146] Personalized medical decision support: By adjusting parameters, highly customized assessments can be performed for different patients' specific conditions; the algorithm's high degree of customization makes it a powerful tool for personalized medical decision-making, especially when dealing with complex or rare cases.

[0147] Data fusion: It can effectively fuse data from different sources (such as motion tracking, CBCT, MRI) to improve the accuracy of assessment.

[0148] Step 4: Intelligent Assessment and Diagnosis

[0149] 4.1) Building a deep learning model: Build an end-to-end deep learning model;

[0150] 4.2) Training and Evaluation: The deep learning model is trained using the output of step 3 as input data, and the model is evaluated and optimized.

[0151] 4.3) Output joint status score: The model outputs a functional status score of the temporomandibular joint;

[0152] Step 5: Model Deployment and User Interaction

[0153] 5.1) Model Deployment: Deploy the optimized model to cloud servers or edge computing devices;

[0154] 5.2) Develop user interface: Develop mobile and web applications to display the visualization results of joint movement and intelligent assessment and diagnostic reports.

[0155] In step 4, the deep learning model is represented as follows:

[0156]

[0157] Where, input x,t: x is a spatial position vector, representing the position of the temporomandibular joint in space; t is time, representing the moment of evaluation; From step 3, it contains feature information about the temporomandibular joint; weight matrix W k Each W kIt is a matrix used to transform input features at the k-th layer; bias vector b k : Each b k It is a vector used to add bias at the k-th layer; the weight coefficient α k : Each α k is a scalar used to adjust the output contribution of the k-th layer; layer number K: the number of layers in the network, which determines the depth of the model; activation function: tanh is the hyperbolic tangent function, used to introduce non-linearity; softmax: is the normalized exponential function, used in the output layer to convert the output into a probability distribution;

[0158] Please refer to Figure 3 The specific calculation process of the deep learning model is as follows:

[0159] Feature transformation: For each layer k, use the weight matrix W k and bias vector b k Transform input features;

[0160] Nonlinear activation: Applying the tanh function to increase the nonlinearity of the model;

[0161] Weighted summation: using weighting coefficient α k Weight the outputs of each layer, then sum them.

[0162] Normalized output: The output is normalized using the softmax function to obtain the functional status score of the temporomandibular joint.

[0163] The hyperbolic tangent function tanh(x) is defined as:

[0164]

[0165] Where e is the base of the natural logarithm; the tanh function is used to process the intermediate layer output in the deep learning model, increasing the model's ability to handle complex joint data; in neural networks, nonlinear functions such as tanh can help the network learn complex data patterns; the output range of the tanh function is [-1, 1], which helps to normalize the data and keep it stable during training.

[0166] The softmax function for a vector z is defined as follows:

[0167]

[0168] Among them, z iLet be the i-th element in vector z, and K be the length of the vector. The softmax function converts the output into a probability distribution, where each element's value is between 0 and 1, and the sum is 1. In multi-class classification problems, the softmax function is often used in the output layer to provide probability predictions for each class. In this case, the softmax function is used in the model's output layer to convert the deep learning model's output into a probability distribution of temporomandibular joint functional status, thus providing doctors with intuitive diagnostic information.

[0169] Specifically, step 1: data acquisition and preprocessing

[0170] 1.1) Data Acquisition

[0171] Temporomandibular joint motion tracking data F: High-density marker data of joint motion are collected. This data is obtained through a motion capture system and contains spatial and temporal information of joint motion.

[0172] CBCT 3D Images I1: Using cone-beam computed tomography (CBCT) technology to acquire 3D structural images of joints, providing information on the bone structure of the joints;

[0173] MRI 3D Image I2: Images of the soft tissue structure of a joint are obtained through magnetic resonance imaging (MRI) technology, providing information about the soft tissue of the joint;

[0174] 1.2) Data Preprocessing

[0175] Denoising: Using filtering algorithms to remove noise from an image;

[0176] The following algorithm formula is used: Where G is a Gaussian filter, σ is the standard deviation, and x and y are pixel positions;

[0177] Standardization: Standardizing image data to a certain range or distribution to reduce differences between different devices, using the following algorithm formula: Where I′ is the standardized image, I is the original image, and μ and σ are the mean and standard deviation of the image, respectively;

[0178] Formatting: The data is converted into a format suitable for subsequent processing, and the image is converted into a specific resolution and color space. The following formula is used: I″ = resize(I′, w, h), where I″ is the formatted image, and w and h are the width and height of the target resolution, making it suitable for subsequent analysis and processing. The data will be converted into a unified digital format. The preprocessed data will be used to construct a 3D model of the temporomandibular joint, analyze the joint's motion characteristics, and serve as input for a deep learning model.

[0179] In 2.1), the basic components of the UNet network structure include two parts: the shrinking path and the expanding path;

[0180] The shrinkage path consists of multiple convolutional and pooling layers, which gradually reduce the spatial dimension of the image and increase the number of feature channels.

[0181] The extended path gradually restores the spatial dimension of the image through upsampling and convolution operations, while reducing the number of feature channels;

[0182] Input layer: Receives preprocessed images I1 and I2;

[0183] Convolutional layers: Multiple convolutional layers are used to extract features from an image; each convolutional layer includes a convolution operation, an activation function, and batch normalization;

[0184] Pooling layer: Apply max pooling operation to reduce the spatial dimensionality of the image;

[0185] Upsampling layer: Uses transpose convolution to restore the spatial dimensions of the image;

[0186] Skip connections combine feature maps from the contracted path with those from the expanded path to recover local information of the image.

[0187] Output layer: Generates segmented Masks M1 and M2, corresponding to the joint anatomy in CBCT and MRI images;

[0188] Each of the convolutional layers contains 64, 128, 256, 512, and 1024 convolutional kernels, with a kernel size typically of 3x3;

[0189] The pooling layer uses 2x2 max pooling;

[0190] The upsampling layer uses a 2x2 transposed convolution kernel for upsampling;

[0191] The training process is as follows:

[0192] Loss function: Use the cross-entropy loss function;

[0193] Optimizer: Use Adam or SGD to adjust network weights;

[0194] Training data: Supervised learning using labeled medical images;

[0195] Number of iterations: Adjusted based on dataset size and model performance.

[0196] In 2.2), the following steps are involved.

[0197] Edge detection:

[0198] Applying the Sobel operator:

[0199] Texture analysis:

[0200] Calculate the gradient magnitude:

[0201] GLCM calculation: Analyze the spatial relationships between pixels in an image; select a specific direction and distance, traverse the image, and for each pixel, calculate its co-occurrence relationship with its neighboring pixels at the specified direction and distance;

[0202] Extracting texture features:

[0203] Contrast: Contrast = ∑ i,j (ij) 2 C i,j ;

[0204] Uniformity:

[0205] Entropy: Entropy = -∑ i,j C i,j log(C i,j );

[0206] Shape descriptor extraction, application of edge detection algorithms to identify contours in images, and calculation of geometric features:

[0207] Perimeter: The total length of the outline.

[0208] Area: Calculates the total number of pixels inside the outline;

[0209] Shape factor:

[0210] The vector field A(x) defines a vector at each point x, representing the displacement or velocity at that point. The specific steps are as follows:

[0211] Data preprocessing: Converting motion tracking data F into a format suitable for analysis;

[0212] Calculate the displacement vector: For each time point, calculate the displacement of each part of the joint;

[0213] Constructing a vector field: Mapping displacement vectors to corresponding positions in space to form a vector field; the vector field A(x) is represented as a numerical array or matrix; it is used to analyze the motion characteristics of joints and provides a foundation for subsequent feature fusion.

[0214] Please refer to Figure 2 The calculation process of the comprehensive algorithm formula is as follows:

[0215] Vector field calculation: First, calculate the vector field A(x) of the joint motion;

[0216] Divergence calculation: Calculate the divergence of a vector field to analyze the spatial changes in joint motion;

[0217] Application of Gaussian function: Applying the Gaussian function to simulate the influence of specific points inside a joint;

[0218] Logical function processing: Processing image data and extracting features;

[0219] Differential geometric analysis: analyzing the curvature and shape changes of joint surfaces;

[0220] Overall score: The above calculation results are combined to obtain a score of joint function status.

[0221] 5.1) The specific steps for model deployment are as follows:

[0222] Model optimization: Compress and optimize the trained model to adapt to the resource constraints of the deployment environment;

[0223] Model conversion: Converting a model into a format suitable for a specific deployment environment;

[0224] Deployment: Deploy the model to cloud servers or edge computing devices;

[0225] Performance monitoring: Monitor the performance and stability of the model to ensure its effective operation in the production environment;

[0226] 5.2) The specific steps for developing the user interface are as follows:

[0227] Interface design: Design an intuitive and user-friendly interface to display the visualization results of joint movements;

[0228] Functionality Implementation: Develop a functional module for displaying diagnostic reports and scores;

[0229] Interaction design: Implementing functions for user interaction with the system;

[0230] Mobile and Web Development: Develop applications using the appropriate technology stack.

[0231] The test was conducted using a multimodal medical imaging dataset, including standard temporomandibular joint motion data, CBCT, and MRI images.

[0232] Cross-validation was used to evaluate the model's precision, recall, and F1 score. Precision of 90%-95% indicates high accuracy in diagnosing temporomandibular joint (TMJ) functional status; recall of 86%-92% indicates the model's ability to identify TMJ-related cases; F1, the harmonic mean of precision and recall, measures the model's overall performance and is 88%-92%; response time, the time from input data to model generating evaluation results, is 1-3 seconds; and model size, the storage size for deploying the model, affects loading and running speed and is 100MB-500MB.

[0233] The above description is merely a preferred embodiment of the present invention; however, the scope of protection of the present invention is not limited thereto; any substitutions or modifications made by those skilled in the art within the technical scope disclosed in the present invention, based on the technical solution and its improved concept, should be covered within the scope of protection of the present invention.

Claims

1. A multimodal fusion-based intelligent assisted assessment method for the temporomandibular joint, characterized by: Includes the following steps: Step 1: Data Acquisition and Preprocessing 1.1) Data Collection: Collect temporomandibular joint movement tracking data. CBCT 3D images and MRI three-dimensional images ; 1.2) Data preprocessing: Perform necessary preprocessing on the collected data, including noise reduction, standardization, and formatting, to facilitate subsequent analysis; Step 2: Feature Extraction 2.1) Applying Convolutional Neural Networks: Using the UNet network to... and The joint anatomical structure is segmented to obtain a segmentation mask. and ; 2.2) Feature vector extraction: from and Extracting key feature vectors of joints ; Step 3: Feature Fusion and Analysis 3.1) Constructing a vector field: based on motion tracking data Constructing the vector field of joint motion ; 3.2) Deep analysis using comprehensive algorithm formulas: The results of steps 2 and 3 are combined, and comprehensive algorithm formulas are applied. A deep analysis is performed to obtain a comprehensive feature description that reflects the functional state of the joint; the comprehensive algorithm formula is as follows: （1） Among them, vector field The vector field representing joint motion, where x is the three-dimensional position vector of the joint; divergence The divergence of the joint motion vector field is used to analyze the spatial variation of joint motion; Gaussian function. Used to simulate the effects on specific points inside a joint, Indicates the location of a specific marker point. It is the attenuation parameter; the logic function. Used to process image data, among which The feature vector representing the image data, It refers to adjusting parameters; Used to describe the curvature and shape changes of joint surfaces; Step 4: Intelligent Assessment and Diagnosis 4.1) Building a deep learning model: Build an end-to-end deep learning model; 4.2) Training and Evaluation: The deep learning model is trained using the output of step 3 as input data, and the model is evaluated and optimized. 4.3) Output joint status score: The model outputs a functional status score of the temporomandibular joint; Step 5: Model Deployment and User Interaction 5.1) Model Deployment: Deploy the optimized model to cloud servers or edge computing devices; 5.2) Develop user interface: Develop mobile and web applications to display the visualization results of joint movement and intelligent assessment and diagnostic reports.

2. The intelligent assisted assessment method for the temporomandibular joint based on multimodal fusion according to claim 1, characterized in that: In step 4, the deep learning model is represented as follows: （2）， Input x, , where x is a spatial position vector, representing the position of the temporomandibular joint in space; The time period indicates the moment of evaluation; From step 3, containing feature information of the temporomandibular joint; weight matrix Each It is a matrix used in the first... Layer transformation input features; bias vector Each It is a vector used in the first... Add bias to the layer; weight coefficient Each It is a scalar used to regulate the first Layer output contribution; number of layers The number of layers in the network determines the depth of the model; activation functions: tanh is the hyperbolic tangent function, used to introduce non-linearity; softmax is the normalized exponential function, used in the output layer to convert the output into a probability distribution; The specific calculation process is as follows: Feature transformation: for each layer Using weight matrix and bias vector Transform input features; Nonlinear activation: Applying the tanh function to increase the nonlinearity of the model; Weighted summation: using weighting coefficients Weight the outputs of each layer, then sum them. Normalized output: The output is normalized using the softmax function to obtain the functional status score of the temporomandibular joint.

3. The intelligent assisted assessment method for the temporomandibular joint based on multimodal fusion according to claim 1, characterized in that: Step 1: Data Acquisition and Preprocessing 1.1) Data Acquisition Temporomandibular joint motion tracking data High-density marker data of joint movement is collected. This data is obtained through a motion capture system and contains spatial and temporal information about joint movement. CBCT 3D Images Cone-beam computed tomography (CBCT) is used to obtain three-dimensional images of joints, providing information on the bone structure of the joints. MRI 3D images : Obtaining images of the soft tissue structure of joints using magnetic resonance imaging (MRI) technology provides information about the soft tissue of the joints; 1.2) Data Preprocessing Denoising: Using filtering algorithms to remove noise from an image; The following algorithm formula is used: ,in It is a Gaussian filter. It is the standard deviation. It is the pixel position; Standardization: Standardizing image data to a certain range or distribution to reduce differences between different devices, using the following algorithm formula: ,in It is a standardized image. It is the original image. and These are the mean and standard deviation of the image, respectively. Formatting: Converting data into a format suitable for subsequent processing, and converting images to a specific resolution and color space; using the following formula: and The target resolution is determined by its width and height to make it suitable for subsequent analysis and processing. The data will be converted into a unified digital format, and the preprocessed data will be used to construct a three-dimensional model of the temporomandibular joint, analyze the joint's motion characteristics, and serve as input for a deep learning model.

4. The intelligent assisted assessment method for the temporomandibular joint based on multimodal fusion according to claim 1, characterized in that: In 2.1), the basic components of the UNet network structure include two parts: the shrinking path and the expanding path; The shrinkage path consists of multiple convolutional and pooling layers, which gradually reduce the spatial dimension of the image and increase the number of feature channels. The extended path gradually restores the spatial dimension of the image through upsampling and convolution operations, while reducing the number of feature channels; Input layer: Receives the preprocessed image. and ; Convolutional layers: Multiple convolutional layers are used to extract features from an image; each convolutional layer includes a convolution operation, an activation function, and batch normalization; Pooling layer: Apply max pooling operation to reduce the spatial dimensionality of the image; Upsampling layer: Uses transpose convolution to restore the spatial dimensions of the image; Skip connections combine feature maps from the contracted path with those from the expanded path to recover local information of the image; Output layer: Generates segmentation masks and , corresponding to the joint anatomy in CBCT and MRI images; Each of the convolutional layers contains 64, 128, 256, 512, and 1024 convolutional kernels, with a kernel size typically of 3x3; The pooling layer uses 2x2 max pooling; The upsampling layer uses a 2x2 transposed convolution kernel for upsampling; The training process is as follows: Loss function: Use the cross-entropy loss function; Optimizer: Use Adam or SGD to adjust network weights; Training data: Supervised learning using labeled medical images; Number of iterations: Adjusted based on dataset size and model performance.

5. The intelligent assisted assessment method for the temporomandibular joint based on multimodal fusion according to claim 1, characterized in that: In 2.2), the following steps are involved. Edge detection: Applying the Sobel operator: ; Texture analysis: Calculate gradient magnitude: Mag ; GLCM calculation: Analyze the spatial relationships between pixels in an image; select a specific direction and distance, traverse the image, and for each pixel, calculate its co-occurrence relationship with its neighboring pixels at the specified direction and distance; Extract texture feature: Contrast: ; Homogeneity ; entropy: ; Shape descriptor extraction, application of edge detection algorithms to identify contours in images, and calculation of geometric features: Perimeter: The total length of the outline. Area: Calculates the total number of pixels inside the outline; Shape Factor .

6. The intelligent assisted assessment method for the temporomandibular joint based on multimodal fusion according to claim 1, characterized in that: The vector field A vector is defined at each point x to represent the displacement or velocity at that point. The specific steps are as follows: Data preprocessing: processing motion tracking data Convert to a format suitable for analysis; Calculate the displacement vector: For each time point, calculate the displacement of each part of the joint; Constructing a vector field: Mapping displacement vectors to their corresponding positions in space forms a vector field; vector field It is represented as a numerical array or matrix; it is used to analyze the motion characteristics of joints and provide a basis for subsequent feature fusion.

7. The intelligent assisted assessment method for the temporomandibular joint based on multimodal fusion according to claim 1, characterized in that: The calculation process of the comprehensive algorithm formula is as follows: Vector field calculation: First, calculate the vector field of joint motion. ; Divergence calculation: Calculate the divergence of a vector field to analyze the spatial changes in joint motion; Application of Gaussian function: Applying the Gaussian function to simulate the influence of specific points inside a joint; Logical function processing: Processing image data and extracting features; Differential geometric analysis: analyzing the curvature and shape changes of joint surfaces; Overall score: The above calculation results are combined to obtain a score of joint function status.

8. The intelligent assisted assessment method for the temporomandibular joint based on multimodal fusion according to claim 7, characterized in that: 5.1) The specific steps for model deployment are as follows: Model optimization: Compress and optimize the trained model to adapt to the resource constraints of the deployment environment; Model conversion: Converting a model into a format suitable for a specific deployment environment; Deployment: Deploy the model to cloud servers or edge computing devices; Performance monitoring: Monitor the performance and stability of the model to ensure its effective operation in the production environment; 5.2) The specific steps for developing the user interface are as follows: Interface Design: Design an intuitive and user-friendly interface to display the visualization results of joint movements; Functionality Implementation: Develop a functional module for displaying diagnostic reports and scores; Interaction design: Implementing functions for user interaction with the system; Mobile and Web Development: Develop applications using the appropriate technology stack.

9. The intelligent assisted assessment method for the temporomandibular joint based on multimodal fusion according to claim 2, characterized in that: hyperbolic tangent function Defined as: ， in, It is the base of the natural logarithm; the tanh function is used to process the intermediate layer output in deep learning models, increasing the model's ability to process complex joint data; The softmax function for a vector z is defined as follows: ， in, It is the first in vector z One element, is the length of the vector. The softmax function is used in the output layer of the model to convert the output of the deep learning model into a probability distribution of the temporomandibular joint functional state, thereby providing doctors with intuitive diagnostic information.

Images