An intelligent detection method for interproximal caries in multi-modal imaging

Abstract

The application provides a kind of multi-modal imaging interproximal caries intelligent detection method, comprising the following steps: collecting the SWIR transmission image of tooth sample, collecting the corresponding OCT three-dimensional data, pre-processing data, constructing SWIR-OCT multi-modal dataset, inputting the dataset into DMRA-Net model for training and prediction, visualizing analysis and display of model discrimination basis, realizing the interpretable identification of tooth health status and pathological state.The application can effectively fuse SWIR and OCT multi-modal image information, realize the automatic, precise and interpretable detection of interproximal caries, realize the automatic identification of tooth health status and pathological state, and has good application value.

Description

Technical Field

[0001] This invention relates to the field of medical image processing and intelligent diagnostic technology, specifically a multimodal imaging intelligent detection method for proximal caries. Background Technology

[0002] Proximal caries, or caries between adjacent teeth, is often difficult to detect accurately in its early stages due to its hidden location and the inherent subjectivity of traditional visual and probe examinations. Current clinical diagnosis still primarily relies on traditional methods such as visual examination by dentists, probe examination, and X-ray imaging. However, these methods have significant limitations in practical application: visual examination and probe examination are highly dependent on the operator's experience and are subject to considerable subjectivity, making it difficult to accurately identify early caries lesions and occult lesions; while X-ray imaging can be used to detect proximal caries, it is not sensitive to early demineralization changes and carries the risk of ionizing radiation, making it unsuitable for high-frequency screening and continuous monitoring.

[0003] To overcome the limitations of traditional methods, non-invasive optical imaging techniques have gradually gained attention. Short-wavelength infrared (SWIR) transmission imaging, within the 1300–1700 nm wavelength range, can clearly distinguish between diseased and healthy dental tissues, exhibiting high sensitivity in detecting dental demineralization and occult lesions. Furthermore, the imaging process does not generate ionizing radiation, ensuring high safety. Optical coherence tomography (OCT), as a high-resolution, non-invasive tomographic imaging technique, can clearly present the layered structure and internal changes of enamel and dentin at micrometer-level resolution, offering significant advantages in assessing early lesions and caries depth within teeth.

[0004] However, SWIR transmission imaging and OCT imaging have different focuses in terms of imaging mechanisms and information representation. SWIR transmission imaging is based on the changes in the absorption and scattering characteristics of short-wave infrared light by the tooth, and is suitable for reflecting the differences in overall transmission characteristics caused by lesions; OCT imaging, on the other hand, obtains tomographic information of the tooth's internal structure, achieving a fine representation of local structural changes and lesion layers. The two have good complementarity at the information level. Existing studies mostly use a single imaging modality for dental lesion analysis. This single imaging modality cannot simultaneously take into account both overall transmission characteristics and local microstructural information, and does not fully utilize the complementary information between different imaging modalities, resulting in shortcomings in the automatic identification and accurate diagnosis of complex lesions, especially proximal caries. Existing multimodal imaging lacks effective methods for data pairing and feature fusion, affecting model stability and generalization ability.

[0005] Therefore, there is an urgent need for an intelligent detection method that can effectively integrate multimodal information from SWIR transmission and OCT imaging to achieve automatic identification and discrimination of dental health and disease status, thereby improving the accuracy and reliability of early detection of dental lesions such as proximal caries. Summary of the Invention

[0006] The purpose of this invention is to provide a multimodal imaging-based intelligent detection method for proximal caries, in order to solve the problems in existing methods, such as the difficulty of simultaneously taking into account the overall transmission characteristics and local microstructure information in a single imaging modality, and the lack of effective methods for pairing and feature fusion of multimodal image data.

[0007] This invention is implemented as follows: a multimodal imaging-based intelligent detection method for proximal caries, comprising the following steps:

[0008] S1: Construct a tooth image acquisition system based on SWIR transmission imaging and acquire SWIR transmission images;

[0009] S2: Acquire OCT three-dimensional data corresponding to the SWIR transmission image;

[0010] S3: Preprocess the SWIR transmission image and the OCT data respectively;

[0011] S4: Construct a SWIR–OCT multimodal dataset based on the preprocessed SWIR transmission image and the OCT three-dimensional data;

[0012] S5: Construct the DMRA-Net model by inputting the SWIR–OCT multimodal dataset into the DMRA-Net model for training and prediction;

[0013] S6: Visualize and analyze the criteria used in the model to achieve interpretable identification of healthy and diseased dental conditions.

[0014] Furthermore, the present invention can be implemented according to the following technical solution:

[0015] In step S3, the preprocessing includes: sequentially performing guided filtering and contrast-limited adaptive histogram equalization on the SWIR transmission image;

[0016] The OCT three-dimensional data is sequentially subjected to median filtering for noise reduction, linear grayscale transformation, and two-dimensional slicing along the scanning direction to obtain a continuous OCT slice sequence.

[0017] In step S4, constructing the SWIR–OCT multimodal dataset includes:

[0018] Using individual teeth as the smallest data organization unit, a unique identifier is established for each tooth sample;

[0019] For each tooth sample, a SWIR transmission image and the corresponding left adjacent plane OCT three-dimensional data and right adjacent plane OCT three-dimensional data were acquired.

[0020] The left adjacent plane OCT 3D data and the right adjacent plane OCT 3D data are sliced ​​along the depth direction to obtain the left adjacent plane OCT slice sequence and the right adjacent plane OCT slice sequence;

[0021] The left adjacent plane OCT slice sequence and the right adjacent plane OCT slice sequence are integrated into a complete OCT slice sequence in a fixed order;

[0022] Based on the unique identifier, the complete OCT slice sequence is paired one by one with the corresponding SWIR transmission image to form SWIR–OCT paired multimodal samples.

[0023] The construction of the SWIR–OCT multimodal dataset also includes a multimodal dataset partitioning step:

[0024] The SWIR transmission image and its complete OCT slice sequence corresponding to the same tooth sample are treated as a whole sample unit;

[0025] According to the unique identifier number, the overall sample unit is divided into a training set, a validation set, or a test set to ensure that the same tooth sample appears in only one of the data subsets.

[0026] The DMRA-Net model includes:

[0027] The dual-branch backbone network, based on the ResNet50 architecture, is divided into an OCT branch and a SWIR branch, which are used to process SWIR transmission images and OCT three-dimensional data, respectively.

[0028] A convolutional block attention module, inserted after each residual block, is used to fuse channel attention and spatial attention to generate enhanced feature maps;

[0029] The multi-instance feature selection module, located in the feature extraction path of the OCT branch, is used to select the most discriminative slice features from OCT 3D data; and

[0030] The feature fusion and classification module is used to fuse the features extracted from the SWIR branch and the OCT branch and output the classification result.

[0031] The multi-instance feature selection module performs the following operations:

[0032] Global average pooling and global max pooling are performed on the feature maps output by the OCT branch backbone network to obtain slice-level channel response vectors.

[0033] For each feature channel, the response value of that channel is extracted from the feature map corresponding to all OCT slices, and the slice index with the largest response is determined based on the response value, so that each feature channel corresponds to the OCT slice with the strongest response.

[0034] The feature fusion and classification module performs the following operations:

[0035] Global average pooling is performed on the feature maps output by the SWIR and OCT branches respectively to obtain a one-dimensional global feature vector.

[0036] The one-dimensional feature vector of the SWIR mode and the one-dimensional feature vector of the OCT mode are concatenated along the feature dimension to form a fused multimodal feature vector;

[0037] The fused multimodal feature vectors are input into the classifier, and the predicted probabilities of each category are output through the Softmax function.

[0038] The fusion method of the multimodal feature vectors also includes:

[0039] Linear mapping summation and fusion method; or

[0040] Attention-weighted fusion method, where the fusion weights are learnable parameters.

[0041] In step S5, a transfer learning strategy is adopted during model training. The parameters of the ResNet50 network pre-trained on a large-scale natural image dataset are loaded into the dual-branch backbone network. The parameters of the convolutional block attention module and the multi-instance feature selection module are adaptively learned during training. The cross-entropy loss function is used to update the network parameters through the backpropagation algorithm. The network parameters include ResNet parameters, convolutional block attention module parameters, multi-instance feature selection module parameters, feature fusion layer parameters, and classifier parameters.

[0042] During model prediction, the same data acquisition and processing procedures as in the training phase are performed on the tooth samples to be tested to construct multimodal input data. The multimodal input data is then input into the trained DMRA-Net model for forward inference to obtain the predicted probability and category of each tooth sample, thereby obtaining the health status and proximal caries probability of the target tooth sample.

[0043] The visualization interpretation in step S6 includes: calculating the gradient information of the classification result relative to the intermediate feature layer of the network based on the gradient weighted class activation mapping method; generating corresponding response weights according to the gradient information, and weighting and superimposing the feature maps to obtain a heatmap of the model's focus area; displaying the heatmap superimposed on the corresponding SWIR transmission image and OCT slice to intuitively show the key lesion areas that the model focuses on during the discrimination process; the visualization interpretation is executed in real time during the model prediction stage, so that the model output results simultaneously include the prediction results and the corresponding discrimination basis heatmap.

[0044] This invention solves the problem of spatial registration between SWIR and OCT images by constructing a multimodal imaging acquisition system, designing a consistent data pairing and organization strategy, and designing a bimodal residual attention network and a multi-instance feature selection mechanism. This establishes a spatial correspondence between the tooth structure regions in the two modalities, laying the foundation for subsequent feature fusion. Furthermore, the feature extraction and fusion algorithms for these two heterogeneous imaging data not only need to extract key features valuable for lesion diagnosis from each modality, but also need to fuse these complementary features through a deep fusion strategy to form a more discriminative comprehensive feature representation. This improves the model's accuracy and generalization ability in identifying complex lesions, especially proximal caries. This invention can effectively fuse SWIR and OCT multimodal image information to achieve automated, accurate, and interpretable detection of proximal caries, and to automatically identify the healthy and diseased states of teeth, demonstrating significant application value. Attached Figure Description

[0045] Figure 1 is a flowchart of the intelligent detection method for proximal caries based on short-wave infrared transmission and OCT multimodal imaging.

[0046] Figure 2 is a schematic diagram of the SWIR transmission system device;

[0047] Figure 3 This is a diagram showing the correspondence between SWIR transmission and OCT. The diagram is divided into two parts, a and b. Part a is the SWIR transmission image, and the area within its frame is the OCT acquisition area. Part b is the acquired OCT 3D image data.

[0048] Figure 4 This is the DMRA-Net architecture diagram.

[0049] Figure 5 This is a Grad-CAM visualization result. The image is divided into four parts: c, d, e, and f. Part c is the SWIR transmission image, part d is the Grad-CAM image corresponding to the SWIR transmission, part e is the OCT slice image, and part f is the Grad-CAM image corresponding to the OCT slice. Detailed Implementation

[0050] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0051] A multimodal imaging-based intelligent detection method for proximal caries includes the following steps:

[0052] S1: First, construct a tooth image acquisition system based on SWIR transmission imaging and acquire SWIR transmission images.

[0053] First, the tooth samples were prepared and processed. Specifically, several extracted human teeth were selected, sourced from clinically discarded teeth generated during orthodontic treatment or impacted tooth extraction at a hospital's dental department. After collection, the teeth were first immersed in a 0.5% chloramine-T solution for 30 minutes for initial disinfection. Subsequently, ultrasonic scaling was used to remove tartar and soft tissue residue from the tooth surface. The processed tooth samples were stored in physiological saline at 4°C, and subsequent SWIR and OCT imaging were performed within one week to preserve the original tooth structure and optical properties to the greatest extent possible.

[0054] When acquiring SWIR transmission images, the tooth sample is fixed between the short-wave infrared light source and the imaging detection module.

[0055] The system structure of the dental image acquisition system includes: a short-wave infrared light source, a beam splitter module, an optical fiber coupling collimation module, an engineering diffuser module, a tooth sample fixation device, and an imaging detection module. The short-wave infrared light source is a stable continuous light source with a center wavelength of 1310 nm. The output beam is split into two equal-intensity beams by the beam splitter module, then coupled to the optical fiber coupling collimation module via optical fiber, and finally illuminated by the engineering diffuser module with a scattering angle of 20° onto the cervical regions on both sides of the tooth sample.

[0056] Transmitted light, after passing through the tooth tissue, is received by an InGaAs short-wave infrared camera with a resolution of 1280×1024 pixels. A lens group is set at the front end of the detector to adjust the imaging magnification and complete optical imaging, thereby obtaining a SWIR transmission image covering the entire tooth area. Only one SWIR transmission image is acquired for each tooth sample to characterize the overall transmission characteristics of that tooth.

[0057] S2: Acquire OCT three-dimensional data corresponding to the SWIR transmission image.

[0058] After completing SWIR imaging, the same tooth sample is fixed on the sample stage of the OCT imaging system with the occlusal surface of the tooth facing the OCT probe. The probe position is adjusted so that the scanning area covers the proximal surface region of the tooth. A scan is performed on both the left and right proximal surfaces of the tooth, acquiring one set of OCT 3D data for each surface. Thus, each tooth sample corresponds to two sets of OCT 3D data, used to characterize the local tomographic structure information of the proximal surface region.

[0059] S3: Preprocess the SWIR transmission image and the OCT data respectively.

[0060] Guided filtering and contrast-limited adaptive histogram equalization (CLAHE) are sequentially performed on the SWIR transmission images. The preprocessing of the SWIR transmission images first employs guided filtering to suppress noise and preserve tooth edge structure, and then uses contrast-limited adaptive histogram equalization (CLAHE) to enhance the contrast between lesion areas and healthy areas.

[0061] The OCT 3D data is sequentially subjected to median filtering for denoising, linear grayscale transformation, and 2D slicing along the scanning direction to obtain a continuous OCT slice sequence. Specifically, median filtering is used to denoise the 3D data, linear grayscale transformation is performed on the OCT data, and 2D slicing is performed on the OCT 3D data along the scanning direction to obtain a continuous OCT slice sequence.

[0062] S4: Construct a SWIR–OCT multimodal dataset based on the preprocessed SWIR transmission image and the OCT three-dimensional data.

[0063] For consistency, the symbols in the formulas of this invention are defined as follows:

[0064] N: Total number of tooth samples.

[0065] i: Tooth sample index, i ∈ {1,2,…,N}.

[0066] H, W: Height and width of the SWIR image.

[0067] H O W O The height and width of the OCT slice.

[0068] K L Number of OCT slices on the left adjacent face.

[0069] K R Number of OCT slices on the right adjacent face.

[0070] K: The total number of slices in the complete OCT slice sequence, K = K L+ K R。

[0071] k: OCT slice index, k ∈ {1,2,…,K}.

[0072] C: Number of feature map channels.

[0073] H f W f Feature map spatial dimensions.

[0074] y i : The category label of the i-th tooth sample.

[0075] GAP(·): Global average pooling function.

[0076] GMP(·): Global max pooling function.

[0077] argmax: The index function that retrieves the maximum value.

[0078] [·,·]: Feature splicing operation.

[0079] set of real numbers

[0080] SWIR transmission images reflect the overall transmission characteristics of the entire tooth, while OCT data reflect the local tomographic structure information of the left and right adjacent surfaces. When the left and right adjacent surfaces of a tooth exhibit different health conditions, constructing independent samples from the same SWIR image and the OCT data from the left and right adjacent surfaces will lead to semantic inconsistencies in the lesion labels of multimodal inputs and incorrect cross-modal associations learned by the model. Furthermore, to ensure semantic consistency of multimodal data at the sample level...

[0081] Using individual teeth as the smallest data unit, a unique identifier is created for each tooth sample.

[0082] The set of tooth samples is represented as follows:

[0083]

[0084] in, Represents the SWIR transmission image of the i-th tooth sample; Represents the OCT 3D data of the left adjacent face; Represents the OCT 3D data of the right adjacent plane; This represents the tooth sample-level label. This structure ensures that all modal data originates from the same tooth sample, thereby ensuring consistency of cross-modal data in the semantic space.

[0085] For each tooth sample, a SWIR transmission image and the corresponding left and right adjacent OCT 3D data were acquired.

[0086] The OCT 3D data of the left and right adjacent planes are sliced ​​along the depth direction to obtain the OCT slice sequence of the left and right adjacent planes.

[0087] Left adjacent slice sequence:

[0088] Right adjacent slice sequence:

[0089] in, This represents the k-th slice from the left adjacent surface of the i-th tooth; This represents the k-th slice on the right adjacent surface of the i-th tooth.

[0090] The left and right adjacent OCT slice sequences are integrated into a complete OCT slice sequence in a fixed order. The fixed order is either the left adjacent slice sequence first and the right adjacent slice sequence second, or the right adjacent slice sequence first and the left adjacent slice sequence second, and the order remains consistent within the same dataset.

[0091] The OCT slice sequences from the left and right adjacent planes are integrated and stored in a fixed order:

[0092]

[0093] Obtain the complete OCT slice sequence:

[0094]

[0095] in:

[0096]

[0097] This method ensures: continuity of the spatial structure of adjacent surfaces in the OCT; complete preservation of the structure of the left and right adjacent surfaces; and avoids disruption of spatial topological relationships.

[0098] Based on the unique identifier, the complete OCT slice sequence is paired one by one with the corresponding SWIR transmission image to form SWIR–OCT paired multimodal samples.

[0099] The formula for SWIR–OCT paired multimodal samples is:

[0100]

[0101] Furthermore, to adapt to the input format of deep neural networks, the data is converted into tensor form, the SWIR transmission image is converted into tensor form as the SWIR modal input, and the complete OCT slice sequence is converted into tensor form as the OCT modal input.

[0102] SWIR input is represented as:

[0103]

[0104] Where 3 represents the number of channels.

[0105] OCT input is represented as:

[0106]

[0107] Where K represents the number of OCT slices; 3 represents the number of channels.

[0108] This approach ensures the continuity of OCT slice space while maintaining the semantic consistency and integrity of multimodal data.

[0109] The construction of the SWIR–OCT multimodal dataset also includes a multimodal dataset partitioning step. First, the SWIR transmission image and its complete OCT slice sequence corresponding to the same tooth sample are treated as a whole sample unit. Then, according to the unique identifier number, the whole sample unit is divided into a training set, a validation set, or a test set, ensuring that the same tooth sample appears in only one of the data subsets.

[0110] The tooth sample set is as follows:

[0111] in, Here is the SWIR image of the i-th tooth sample; For the corresponding OCT slice sequence; For tooth sample level labeling.

[0112] The training set, validation set, and test set are represented as follows:

[0113]

[0114] satisfy:

[0115] in, This represents the empty set.

[0116] This partitioning method ensures that the model does not access test sample information during training, thereby avoiding data leakage and improving the model's generalization ability.

[0117] S5: Construct the DMRA-Net model by inputting the SWIR–OCT multimodal dataset into the DMRA-Net model for training and prediction.

[0118] The DMRA-Net model comprises a two-branch backbone network, a convolutional block attention module, a multi-instance feature selection module, and a feature fusion and classification module. The two-branch backbone network, based on the ResNet50 architecture, is divided into an OCT branch and a SWIR branch, used to process SWIR transmission images and OCT 3D data, respectively. The convolutional block attention module is inserted after each residual block to fuse channel attention and spatial attention to generate enhanced feature maps.

[0119] The multi-instance feature selection module is set in the feature extraction path of the OCT branch and is used to select the most discriminative slice features from OCT 3D data.

[0120] Suppose that the OCT slice sequence corresponding to a certain tooth sample contains K two-dimensional slices, and the feature map is obtained after passing through the OCT backbone network:

[0121]

[0122] in, Let C represent the feature map of the k-th slice, and C represent the number of feature channels. , Indicates the feature space size.

[0123] The multi-instance feature selection module performs the following operations: First, it performs global average pooling and global max pooling on the feature maps output by the OCT branch backbone network to obtain slice-level channel response vectors. To fully characterize the channel response characteristics of each slice, it simultaneously performs global average pooling (GAP) and global max pooling (GMP) on each slice feature map.

[0124] The global average pooling formula is:

[0125]

[0126] The global max pooling formula is:

[0127]

[0128] in, This represents the average response intensity of the c-th channel in the k-th slice. This represents the maximum response intensity of the c-th channel in the k-th slice.

[0129] To comprehensively utilize the average response and maximum response information, the two are fused to obtain a slice-level channel response vector:

[0130]

[0131] in, This represents the response value of the c-th channel in the k-th slice.

[0132] For each feature channel, the response value of that channel is extracted from the feature map corresponding to all OCT slices, and the slice index with the largest response is determined based on the response value, so that each feature channel corresponds to the OCT slice with the strongest response. For example, for the c-th feature channel, the slice index with the largest response is selected from all slices:

[0133] in, This represents the slice number with the largest response on channel c, and argmax represents the index function corresponding to the maximum value.

[0134] Based on the index, the corresponding channel features are extracted from the original feature map and reorganized into a filtered OCT feature map.

[0135] Extract the corresponding channels from the original feature map:

[0136]

[0137] The final filtered OCT feature map This is used for subsequent multimodal fusion.

[0138] This mechanism enables the model to adaptively focus on the slice position that is most discriminative for the current channel, effectively mitigating the interference caused by irrelevant slices in OCT data.

[0139] The feature fusion and classification module is used to fuse the features extracted from the SWIR branch and the OCT branch and output the classification result.

[0140] Specifically, global average pooling is performed on the feature maps output by the SWIR branch and the OCT branch respectively to obtain a one-dimensional global feature vector.

[0141] Let the feature map of the SWIR branch output be:

[0142] The feature map output by the OCT branch through the multi-instance feature selection module is as follows:

[0143] The corresponding one-dimensional global feature vectors are as follows:

[0144]

[0145]

[0146] in, This indicates a global average pooling operation.

[0147] The one-dimensional feature vector of the SWIR mode and the one-dimensional feature vector of the OCT mode are concatenated along the feature dimension to form a fused multimodal feature vector;

[0148] The formula for multimodal eigenvectors is: The fused feature vector is used to jointly characterize the overall transmissivity of the tooth and the local structural information of the adjacent surfaces.

[0149] The fused multimodal feature vectors are input into the classifier, and the predicted probabilities of each category are output through the Softmax function.

[0150]

[0151] Where W and b represent the weight and bias parameters of the classifier, respectively, and p represents the predicted probability distribution of the tooth sample belonging to each category. Through the above steps, automatic classification and identification of the healthy state of teeth and the state of proximal caries can be achieved.

[0152] In addition to the above fusion methods, multimodal feature vectors can also be fused using linear mapping summation or attention-weighted fusion, where the fusion weights are learnable parameters.

[0153] Linear mapping summation and fusion method:

[0154]

[0155] in, This represents a learnable linear mapping weight matrix used to linearly transform OCT modal features so that they are aligned with SWIR modal features in the feature space.

[0156] Attention-weighted fusion method:

[0157]

[0158] in, These are learnable parameters.

[0159] S5: Construct the DMRA-Net model by inputting the SWIR–OCT multimodal dataset into the DMRA-Net model for training and prediction.

[0160] During model training, a transfer learning strategy is employed, loading the parameters of the ResNet50 network pre-trained on a large-scale natural image dataset into the dual-branch backbone network. The parameters of the convolutional block attention module and the multi-instance feature selection module are adaptively learned during training. The cross-entropy loss function is used, and the network parameters are updated through the backpropagation algorithm. The network parameters include ResNet parameters, convolutional block attention module parameters, multi-instance feature selection module parameters, feature fusion layer parameters, and classifier parameters.

[0161] Using the cross-entropy loss function:

[0162]

[0163] Where M represents the number of categories; Indicates the true label; This represents the predicted probability that a tooth sample belongs to class c.

[0164] Optimization goal:

[0165] Here, θ includes: ResNet parameters, convolutional block attention module parameters, multi-instance feature selection module parameters, feature fusion layer parameters, and classifier parameters, which are updated through the backpropagation algorithm. This represents the optimal model parameters, and argmin represents the index function that takes the maximum value.

[0166] During model prediction, the same data acquisition and processing procedures as in the training phase are performed on the tooth samples to be tested to construct multimodal input data. The multimodal input data is then input into the trained DMRA-Net model for forward inference to obtain the predicted probability and category of each tooth sample, thereby obtaining the health status and proximal caries probability of the target tooth sample.

[0167] Its prediction process is expressed as follows:

[0168]

[0169] in, This represents the predicted probability that a tooth sample belongs to class c. This indicates the predicted category of the model output, and argmax represents the index function corresponding to the maximum value.

[0170] Through the above prediction steps, the health status or the probability of interproximal caries in the target tooth sample can be obtained.

[0171] S6: Visualize and analyze the model's discrimination criteria to achieve interpretable identification of dental health and disease states. The visualization includes: calculating the gradient information of the classification result relative to the intermediate feature layers of the network based on the Grad-Weighted Class Activation Mapping (Grad-CAM) method. Corresponding response weights are generated based on the gradient information, and the feature maps are weighted and superimposed to obtain a heatmap of the model's focus areas. This heatmap is then overlaid on the corresponding SWIR transmission images and OCT slices to visually demonstrate the key disease areas the model focuses on during the discrimination process. The visualization is executed in real-time during the model prediction phase, ensuring that the model output simultaneously includes the prediction results and the corresponding heatmap of the discrimination criteria.

[0172] The method described in this embodiment can be integrated into an intelligent interproximal caries detection system and deployed on a local computing device or remote server in a dental clinic to achieve automatic analysis and assisted diagnosis of dental health status.

Claims

1. A multimodal imaging-based intelligent detection method for proximal caries, characterized in that, Includes the following steps: S1: Construct a tooth image acquisition system based on SWIR transmission imaging and acquire SWIR transmission images; S2: Acquire OCT three-dimensional data corresponding to the SWIR transmission image; S3: Preprocess the SWIR transmission image and the OCT data respectively; S4: Construct a SWIR–OCT multimodal dataset based on the preprocessed SWIR transmission image and the OCT three-dimensional data; S5: Construct the DMRA-Net model by inputting the SWIR–OCT multimodal dataset into the DMRA-Net model for training and prediction; S6: Visualize and analyze the criteria used in the model to achieve interpretable identification of healthy and diseased dental conditions.

2. The intelligent detection method for proximal caries using multimodal imaging according to claim 1, characterized in that, In step S3, the preprocessing includes: sequentially performing guided filtering and contrast-limited adaptive histogram equalization on the SWIR transmission image; The OCT three-dimensional data is sequentially subjected to median filtering for noise reduction, linear grayscale transformation, and two-dimensional slicing along the scanning direction to obtain a continuous OCT slice sequence.

3. The intelligent detection method for proximal caries using multimodal imaging according to claim 1, characterized in that, In step S4, constructing the SWIR–OCT multimodal dataset includes: Using individual teeth as the smallest data organization unit, a unique identifier is established for each tooth sample; For each tooth sample, a SWIR transmission image and the corresponding left adjacent plane OCT three-dimensional data and right adjacent plane OCT three-dimensional data were acquired. The left adjacent plane OCT 3D data and the right adjacent plane OCT 3D data are sliced ​​along the depth direction to obtain the left adjacent plane OCT slice sequence and the right adjacent plane OCT slice sequence; The left adjacent plane OCT slice sequence and the right adjacent plane OCT slice sequence are integrated into a complete OCT slice sequence in a fixed order; Based on the unique identifier, the complete OCT slice sequence is paired one by one with the corresponding SWIR transmission image to form SWIR–OCT paired multimodal samples.

4. The intelligent detection method for proximal caries using multimodal imaging according to claim 3, characterized in that, The construction of the SWIR–OCT multimodal dataset also includes a multimodal dataset partitioning step: The SWIR transmission image and its complete OCT slice sequence corresponding to the same tooth sample are treated as a whole sample unit; According to the unique identifier number, the overall sample unit is divided into a training set, a validation set, or a test set to ensure that the same tooth sample appears in only one of the data subsets.

5. The intelligent detection method for proximal caries using multimodal imaging according to claim 1, characterized in that, The DMRA-Net model includes: The dual-branch backbone network, based on the ResNet50 architecture, is divided into an OCT branch and a SWIR branch, which are used to process SWIR transmission images and OCT three-dimensional data, respectively. A convolutional block attention module, inserted after each residual block, is used to fuse channel attention and spatial attention to generate enhanced feature maps; The multi-instance feature selection module, located in the feature extraction path of the OCT branch, is used to select the most discriminative slice features from OCT 3D data; and The feature fusion and classification module is used to fuse the features extracted from the SWIR branch and the OCT branch and output the classification result.

6. The intelligent detection method for proximal caries using multimodal imaging according to claim 5, characterized in that, The multi-instance feature selection module performs the following operations: Global average pooling and global max pooling are performed on the feature maps output by the OCT branch backbone network to obtain slice-level channel response vectors. For each feature channel, the response value of that channel is extracted from the feature map corresponding to all OCT slices, and the slice index with the largest response is determined based on the response value, so that each feature channel corresponds to the OCT slice with the strongest response.

7. The intelligent detection method for proximal caries using multimodal imaging according to claim 5, characterized in that, The feature fusion and classification module performs the following operations: Global average pooling is performed on the feature maps output by the SWIR and OCT branches respectively to obtain a one-dimensional global feature vector. The one-dimensional feature vector of the SWIR mode and the one-dimensional feature vector of the OCT mode are concatenated along the feature dimension to form a fused multimodal feature vector; The fused multimodal feature vectors are input into the classifier, and the predicted probabilities of each category are output through the Softmax function.

8. The intelligent detection method for proximal caries using multimodal imaging according to claim 5, characterized in that, The fusion method of the multimodal feature vectors also includes: Linear mapping summation and fusion method; or Attention-weighted fusion method, where the fusion weights are learnable parameters.

9. The intelligent detection method for proximal caries using multimodal imaging according to claim 5, characterized in that, In step S5, During model training, a transfer learning strategy is adopted, in which the parameters of the ResNet50 network pre-trained on a large-scale natural image dataset are loaded into the dual-branch backbone network, and the parameters of the convolutional block attention module and the multi-instance feature selection module are adaptively learned during training. The cross-entropy loss function is used to update the network parameters through the backpropagation algorithm. The network parameters include ResNet parameters, convolutional block attention module parameters, multi-instance feature selection module parameters, feature fusion layer parameters, and classifier parameters. During model prediction, the same data acquisition and processing procedures as in the training phase are followed for the tooth samples to be tested, and multimodal input data is constructed. The multimodal input data is input into the trained DMRA-Net model for forward inference to obtain the predicted probability and category of each tooth sample, so as to obtain the health status and the probability of interproximal caries of the target tooth sample.

10. The intelligent detection method for proximal caries using multimodal imaging according to claim 1, characterized in that, The visualization interpretation in step S6 includes: calculating the gradient information of the classification result relative to the intermediate feature layer of the network based on the gradient weighted class activation mapping method; generating corresponding response weights according to the gradient information, and weighting and superimposing the feature maps to obtain a heatmap of the model's focus area; displaying the heatmap superimposed on the corresponding SWIR transmission image and OCT slice to intuitively show the key lesion areas that the model focuses on during the discrimination process; the visualization interpretation is executed in real time during the model prediction stage, so that the model output results simultaneously include the prediction results and the corresponding discrimination basis heatmap.

Images