Multimodal caries depth intelligent interpretation method fusing intraoral photos and x-ray films

Abstract

The application provides a multi-modal caries depth intelligent interpretation method fusing intraoral photos and X-ray films, and through standardized oral multi-modal medical image and pathological data synchronous collection, spatial registration and pixel-level normalization, the spatiotemporal consistency of clinical images and histopathological data is realized, the application utilizes a double-channel convolutional neural network and a lightweight LSTM, applies a physical constraint gate unit to perform channel re-labeling on a double-modal shallow feature map, generates a re-labeled feature map, and after splicing, outputs a caries depth grading interpretation result through a decoding network, so that the accuracy and consistency of caries depth grading interpretation are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent medical image analysis and auxiliary diagnosis of oral diseases, and in particular to a multimodal intelligent method for interpreting the depth of dental caries by integrating intraoral photographs and X-rays. Background Technology

[0002] In the current fields of intelligent medical image analysis and auxiliary diagnosis of oral diseases, intelligent image interpretation technology is increasingly becoming an important means of clinical auxiliary diagnosis. This is especially true as the demand for early diagnosis of oral diseases, particularly dental caries, continues to rise.

[0003] Current intelligent image interpretation technology faces the following problems in the field of dental caries interpretation: First, existing methods have rigid modal weight allocation under different caries depths and disease stages, making it difficult to dynamically adapt to content and physiological changes. Second, it is necessary to achieve a fundamental leap in model modal weight allocation from being driven by statistical content to being guided by pathological mechanisms. Third, it is necessary to introduce interpretable dynamic modeling so that each interpretation output is supported by clinical physical variables, enabling traceable review, assisting doctors in decision-making, and improving the system robustness and application safety in complex cases. Summary of the Invention

[0004] This invention provides a multimodal intelligent caries depth interpretation method that integrates intraoral photographs and X-rays, aiming to solve the problems of the prior art mentioned in the background section.

[0005] The technical solution of this invention is: a multimodal intelligent caries depth interpretation method integrating intraoral photographs and X-rays, comprising the following steps: S1: Collect multimodal oral medical imaging data, including intraoral photographs and registered X-rays from multiple follow-up visits, and generate multimodal medical imaging sequences; S2: Perform spatial registration and pixel-level normalization preprocessing on multimodal medical image sequences to generate registered and normalized image data; S3: Feature extraction is performed on the registered and normalized image data through a dual-channel convolutional neural network to generate RGB feature maps of intraoral photographs and grayscale feature maps of X-ray films as dual-modal shallow feature maps; S4: Based on the registration and normalization image data, a caries dynamic encoder is constructed, wherein the caries dynamic encoder utilizes a shared spatial coordinate embedding module and a lightweight LSTM unit to model the local demineralization propagation direction along the spatial principal axis to generate the caries dynamic encoder; S5: Input the dual-modal shallow feature map into the caries lesion dynamic encoder to generate a dynamic evolution state vector; S6: Based on the dynamic evolution state vector and the preset clinical rules, a physical constraint gating unit is constructed to map the dynamic evolution state vector output by the caries lesion dynamic encoder into a gating weight. This weight reflects the current development trend of caries lesions in the time series. The preset clinical rules include a preset clinical prior rule matrix based on the known physical density and X-ray attenuation coefficient of enamel, dentin and pulp tissue. S7: Apply physical constraint gating units to recalibrate the channels of the dual-modal shallow feature map, generate recalibrated feature maps, and output the caries depth grading results through the decoding network after splicing.

[0006] Furthermore, in step S6, the caries dynamic encoder includes a shared spatial coordinate embedding module and a lightweight LSTM unit. It performs slice sampling on the spatial enhancement feature map along the normal direction of the occlusal surface. The LSTM unit inputs the demineralization propagation sequence and outputs a dynamic evolution state vector to characterize the degree of enamel demineralization and the slope of the mineral density gradient.

[0007] Furthermore, in step S7, the dynamic evolution state vector is mapped to the core physical variable space after being processed by fully connected decoding, and a gating unit bias term is generated by comparing it with the clinical threshold. The physical constraint gating function concatenates features based on preset initial bias parameters, learnable weight matrix, dynamic evolution state vector, and global pooling results of shallow feature maps. After linear weighting and Sigmoid activation function processing, it outputs two gating weights for recalibrating the channels of intraoral photographs and X-ray films, respectively. The recalibrated feature map of the intraoral photograph after channel recalibration and the recalibrated feature map of the X-ray are concatenated along the feature dimension. After being input into the decoding network for feature decoding processing, the probability distribution of three categories, superficial caries, enamel caries, and deep caries, is output through the Softmax classification layer.

[0008] The beneficial technical effects of this invention are as follows: 1) This invention achieves a leap from morphological representation to pathological process reasoning by deeply integrating prior knowledge of oral clinical pathology with a deep neural network architecture, constructing a caries dynamic encoder, and capturing the local lesion propagation trend along the spatial principal axis using a lightweight LSTM. 2) By introducing a dynamic consistency loss function, this invention achieves explicit constraints on the temporal consistency of pathological processes in an end-to-end learning framework, thereby breaking through the limitation of traditional classification models that only focus on the terminal state and ignore intermediate evolution. This not only enhances the model's ability to integrate continuous monitoring data over time, but also enables it to reverse-engineer the potential states of invisible intermediate stages, providing support for clinical retrospective analysis and intervention window prediction. Attached Figure Description

[0009] Figure 1This is a flowchart illustrating the method of an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the process of generating registration-normalized image data according to an embodiment of the present invention. Figure 3 This is a schematic diagram of the process for generating a dual-modal shallow feature map according to an embodiment of the present invention. Detailed Implementation

[0010] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0011] The following disclosure provides many different embodiments or examples for implementing different structures of the invention. To simplify the disclosure, specific examples of components and arrangements are described below. Of course, these are merely examples and are not intended to limit the invention. Furthermore, reference numerals and / or letters may be repeated in different examples; such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0012] like Figure 1 As shown in the figure, this embodiment provides a multimodal intelligent caries depth interpretation method that integrates intraoral photographs and X-rays, including the following steps: S1: Collect multimodal oral medical imaging data, including intraoral photographs and registered X-rays from multiple follow-up visits, and generate multimodal medical imaging sequences; S2: Perform spatial registration and pixel-level normalization preprocessing on the multimodal medical image sequence to eliminate imaging scale differences and generate registered and normalized image data; S3: The registered and normalized image data are extracted using a dual-channel convolutional neural network to generate an RGB feature map of the intraoral photograph and a grayscale feature map of the X-ray as dual-modal shallow feature maps. S4: Based on the registered and normalized image data, a caries dynamic encoder is constructed, wherein the caries dynamic encoder utilizes a shared spatial coordinate embedding module and a lightweight LSTM unit to model the local demineralization propagation direction along the spatial principal axis to generate the caries dynamic encoder; S5: Input the dual-modal shallow feature map into the caries kinetic encoder to generate a dynamic evolution state vector; S6: Based on the dynamic evolution state vector and the preset clinical rules, a physical constraint gating unit is constructed to map the dynamic evolution state vector output by the caries kinetic encoder into gating weights, which reflect the current caries development trend in the time series; the preset clinical rules include a preset clinical prior rule matrix based on the known physical density and X-ray attenuation coefficient of enamel, dentin and pulp tissue.

[0013] S7: The physical constraint gating unit is used to recalibrate the channels of the dual-modal shallow feature map, generate a recalibrated feature map, and output the caries depth grading result through the decoding network after splicing.

[0014] In this embodiment, step S1 involves acquiring a multimodal oral medical image data sequence, including intraoral photographs and registered X-ray films from multiple follow-ups, and generating a multimodal medical image sequence, including the following steps: S1.1: Develop and process multimodal data acquisition protocols for oral clinical diagnosis and treatment needs. Based on the medical prior knowledge of caries evolution dynamics modeling, clarify the follow-up interval ≥3 months, intraoral photograph resolution ≥12 million pixels and X-ray exposure parameters to generate standardized oral multimodal medical image acquisition specifications. S1.2: Based on the oral multimodal medical image acquisition specifications, perform simultaneous acquisition of intraoral photographs and X-rays for multiple follow-up visits, and use an oral endoscope system to acquire RGB three-channel surface texture image data and a dental digital X-ray imaging device to acquire grayscale density distribution image data to generate an original oral multimodal medical image dataset. S1.3: Based on the spatiotemporally aligned pathological association dataset, perform image sequence timestamp calibration and spatial coordinate registration processing, and use a rigid transformation algorithm to align the anatomical coordinate systems of intraoral photographs and X-ray films to generate a spatiotemporally consistent multimodal medical image sequence dataset. In this embodiment, step S2 involves performing spatial registration and pixel-level normalization preprocessing on the multimodal medical image sequence to eliminate imaging scale differences and generate registered and normalized image data, such as... Figure 2 As shown, it includes the following steps: S2.1: Obtain the multimodal medical image sequence generated in step S1 as the input object, which includes the RGB three-channel pixel matrix of intraoral photographs and the single-channel grayscale pixel matrix of X-ray films; perform initial rigid alignment processing based on prior knowledge of oral anatomy, using the tooth midline and occlusal surface contour as the reference point set, calculate the initial spatial transformation parameters through the iterative nearest point algorithm, and generate intermediate image data with preliminary geometric alignment relationship; S2.2: Perform non-rigid registration processing based on the mutual information maximization criterion on the preliminary geometrically aligned image data output from S2.1, and optimize the control point displacement field of the B-spline free deformation model layer by layer using the multi-resolution pyramid strategy; input the normalized color gradient features of the intraoral photograph and the local phase consistency features of the X-ray film, and iteratively solve the deformation field parameters through the conjugate gradient optimization algorithm to output the registered image pair with completely consistent spatial coordinates; S2.3: Based on the registered image pairs with perfectly consistent spatial coordinates generated in S2.2, calculate the target registration error index to verify the geometric alignment quality; input the three-dimensional coordinate residuals of key anatomical landmarks of the maxilla and mandible, evaluate the registration accuracy using the Mahalanobis distance threshold method, and trigger the registration re-optimization process when the residual vector norm exceeds 0.5 mm to generate spatially registered image data that meets clinical accuracy requirements. S2.4: Perform pixel-level radiometric normalization on the spatially registered image data output by S2.3, using the mineral density reference distribution of the X-ray as the baseline template; input the HSV color space saturation component of the intraoral photograph and the gray-scale cumulative distribution function of the X-ray, and dynamically adjust the color intensity mapping curve through the histogram matching algorithm to generate intermediate normalized image data with radiometric consistency and elimination of illumination variation. For example, in a dental caries follow-up case, the spatially registered intraoral photograph had a resolution of 4000×3000 pixels, an average saturation of 0.62 after HSV conversion of the RGB channels, and a grayscale range of 0 to 255 for the X-ray film. Its cumulative distribution function (CDF) reached 0.52 at a grayscale value of 120. When executing the histogram matching algorithm, a mapping function was set. for: in, For intraoral photograph saturation CDF, It is the inverse function of the grayscale CDF of X-ray film. Enter the saturation value.

[0015] After applying this function, the mean saturation value was adjusted to 0.48, and the variance was reduced to 0.71 times the original value. After inverse conversion and synthesis of RGB images, the non-uniformity of visible light illumination was significantly reduced compared to the image before registration, and the radiometric consistency of the dual-modal image in the registration area was enhanced, meeting the input requirements for subsequent multimodal intensity range standardization processing. S2.5: Based on the intermediate normalized image data generated in S2.4, perform multimodal intensity range normalization processing, inputting the normalized color intensity mean of the intraoral photograph and the dynamic range of mineral density of the X-ray film; map the bimodal data to a unified grayscale space through a linear transformation algorithm, where the intraoral photograph is converted into a pseudo-grayscale image with an intensity scale of 0-255, and the X-ray film retains the negative value preservation characteristic, and finally outputs a registered normalized image dataset that eliminates imaging scale differences.

[0016] In this embodiment, in step S3, a dual-channel convolutional neural network is used to extract features from the registered and normalized image data, generating an RGB feature map of the intraoral photograph and a grayscale feature map of the X-ray film as dual-modal shallow feature maps, such as... Figure 3 As shown, the specific steps include the following: S3.1: Based on standards in the fields of electrical engineering and deep learning, a dual-branch convolutional neural network model is constructed to generate a network architecture with modality-specific feature extraction capabilities. Specifically, the first branch convolutional layer and the second branch convolutional layer are initialized using a deep learning framework. The first branch convolutional layer is configured to process three-channel RGB input, and the second branch convolutional layer is configured to process single-channel grayscale input. The convolutional kernel parameters are optimized through a shared weight initialization strategy to obtain the dual-branch convolutional neural network model. For example, when performing this sub-step, the input registration-normalized image dataset is 256×256 pixels, the intraoral image is RGB three-channel, and the X-ray is a single grayscale channel. The first branch has 64 convolutional kernels, a kernel size of 3×3, and a stride of 1; the second branch has 32 convolutional kernels, both with a kernel size of 3×3 and a stride of 1. When applying the shared weight initialization strategy, the convolutional kernel weights are initialized using a HeNormal distribution, with all biases set to zero to ensure stable initial gradient propagation of the convolutional layers. The convolutional module of each branch is configured as a stacked sequence of convolutional layer → batch normalization layer → ReLU activation layer, with a total of three layers output in series. The compilation parameters of the completed dual-branch convolutional model are: optimizer Adam, learning rate 0.001, and loss function is mean squared error, where the mean squared error is calculated using the following formula: in, For the sample size, For predicted values, subscript The values ​​represent the true target values. This embodiment runs the model building process after loading the training data, outputting a dual-branch convolutional neural network model structure file (JSON format) and an initial weight file (HDF5 format). It also verifies that the shallow feature extraction response quality on the validation set is significantly improved, ensuring the effectiveness and stability of subsequent cross-modal feature fusion.

[0017] S3.2: Perform channel separation processing on the registered and normalized image data to extract the RGB data of the intraoral photograph and the grayscale data of the X-ray film; specifically, based on the channel dimension characteristics of the image data, execute the RGB channel segmentation algorithm to separate the three-channel data of the intraoral photograph, and simultaneously execute the grayscale channel extraction algorithm to obtain the single-channel data of the X-ray film, generating the separated RGB data of the intraoral photograph and the grayscale data of the X-ray film; S3.3: The intraoral photograph RGB data is input into the first branch convolutional layer of the dual-branch convolutional neural network model. Through multi-level convolution operations and non-linear activation processing, an intraoral photograph RGB feature map is generated. Specifically, based on the separated intraoral photograph RGB data, the convolution kernel sliding window algorithm is applied to perform local feature extraction, and a non-linear transformation is performed through the ReLU activation function to output an intraoral photograph RGB feature map with a spatial hierarchical structure. S3.4: Input the X-ray grayscale data into the second branch convolutional layer of the dual-branch convolutional neural network model, and generate the X-ray grayscale feature map through multi-level convolution operations and downsampling processing; specifically, based on the separated X-ray grayscale data, perform convolution kernel feature response calculation and combine max pooling algorithm for resolution compression to obtain the X-ray grayscale feature map that retains mineral density gradient information. Furthermore, by using a multi-level convolution operation method, the edge and texture information of local high-density and low-density regions of X-ray films under different receptive fields is captured in a layered manner, and a multi-scale gray-scale feature response matrix is ​​obtained.

[0018] S3.5: Perform format alignment processing on the RGB feature map of intraoral photographs and the grayscale feature map of X-ray films to generate a bimodal shallow feature map; specifically, based on the spatial dimension parameters of the feature maps, apply a bilinear interpolation algorithm to unify the feature map resolution, and integrate the two types of feature maps through channel dimension splicing operation to output a bimodal shallow feature map that meets the input requirements of the subsequent caries dynamics encoder.

[0019] In this embodiment, in step S4, a caries dynamic encoder is constructed based on the registered and normalized image data. The caries dynamic encoder utilizes a shared spatial coordinate embedding module and a lightweight LSTM unit to model the local demineralization propagation direction along the spatial principal axis to generate the caries dynamic encoder. Specifically, the steps include the following: S4.1: Obtain the dual-modal shallow feature map as input data, perform spatial coordinate grid generation processing based on image resolution parameters, generate a standardized two-dimensional spatial coordinate grid, and establish a pixel-level spatial position reference system; S4.2: Input the standardized two-dimensional spatial coordinate grid into the shared spatial coordinate embedding module, perform learnable linear transformation processing, generate a position-aware tensor to encode prior information about spatial location and enhance the geometric consistency of the feature map; S4.3: The position-aware tensor and the bimodal shallow feature map are concatenated along the channel dimension to generate a spatially enhanced feature map, which integrates prior position information with image feature representation to improve the spatial semantic representation capability of the features. For example, in longitudinal follow-up image data, the input conditions for constructing the spatially enhanced feature map are H=256, W=256, the number of channels in the bimodal shallow feature map are C1=64 and C2=64, and the number of channels in the position-aware tensor is 2. Execution After channel concatenation, the fused feature matrix has 130 channels. Normalization is performed on the concatenated matrix, and the mean is calculated. With variance The formula is: in, For the first Feature values ​​of each pixel This represents the total number of pixels.

[0020] variance The calculation formula is: The local consistency index is calculated by the convolution smoothing operator, and the mutual information value MI=0.087 of the enhanced feature ROI region is obtained, which greatly improves the spatial semantic representation ability. The final output spatial enhanced feature map significantly improves the feature separability of the demineralization propagation direction in the subsequent S5.4 slice sampling modeling.

[0021] S4.4: Based on the interlocking surface normal, the spatial principal axis direction is determined, and the spatial enhancement feature map is sliced ​​and sampled along the principal axis to generate a local demineralization propagation sequence in order to extract the temporal evolution features of the demineralization front along the spatial principal axis; S4.5: Input the local demineralization propagation sequence into a lightweight LSTM unit to perform sequence modeling processing, and output a dynamic evolution state vector to quantitatively characterize the diffusion rate of the caries front and the decay law of the mineral density gradient, forming a mathematical state representation of the pathological evolution.

[0022] Furthermore, by calculating the dynamic weights of the input gate, forget gate, and output gate through a gating mechanism, and by employing matrix multiplication and nonlinear activation operations, the gradual memorization and selective forgetting of the local demineralization propagation sequence features are achieved, and the hidden state and cell state at each time step are obtained.

[0023] Furthermore, by performing time-dimensional global average pooling processing using the hidden state sequence, consistent aggregation of features related to the degree of glaze demineralization is achieved, and an intermediate quantization value characterizing the average diffusion rate is generated.

[0024] Furthermore, using the gradient calculation method along the principal spatial axis, the mineral density gradient decay is calculated using the following formula. law: in, For mineral density, The coordinates of the current time step. The position coordinates for the next time step. This represents the length increment along the principal axis of space.

[0025] In this embodiment, step S5 involves inputting the dual-modal shallow feature map into the caries lesion dynamic encoder to generate a dynamic evolution state vector, specifically including the following steps: S5.1: Perform spatial coordinate embedding processing on the dual-modal shallow feature map. Based on the shared spatial coordinate embedding module, convert the pixel coordinate information of the intraoral photo RGB feature map and the X-ray grayscale feature map into a two-dimensional position encoding matrix to generate a position-aware feature tensor as the input basis for modeling the demineralization propagation direction. S5.2: Spatial principal axis sequence modeling based on position-aware feature tensor execution of lightweight LSTM units, temporally processing the feature sequence of the enamel-dentin interface region along the occlusal surface normal direction to extract local demineralization propagation direction features and generate demineralization propagation feature sequence as an intermediate representation for dynamic evolution state modeling. S5.3: Calculate the dynamic evolution state vector of the demineralization propagation feature sequence, and use the hidden state output layer of the lightweight LSTM unit to aggregate the sequence features to generate a dynamic evolution state vector as the basis for the regulation of the physical constraint gating unit; perform pathological consistency verification processing based on the dynamic evolution state vector to generate an effective dynamic evolution state vector that conforms to the dynamic law of caries evolution as the input parameter of the subsequent physical constraint gating unit.

[0026] S6: Based on the dynamic evolution state vector and preset clinical patterns, a physical constraint gating unit is constructed. The dynamic evolution state vector output by the caries kinetic encoder is mapped to gating weights, which reflect the current development trend of caries over time. The preset clinical patterns include a preset clinical prior pattern matrix based on the known physical density and X-ray attenuation coefficient of enamel, dentin, and pulp tissue. The gating weights are multiplied element-wise with the prior pattern matrix to generate a physical constraint gating signal, thereby ensuring that the feature selection process conforms to the anatomical structure of the teeth and the evolution of the disease course.

[0027] S6.1: Gated weight mapping of dynamically evolving state vectors.

[0028] The dynamic evolution state vector output by the caries kinetic encoder is converted into initial gating weights. These weights are directly correlated with the channel dimension of the multimodal feature map and are used for subsequent feature recalibration. Specifically, the dynamic evolution state vector is a high-dimensional tensor whose dimension is determined by the number of LSTM units and the temporal step size in the encoder. It contains information on the local demineralization propagation direction modeled along the spatial principal axis, such as the rate and direction of diffusion from the enamel surface to the deeper dentin. To map it into gating weights that match the number of channels in the bimodal shallow feature map (i.e., the RGB feature map of intraoral photographs and the grayscale feature map of X-ray films), this step first uses a fully connected layer or a one-dimensional convolutional layer to reduce or transform the state vector, generating an initial weight vector whose length is equal to the total number of channels in the bimodal feature map. Subsequently, the weight values ​​are compressed to between 0 and 1 using the sigmoid activation function to form the initial gating weights. For example, assuming the intraoral photograph feature map and the X-ray feature map extracted by the dual-channel convolutional neural network contain 64 channels, the total number of channels is 128. The initial gating weights are 128-dimensional vectors, where each element represents the importance of the corresponding channel feature at the current stage of caries evolution. For instance, when caries progresses from enamel to dentin, the component representing dentin demineralization in the state vector increases. After mapping, the channel weights reflecting dentin density changes in the corresponding X-ray feature map increase, thus providing a basis for subsequent physical constraints. This mapping process not only realizes the transformation of temporal dynamic information into spatial feature weights but also ensures that the weights can capture the differences in disease progression among individual patients through adaptive adjustment of learnable network parameters.

[0029] S6.2: Construction of a clinical prior rule matrix based on clinical prior knowledge.

[0030] The clinical prior rule matrix, based on standard pathological knowledge of tooth anatomy, tissue physical properties, and caries evolution, assigns prior importance to feature channels of different tissue types. In its implementation, a set of prior rules is first defined according to the anatomical regions of the tooth (e.g., enamel, dentin, pulp chamber) and common caries sites (e.g., pits, fissures, proximal surfaces, smooth surfaces). For example, enamel has a high X-ray attenuation coefficient, appearing as a high-density area on X-rays, and its texture is crucial for early caries detection; while dentin has a high organic content, resulting in more pronounced color changes in intraoral photographs. Therefore, the pre-defined clinical prior rule matrix is ​​a vector or matrix with the same dimension as the number of channels in the bimodal feature map, where each element corresponds to a prior weight for a feature channel. This weight is initialized based on the anatomical region and tissue type corresponding to that channel. For example, if a certain X-ray feature channel is primarily sensitive to grayscale changes in the enamel region, its prior weight is set to a higher value (e.g., 0.9) to emphasize its role in interpreting superficial caries; conversely, if a certain intraoral photograph feature channel primarily corresponds to gingival soft tissue, its prior weight is set to a lower value (e.g., 0.1) to avoid noise interference. Furthermore, this clinical prior rule matrix incorporates the progression of the disease; for instance, as the depth of caries increases, the prior weight of dentin-related channels should gradually increase, while the weight of enamel channels relatively decreases. This matrix can be constructed using a dental expert knowledge base or by statistically analyzing labeled data from a large number of clinical samples, forming a hard or soft constraint to ensure that subsequent gating mechanisms conform to physiological and pathological common sense. For example, in interpreting proximal caries, since X-rays are often blurred due to overlap, the prior weight of indirect signs (such as color changes) channels in intraoral photographs is appropriately increased to balance multimodal information.

[0031] S6.3: Generation of physical constraint gating signals.

[0032] This step generates the final physically constrained gating signal by fusing the initial gating weights with the clinical prior knowledge matrix. Specifically, the initial gating weight vector obtained in S6.1 is multiplied element-wise with the clinical prior knowledge matrix constructed in S6.2 to obtain the fused weight vector. This multiplication operation achieves joint modulation of dynamic evolutionary information and static prior knowledge: the initial gating weights reflect the current individualized disease progression dynamics of the patient (such as demineralization rate and diffusion direction), while the clinical prior matrix provides anatomical common sense based on population statistics. The multiplication of the two ensures that the final gating signal adapts to individual differences without deviating from basic medical principles. To enhance nonlinear expressive power, an optional activation function (such as ReLU or Tanh) can be introduced after the multiplication, followed by normalization to ensure a reasonable weight distribution. For example, suppose a patient's caries progresses rapidly, with an initial gating weight of 0.8 for the dentin in S6.1, while the prior weight for the dentin channel in the clinical prior matrix is ​​0.7, resulting in a fused weight of 0.56. Simultaneously, for the enamel channel, the initial weight may decrease to 0.3 due to deeper caries, with a prior weight of 0.9, and a fused weight of 0.27. Thus, even if the initial weights indicate a decrease in the importance of enamel features, prior knowledge still retains a certain basic weight, preventing key features from being completely ignored due to individual abnormal fluctuations. Furthermore, this step can introduce spatial constraints, such as grouping weights according to tooth anatomical regions, to ensure a smooth transition of weights between adjacent channels and avoid abrupt changes. The final generated physical constraint gating signal is a weight vector corresponding one-to-one with the feature map channels, used for subsequent feature recalibration.

[0033] S6.4: Feature recalibration based on physical constraint gating.

[0034] This step applies the physical constraint gating signal generated by S6.3 to recalibrate the channels of the bimodal shallow feature map, thereby strengthening features relevant to caries depth interpretation and suppressing irrelevant or interfering features. Specifically, the gating signal vector is multiplied channel by channel by the corresponding bimodal shallow feature map; that is, for each channel's feature map, the pixel values ​​at all spatial locations are multiplied by the gating weight of that channel. This operation is similar to the feature recalibration mechanism in SENet, but the difference lies in that the gating signal incorporates temporal dynamic information and clinical priors, making it more targeted and interpretable. The recalibrated feature maps are then concatenated along the channel dimension to form a fused feature map, which is then used by the subsequent decoding network to output the caries depth grading result. For example, in X-ray feature maps reflecting dentin demineralization, a high gating weight (e.g., 0.9) amplifies the feature value of that channel, thus highlighting anomalies in the dentin region during decoding. Conversely, in intraoral photographs, a low gating weight (e.g., 0.1) weakens the contribution of channels reflecting plaque color, preventing surface plaque from being misclassified as deep caries. Furthermore, the recalibration process can be iterated multiple times, for example, by applying gating to multiple convolutional feature maps to optimize features at different scales. In this way, the decoding network can more accurately focus on key features related to caries depth. For instance, when interpreting moderate caries (reaching the superficial dentin), it simultaneously enhances the low-density dentin areas in X-rays and the brown dark areas in intraoral photographs, thereby improving grading accuracy. Finally, the recalibrated feature map is input into the decoding network (e.g., a fully convolutional network or a Transformer decoder), outputting pixel-level caries depth classifications (e.g., superficial caries, moderate caries, deep caries), achieving intelligent interpretation.

[0035] In this embodiment, step S7 involves applying the physical constraint gating unit to recalibrate the channels of the bimodal shallow feature map, generating a recalibrated feature map, and then stitching it together before outputting the caries depth grading result through a decoding network. Specifically, this includes the following steps: S7.1: Apply the channel recalibration process to the intraoral photo RGB feature map output by the physical constraint gating unit to generate an intraoral photo recalibrated feature map.

[0036] S7.2: Apply the physical constraint gating unit outputting the X-ray film channel weights to the X-ray film grayscale feature map to perform channel recalibration processing, so as to generate an X-ray film recalibration feature map.

[0037] S7.3: Perform feature stitching processing on the feature dimension of the intraoral photograph recalibrated feature map and the X-ray recalibrated feature map to generate a fused feature map.

[0038] S7.4: Input the fused feature map into the decoding network to perform feature decoding processing to generate intermediate interpretation features.

[0039] S7.5: Based on intermediate interpretation features, classification processing is performed through a classification layer to output the caries depth grading results; Furthermore, the linear activation value of each category is converted into a probability value in the range of 0 to 1 by using the Softmax normalization function, and the corresponding reading probability vector for each category is obtained.

[0040] Furthermore, the cross-entropy loss function is used to optimize the classification model and generate classification error data; the formula is as follows: ; in, For category indexing, The first true label vector One portion, For the prediction probability vector, the first... Each component.

[0041] For those skilled in the art, various other corresponding changes and modifications can be made based on the technical solutions and concepts described above, and all such changes and modifications should fall within the protection scope of the claims of this invention.

[0042] Unless otherwise defined, the technical or scientific terms used herein should be understood in their ordinary sense by one of ordinary skill in the art to which this invention pertains. The terms “first,” “second,” “third,” and similar terms used in this patent application specification and claims do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms “an” or “a” and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms “comprising” or “including” and similar terms mean that the elements or objects preceding “comprising” or “including” encompass the elements or objects listed following “comprising” or “including” and their equivalents, and do not exclude other elements or objects. The “multiple” involved in the embodiments of this invention refers to two or more. A and / or B indicate three possibilities: A; B; and A and B.

[0043] The above description is merely an exemplary embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multimodal intelligent method for interpreting dental caries depth by integrating intraoral photographs and X-rays, characterized in that, Includes the following steps: S1: Collect multimodal oral medical imaging data, including intraoral photographs and registered X-rays from multiple follow-up visits, and generate multimodal medical imaging sequences; S2: Perform spatial registration and pixel-level normalization preprocessing on multimodal medical image sequences to generate registered and normalized image data; S3: Feature extraction is performed on the registered and normalized image data through a dual-channel convolutional neural network to generate RGB feature maps of intraoral photographs and grayscale feature maps of X-ray films as dual-modal shallow feature maps; S4: Based on the registration and normalization image data, a caries dynamic encoder is constructed, wherein the caries dynamic encoder utilizes a shared spatial coordinate embedding module and a lightweight LSTM unit to model the local demineralization propagation direction along the spatial principal axis to generate the caries dynamic encoder; S5: Input the dual-modal shallow feature map into the caries lesion dynamic encoder to generate a dynamic evolution state vector; S6: Based on the dynamic evolution state vector and the preset clinical rules, construct a physical constraint gating unit. The preset clinical rules include a preset clinical prior rule matrix based on the known physical density and X-ray attenuation coefficient of enamel, dentin and pulp tissue. S7: Apply physical constraint gating units to recalibrate the channels of the dual-modal shallow feature map, generate recalibrated feature maps, and output the caries depth grading results through the decoding network after splicing. In step S4, a caries kinetic encoder is constructed based on the registered and normalized image data. This caries kinetic encoder utilizes a shared spatial coordinate embedding module and a lightweight LSTM unit to model the local demineralization propagation direction along the spatial principal axis, generating the caries kinetic encoder. This process includes the following steps: The dual-modal shallow feature map is obtained as input data, and spatial coordinate grid generation processing is performed based on the image resolution parameter to generate a standardized two-dimensional spatial coordinate grid to establish a pixel-level spatial position reference system. The standardized two-dimensional spatial coordinate grid is input into the shared spatial coordinate embedding module, and a learnable linear transformation is performed to generate a position-aware tensor. The position-aware tensor and the bimodal shallow feature map are concatenated along the channel dimension to generate a spatially enhanced feature map. Based on the direction of the spatial principal axis determined by the normal of the interlocking surface, the spatial enhancement feature map is sliced ​​and sampled along the principal axis to generate a local demineralization propagation sequence. The local demineralization propagation sequence is input into a lightweight LSTM unit to perform sequence modeling processing and output a dynamic evolution state vector. Step S5 also includes: converting the pixel coordinate information of the intraoral photograph RGB feature map and the X-ray grayscale feature map into a two-dimensional position coding matrix based on the shared spatial coordinate embedding module; Spatial principal axis sequence modeling based on position-aware feature tensor execution of lightweight LSTM units, temporal processing of feature sequences of the enamel-dentin interface region along the occlusal surface normal direction to extract local demineralization propagation direction features and generate demineralization propagation feature sequences; Dynamic evolution state vectors are calculated for the demineralization propagation feature sequences. The hidden state output layer of a lightweight LSTM unit is used to aggregate the sequence features to generate dynamic evolution state vectors that include the predicted value of glaze demineralization degree and the slope of mineral density gradient.

2. The multimodal caries depth intelligent interpretation method integrating intraoral photographs and X-rays according to claim 1, characterized in that, In step S1, RGB three-channel surface texture image data are acquired using an oral endoscope system.

3. The multimodal caries depth intelligent interpretation method integrating intraoral photographs and X-rays according to claim 1, characterized in that, In step S1, grayscale density distribution image data is acquired using a dental digital X-ray imaging device.

4. The multimodal caries depth intelligent interpretation method integrating intraoral photographs and X-rays according to claim 1, characterized in that, In step S3, the first branch convolutional layer and the second branch convolutional layer are initialized using a deep learning framework to obtain a two-branch convolutional neural network model.

5. The multimodal caries depth intelligent interpretation method integrating intraoral photographs and X-rays according to claim 4, characterized in that, In step S3, the first branch convolutional layer is configured to process three-channel RGB input, the second branch convolutional layer is configured to process single-channel grayscale input, and the convolutional kernel parameters are optimized through a shared weight initialization strategy.

6. The intelligent multimodal caries depth interpretation method integrating intraoral photographs and X-rays according to claim 5, characterized in that, In step 3, the intraoral photo RGB data is input into the first branch convolutional layer of the dual-branch convolutional neural network model, and multi-level convolutional operations and non-linear activation processing are used to generate the intraoral photo RGB feature map. The grayscale data of the X-ray film is input into the second branch convolutional layer of the dual-branch convolutional neural network model. Through multi-level convolution operations and downsampling processing, the grayscale feature map of the X-ray film is generated. The RGB feature map of intraoral photographs and the grayscale feature map of X-ray films are format aligned to generate a bimodal shallow feature map.

7. The intelligent multimodal caries depth interpretation method integrating intraoral photographs and X-rays according to claim 1, characterized in that, In step S6, constructing the physical constraint gating unit includes mapping the dynamic evolution state vector output by the caries kinetic encoder to a gating weight, which reflects the current development trend of caries in the time series.

Images