Oral cavity image recognition method and system based on deep learning, and storage medium

Abstract

The invention provides a deep learning-based oral cavity image recognition method, a storage medium and a deep learning-based oral cavity image recognition system. The method comprises the steps of deploying a federated learning framework and collecting a multi-modal oral cavity image data set; extracting local features to obtain image features, and generating a modal adaptive weight map; a multi-head self-attention mechanism is used for fusing the cross-modal features to generate a fused feature map, and deconvolution up-sampling is carried out to form high-resolution multi-modal feature representation. A tooth segmentation mask is generated based on this representation, and an initial diagnostic report is generated. And aggregating the attention weight of each client through an encryption protocol, and generating interpretable decision support data. And finally, generating a structured clinical report by using a natural language. According to the method, the Grad-CAM thermodynamic diagram is combined with the encrypted and aggregated attention weight, so that the privacy security is guaranteed, the model interpretability is enhanced, the clinical credibility and the diagnosis decision efficiency are improved, and the problems of insufficient diagnosis precision of complex lesions and insufficient utilization of multi-modal information in the prior art are solved.

Description

Technical Field

[0001] This invention relates to the field of oral image recognition technology, and in particular to an oral image recognition method, storage medium, and system based on deep learning. Background Technology

[0002] In the field of oral image recognition, traditional methods have long relied on manual image interpretation and experience-based judgment. For example, manually locating lesions such as caries and periodontal disease using X-ray images is inefficient due to the limited number of images and the doctor's experience, and is highly subjective, easily leading to missed or misdiagnosed cases. With the development of machine learning technology, early research attempted to use single-modal images (such as a single X-ray image) combined with traditional classifiers (such as SVM and random forest) to achieve automated lesion detection. However, limited by feature extraction capabilities, the accuracy in recognizing complex lesions (such as early caries and microfractures) is limited. In recent years, deep learning technology has been gradually applied to oral image analysis. For example, single-modal image segmentation or classification models based on convolutional neural networks (CNN) have made some progress in tasks such as tooth segmentation and caries detection. However, these methods are usually only for a single image modality (such as only X-ray images or only CBCT images), failing to fully utilize the complementary information of multimodal images (such as X-ray images, CBCT, and MRI), and the models have weak interpretability, making it difficult for doctors to understand the decision-making basis, resulting in a "black box" trust barrier in clinical applications.

[0003] Existing oral image recognition technologies suffer from insufficient utilization of information in single-modal analysis methods. The complementary features of different modal images (such as X-rays reflecting bone structure, CBCT showing three-dimensional anatomical details, and MRI showing soft tissue lesions) are not effectively integrated, resulting in low diagnostic accuracy for complex lesions (such as deep caries and periapical periodontitis). Summary of the Invention

[0004] This invention aims to at least address the technical problem of low diagnostic accuracy for complex lesions (such as deep caries and periapical periodontitis) in the prior art, and innovatively proposes a deep learning-based oral image recognition method, storage medium, and system.

[0005] To achieve the above-mentioned objectives of this invention, this invention provides a deep learning-based oral image recognition method, the method comprising: S1. Deploy a federated learning framework in multi-center medical institutions to collect multimodal oral imaging datasets and perform spatial alignment; the multimodal oral imaging datasets include oral X-ray, CBCT, and MRI image data; S2. Based on the multimodal oral image dataset, local features are extracted using a lightweight CNN branch to obtain image features. Based on the image features, a modality adaptive weight map is generated through a spatial attention module and a channel attention module. S3. Based on the modal adaptive weight map, cross-modal features are fused using a multi-head self-attention mechanism to generate a fused feature map, and then upsampled to the original image size through deconvolution to form a high-resolution multimodal feature representation; S4. Based on the high-resolution multimodal feature representation, a tooth segmentation mask is generated using the U-Net decoder. At the same time, the caries detection probability is output through global average pooling and fully connected layers to generate an initial diagnostic report containing segmentation and classification results. S5. Based on the initial diagnostic report, a heatmap is generated using the Grad-CAM algorithm, and the attention weights of each client are aggregated within the federated learning framework using the Paillier homomorphic encryption protocol to generate interpretable decision support data. S6. Based on the interpretable decision support data and combined with the clinical knowledge graph, natural language generation technology is used to transform heatmaps, attention weights, and image features into structured clinical reports.

[0006] In another aspect, the present invention also provides a computer-readable storage medium comprising: A memory on which computer programs are stored; A processor is configured to execute the program in the memory to implement the deep learning-based oral image recognition method.

[0007] On the other hand, the present invention also provides a deep learning-based oral image recognition system, the system comprising: processor; Memory used to store processor-executable instructions; The processor is configured to implement the deep learning-based oral image recognition method when executing the executable instructions.

[0008] The beneficial effects of this invention are as follows: This invention significantly improves the diagnostic accuracy of complex oral lesions (such as deep caries and periapical periodontitis) through the synergistic optimization of multimodal image fusion and cross-modal attention mechanisms. Specifically, by using lightweight CNN branches combined with spatial-channel dual attention modules, local features of various modal images (X-ray, CBCT, MRI) can be accurately extracted and modality-adaptive weight maps can be generated, effectively capturing complementary information between different modalities. The multi-head self-attention mechanism enhances the feature expression ability of complex lesions (such as microfractures and early caries) through multi-dimensional weighted fusion of cross-modal features. The joint application of U-Net decoder and global pooling-fully connected layer achieves accurate tooth segmentation while outputting high-confidence caries detection probabilities through Softmax classifier, avoiding information redundancy and missed diagnosis risks in single-modal analysis. Furthermore, the combination of Grad-CAM heatmaps and Paillier homomorphic encrypted aggregation attention weights not only ensures privacy and security within the federated learning framework but also enhances the interpretability of the model through visualized decision-making criteria. This allows doctors to intuitively understand the model's focus logic on lesion areas (such as the enamel-dentin junction), thereby improving clinical trust and diagnostic decision-making efficiency. It effectively solves the problems of insufficient diagnostic accuracy for complex lesions and inadequate utilization of multimodal information in existing technologies.

[0009] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0010] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is a flowchart of a deep learning-based oral image recognition method according to the present invention. Detailed Implementation

[0011] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0012] Example 1 like Figure 1 As shown, a deep learning-based oral image recognition method is described, the method comprising: S1. Deploy a federated learning framework in multi-center medical institutions to collect multimodal oral imaging datasets and perform spatial alignment; the multimodal oral imaging datasets include oral X-ray, CBCT, and MRI image data; In step S1, it is necessary to explain in detail that, in the process of deploying the federated learning framework in multi-center medical institutions, it is essential to first ensure data privacy and security among the institutions. By employing differential privacy technology to perturb the raw image data, it becomes impossible to trace specific patient information when the data is uploaded to the central server. Simultaneously, blockchain technology is used to record data access logs, ensuring transparency and auditability in the data usage process.

[0013] Next, when acquiring multimodal oral imaging datasets, preprocessing of images generated by different devices is required. Standardized workflows are designed for the characteristics of X-ray, CBCT, and MRI image data. For example, contrast enhancement and noise suppression are performed on X-ray images; voxel correction and artifact removal are performed on CBCT images; and grayscale uniformity and boundary sharpness are optimized for MRI images.

[0014] In the spatial alignment stage, existing registration algorithms based on feature point matching are used, combined with affine transformation and nonlinear deformation field, to unify images of different modalities into the same spatial coordinate system, thereby achieving accurate alignment of multimodal images on anatomical structures.

[0015] S2. Based on the multimodal oral image dataset, local features are extracted using a lightweight CNN branch to obtain image features. Based on the image features, a modality adaptive weight map is generated through a spatial attention module and a channel attention module. S3. Based on the modal adaptive weight map, cross-modal features are fused using a multi-head self-attention mechanism to generate a fused feature map, and then upsampled to the original image size through deconvolution to form a high-resolution multimodal feature representation; S4. Based on the high-resolution multimodal feature representation, a tooth segmentation mask is generated using the U-Net decoder. At the same time, the caries detection probability is output through global average pooling and fully connected layers to generate an initial diagnostic report containing segmentation and classification results. S5. Based on the initial diagnostic report, a heatmap is generated using the Grad-CAM algorithm, and the attention weights of each client are aggregated within the federated learning framework using the Paillier homomorphic encryption protocol to generate interpretable decision support data. S6. Based on the interpretable decision support data and combined with the clinical knowledge graph, natural language generation technology is used to transform heatmaps, attention weights, and image features into structured clinical reports.

[0016] In this embodiment, a deep learning-based oral image recognition method works as follows: First, diverse oral image data is collected through collaboration among multiple medical institutions, providing rich training material for the model. Based on this, a lightweight convolutional neural network (CNN) is used to extract features from images of different modalities, ensuring the capture of local details while reducing computational resource consumption. The introduction of spatial attention and channel attention modules further enhances the model's ability to focus on key regions and generates weight maps adapted to different modal characteristics, thereby improving the accuracy and robustness of feature representation.

[0017] In the cross-modal feature fusion stage, the multi-head self-attention mechanism plays a crucial role. By weighted fusion of features from different modalities, the model can uncover potential correlations across multiple dimensions, enhancing its ability to represent complex lesion regions. The application of deconvolution upsampling technology restores the fused feature map to the original image size, preserving more spatial detail information.

[0018] Finally, the U-Net decoder achieves precise tooth segmentation, and by combining global average pooling and fully connected layer operations, the probability distribution of caries detection is output. This process not only realizes end-to-end mapping from image to diagnostic results, but also generates a preliminary diagnostic report by integrating segmentation and classification information.

[0019] As an optional embodiment of the present invention, optionally, in step S2, local features are extracted using a lightweight CNN branch based on the multimodal oral image dataset, and then a modality-adaptive weight map is generated through a spatial attention module and a channel attention module, including: S201. Based on the multimodal oral image dataset, a lightweight CNN branch is constructed using a depthwise separable convolution and inverse residual structure to independently extract local features from each modality of the multimodal oral image dataset and obtain an initial image feature map. The expression for the lightweight CNN branch is: in, This represents the initial image feature map. This represents the overall transformation function for the lightweight CNN branch. This indicates the input multimodal oral imaging data (oral X-ray, CBCT, or MRI). This represents a 1×1 convolution operation, used for channel dimensionality increase or decrease (the "expansion" or "compression" operation in the inverse residual structure). This represents the activation function. Indicates batch normalization, This represents depthwise separable convolution. This indicates a residual connection, which will take the input... Adding it to the processed features alleviates the gradient vanishing problem (the core of the inverse residual structure); In step S201, it is important to explain in detail that the introduction of depthwise separable convolution significantly reduces computational complexity when constructing the lightweight CNN branch, while retaining the ability to effectively extract local features. Specifically, depthwise separable convolution decomposes the traditional convolution operation into two steps: channel-wise convolution and pointwise convolution, thereby greatly reducing the number of parameters and computational cost. Furthermore, the inverse residual structure enhances the richness of feature representation by first expanding the number of channels and then compressing them, while also avoiding the gradient vanishing problem in deep networks through residual connections, ensuring stable model training.

[0020] S202. Based on the initial image feature map, global average pooling and global max pooling operations are performed on each modality feature map through the spatial attention module to generate a spatial location importance descriptor, and a spatial weight map is generated through a convolutional layer and activation function. The expression for the spatial attention module is: in, This represents the output spatial weight map, with dimensions H×W×1, which represents the importance score for each spatial location. This represents the Sigmoid activation function. This represents a 1×1 convolution operation, used to compress the number of channels after concatenation (from 2C to 1), while introducing non-linearity. This represents global average pooling, which takes the average value of each channel in the feature map along its spatial dimension to generate channel-level statistics. This represents the initial image feature map. This represents global max pooling, which takes the maximum value of the spatial dimension for each channel of the feature map and generates channel-level significance statistics. In step S202, it is important to explain in detail that the spatial attention module is designed to capture the differences in importance between different spatial locations in the feature map. Through global average pooling and global max pooling operations, global statistical information and local saliency information of the feature map are extracted, respectively, thereby generating more representative spatial location descriptors. These descriptors, after being processed by convolutional layers, effectively compress the channel dimension while preserving key spatial distribution features. The introduction of the Sigmoid activation function further normalizes the weight values ​​to between 0 and 1, ensuring that the spatial weight map can intuitively reflect the importance score of each location. This method enhances the model's ability to focus on lesion regions.

[0021] Furthermore, by performing a 1×1 convolution operation on the pooling results, high feature representation ability can be maintained while reducing computational overhead. This efficient structure allows the module to be flexibly embedded into multimodal image processing workflows without significantly increasing the overall computational burden. Additionally, the generation process of the spatial weight map is highly interpretable, helping physicians understand the model's decision-making basis in the spatial dimension, thereby enhancing trust in clinical applications.

[0022] S203. Based on the initial image feature map, the multimodal feature map is stacked and global average pooled through the channel attention module to generate a channel correlation descriptor, and then a channel weight vector is generated through a fully connected layer and an activation function. The expression for the channel attention module is: in, This represents the output channel weight vector, with a dimension of C×1, which is the importance score of each channel. This represents the Sigmoid activation function. and This represents the weight matrix of the fully connected layer. This reduces the number of channels from C to C / r (reducing the computational load). This restores the number of channels from C / r to C. This represents the activation function. Indicates global average pooling. Represents the initial image feature map; In step S203, it is necessary to explain in detail that the channel attention module compresses the spatial dimension of each channel into a single value through a global average pooling operation, thereby generating channel-level statistics. These statistics, after dimensionality reduction and expansion processing by fully connected layers, can effectively capture the dependencies between channels while reducing computational complexity. The introduction of the Sigmoid activation function further normalizes the channel weights, ensuring that the importance score of each channel is within a reasonable range. This method enhances the model's ability to focus on key feature channels. In addition, by visualizing the generation process of the channel weight vectors, the model's decision-making logic at the channel dimension can be intuitively displayed, thereby improving the model's interpretability and clinical application value. This mechanism enables doctors to better understand how the model selects and utilizes feature information from different modalities of images, thereby enhancing the trust and reliability of diagnosis.

[0023] S204. Based on the spatial weight map and channel weight vector, the spatial and channel information are fused using existing broadcast multiplication to generate intermediate weight maps for each modality. The intermediate weight maps of all modalities are then spliced ​​together and normalized to generate the modality adaptive weight map.

[0024] In step S204, it is important to explain in detail that during the generation of the modality adaptive weight map, the existing broadcast multiplication operation enables the spatial weight map and channel weight vectors to fuse information across different dimensions. This fusion method fully utilizes the outputs of the spatial attention module and the channel attention module, ensuring that the feature map of each modality receives precise weighting based on its spatial location and channel importance. In this way, the model can dynamically adjust the level of attention given to different modal features, thereby improving its sensitivity to key regions and key channels.

[0025] After stitching together the intermediate weight maps of all modalities, normalization further enhances the consistency and comparability of the weight maps. Normalization not only avoids information imbalance caused by differences in weight value ranges but also makes the weight distribution among modalities smoother and more stable. Furthermore, the generation process of the modality-adaptive weight map is highly flexible and versatile, capable of adapting to different types of multimodal image data. Whether in dental X-rays, CBCT, or MRI images, this method can automatically adjust the weight allocation strategy according to data characteristics, thereby ensuring the robustness and applicability of the model in diverse application scenarios. This adaptive capability improves the model's performance.

[0026] As an optional embodiment of the present invention, optionally, in step S3, based on the modality adaptive weight map, cross-modal features are fused using a multi-head self-attention mechanism to generate a fused feature map, and then upsampled to the original image size through deconvolution to form a high-resolution multimodal feature representation, including: S301. Based on the modality adaptive weight map, it is input into multiple attention heads of the multi-head self-attention mechanism. Each attention head adaptively weights different modality features to capture the complex correlation between different modality features and generate preliminary fusion features corresponding to each attention head. In step S301, it is necessary to explain in detail that the multi-head self-attention mechanism is designed to mine potential correlations between modalities from different subspaces through multiple independent attention heads. Each attention head, through weighted processing of the modality-adaptive weight map, can focus on specific feature combinations, thereby capturing more detailed and diverse cross-modal information. This parallel processing approach not only enhances the model's ability to represent complex lesion regions but also improves the comprehensiveness and robustness of feature fusion. Furthermore, the preliminary fusion features generated by each attention head, after further integration, can form a more representative global feature representation.

[0027] S302. The preliminary fusion features generated by each attention head are spliced ​​together to integrate the cross-modal feature information captured by different attention heads and form a comprehensive cross-modal fusion feature. In step S302, it is necessary to explain in detail that the concatenation operation merges the preliminary fusion features generated by each attention head along the channel dimension, thereby forming a comprehensive cross-modal fusion feature. This integration method not only preserves the local detail information captured by each attention head, but also enhances the global expressive power through multi-view feature combination. To further optimize the feature representation, the concatenated feature map undergoes a 1×1 convolution operation to adjust the number of channels and enhance the non-linear expressive power of the features. This method ensures that the complementarity between different modal features is fully utilized, while avoiding the problem of excessive computational burden caused by increasing the number of channels. In addition, by batch normalizing the concatenation result, the numerical instability that may occur during training can be effectively alleviated, thereby improving the convergence speed and stability of the model.

[0028] S303. Based on the comprehensive cross-modal fusion features, through feature integration and optimization operations within the multi-head self-attention mechanism, the interaction and fusion between cross-modal features are enhanced to generate a fusion feature map; In step S303, it is necessary to explain in detail that the feature integration and optimization operations within the multi-head self-attention mechanism further enhance the expressive power of cross-modal features. By introducing residual connections and layer normalization techniques, the model can effectively alleviate the gradient vanishing problem when processing comprehensive cross-modal fused features, while enhancing the interaction between features. Specifically, residual connections add the input features to the output features processed by multi-head self-attention, ensuring the continuity of information flow; while layer normalization reduces numerical fluctuations during training by standardizing each layer of the feature map, thereby improving the robustness of the model. In addition, to further enhance the fusion effect between features, the model also introduces non-linear activation functions, such as ReLU or GELU, to enhance the non-linearity of feature representation. This design enables efficient integration of cross-modal features within a unified framework.

[0029] Then, after completing the deep integration of cross-modal features, the model restores the fused feature map to the original image size using deconvolution upsampling. In this process, the deconvolution operation learns the upsampling parameters, gradually increasing the spatial resolution of the feature map while preserving its semantic information. To further optimize the upsampling results, the model also incorporates skip connections to fuse spatial detail information from lower-level feature maps with higher-level semantic information, thereby generating a more refined, high-resolution multimodal feature representation. This method not only significantly improves the model's spatial localization ability for lesion regions but also ensures the accuracy and consistency of the final output feature map in terms of detail representation.

[0030] S304. Based on the fused feature map, perform upsampling processing using deconvolution operation to restore the spatial resolution of the fused feature map until it is upsampled to the original image size, forming a high-resolution multimodal feature representation.

[0031] In step S304, it is necessary to explain in detail that the deconvolution operation gradually amplifies the spatial resolution of the fused feature map by learning the upsampling parameters, while preserving its semantic information. To further optimize the upsampling results, the model combines skip connection technology to fuse the spatial detail information of the low-level feature map with the high-level semantic information, thereby generating a more refined high-resolution multimodal feature representation. This method not only significantly improves the model's spatial localization ability of lesion regions but also ensures the accuracy and consistency of the final output feature map in terms of detail representation. In addition, by performing batch normalization on the feature map during the upsampling process, the numerical instability that may occur during training can be effectively alleviated, thereby improving the convergence speed and stability of the model. At the same time, the introduction of nonlinear activation functions further enhances the nonlinearity of feature representation, enabling the model to better capture complex lesion patterns.

[0032] As an optional embodiment of the present invention, optionally, in step S4, based on the high-resolution multimodal feature representation, a tooth segmentation mask is generated using a U-Net decoder, and simultaneously, the caries detection probability is output through global average pooling and a fully connected layer to generate an initial diagnostic report containing segmentation results and classification results, including: S401. Based on the high-resolution multimodal feature representation, the feature map is upsampled and convolved layer by layer using the U-Net decoder structure to restore spatial detail information and integrate multi-scale features to generate a layer-by-layer decoded feature map. In step S401, it is necessary to explain in detail that the U-Net decoder structure gradually restores the spatial resolution of the feature map through layer-by-layer upsampling and convolution operations, while integrating multi-scale feature information. This design can not only effectively restore the details in the image, but also capture the global and local features of the lesion area at different scales. Specifically, each decoder layer introduces skip connections on the basis of upsampling, fusing the low-level features extracted in the encoder stage with the current feature map, thereby enhancing the expressiveness of spatial details. In addition, the convolution operation further optimizes the expressive power of the feature map, ensuring that each decoded feature map has high semantic consistency and spatial accuracy.

[0033] S402. Based on the layer-by-layer decoded feature map, channel splicing is performed with the corresponding encoder layer features through skip connections to fuse low-level texture information and high-level semantic information, thereby generating a fused layer feature map. In step S402, it is important to explain in detail that skip connections aim to enhance the expressiveness of feature maps by fusing low-level texture information extracted from the encoder stage with high-level semantic information generated from the decoder stage. This fusion method not only effectively preserves the detailed information in the image but also achieves multi-scale feature complementarity at different levels. Specifically, the channel stitching operation merges the feature maps from the encoder and decoder along the channel dimension to form a more representative fused hierarchical feature map. To further optimize feature representation, the stitched feature map undergoes a convolution operation to adjust the number of channels and enhance the nonlinearity of the features. This method ensures a balance between low-level texture information and high-level semantic information while avoiding the increase in computational complexity caused by the increase in feature dimension. Furthermore, batch normalization of the fusion result can effectively alleviate numerical instability during training, thereby improving the convergence speed and robustness of the model.

[0034] S403. Based on the fused hierarchical feature map, use convolutional kernels to refine features and enhance edges, eliminate the semantic gap caused by splicing, and generate refined fused features. In step S403, it is important to explain in detail that the 1×1 convolutional kernel aims to further refine the fused layer feature map through local receptive fields, thereby eliminating semantic inconsistencies that may be introduced by channel splicing. This refinement operation not only enhances the expressive power of the feature map but also highlights the edge information of the lesion area, making the segmentation result more accurate and clear. Specifically, the convolutional kernel strengthens the contrast between the target area and the background by weighting the local regions of the feature map, while preserving important texture details. To further improve the edge enhancement effect, the model introduces non-linear activation functions, such as ReLU or LeakyReLU, to ensure that the feature representation remains robust in complex scenes. In addition, batch normalization of the feature map after the convolution operation can effectively reduce numerical fluctuations during training, thereby improving the stability and convergence efficiency of the model.

[0035] Following feature refinement and edge enhancement, the refined fused features generated by the model more accurately reflect the spatial distribution and semantic information of lesion areas in the image. This provides more discriminative input data for classification tasks using global average pooling and fully connected layers. In this way, the model can further improve the reliability of classification results while ensuring segmentation accuracy, thus providing more comprehensive support for clinical diagnosis.

[0036] S404. Based on the refined fusion features, the number of channels is adjusted to the category dimension through convolution, and a tooth segmentation mask is generated through the Sigmoid activation function; The expression for generating the tooth segmentation mask is: in, This represents a tooth segmentation mask, where each pixel value represents the probability that the location belongs to a tooth region (range [0,1]). This represents the Sigmoid activation function. This represents a 1×1 convolutional kernel weight matrix, used to adjust the number of input channels to the category dimension. Indicates the characteristics of refinement and fusion. This represents the bias term, which is added to the convolution output to introduce a trainable offset. In step S404, it is important to explain in detail that the introduction of the Sigmoid activation function enables the model to map refined and fused features to a probability space, thereby generating a tooth segmentation mask with clear semantic information. This design ensures that each pixel value accurately reflects its probability of belonging to a tooth region. Furthermore, the 1×1 convolutional kernel achieves effective compression and integration of multi-channel information through pointwise convolution operations on the feature map. This process not only reduces computational complexity but also enhances the model's ability to handle high-dimensional features.

[0037] S405. Based on the refined fusion features, select high semantic level feature maps for global average pooling, compress the spatial dimension and extract global feature vectors to generate a caries detection feature representation. In step S405, it is necessary to explain in detail that the global average pooling operation compresses the spatial dimension of the high semantic level feature map, transforming the two-dimensional feature map of each feature channel into a single global feature value. This design not only effectively reduces data dimensionality but also retains the most representative global information in the feature map. Specifically, global average pooling generates a fixed-length global feature vector by averaging all spatial locations of each channel, thus providing highly abstract input data for subsequent classification tasks. Furthermore, to further improve the robustness of the feature representation, batch normalization is introduced after global average pooling to reduce potential numerical fluctuations during training. This method ensures the consistency and comparability of the global feature vector across different samples, while also improving the model's convergence speed and stability.

[0038] S406. Based on the dental caries detection feature representation, linear transformation and nonlinear activation are performed through a fully connected layer, and the classification probability distribution of dental caries and non-dental caries is output by a Softmax classifier to generate the dental caries detection probability. The expression for generating the probability of caries detection is: in, This represents the probability distribution of caries detection, containing probability values ​​for two categories: "caries" and "non-caries" (range [0,1], and sum to 1). This represents the Softmax activation function. This represents the weight matrix of the fully connected layer, with a dimension of 2×D, used to map the input feature dimension D to the output class dimension 2. This indicates the characteristics of dental caries detection. This represents the bias vector of the fully connected layer, introducing a trainable offset to enhance the model's expressive power; In step S406, it is important to explain in detail that the introduction of the Softmax activation function enables the model to map the dental caries detection feature representation to a probability space, thereby generating a probability distribution with clear classification significance. This design ensures the interpretability of the output results across categories, while further enhancing the discriminative power of the feature representation through linear transformations and nonlinear activation operations in the fully connected layer. Specifically, the fully connected layer introduces additional degrees of freedom by weighted summation of the input features and incorporating a bias term, allowing the model to flexibly adapt to the characteristics of different samples. Furthermore, the addition of the nonlinear activation function improves the model's ability to learn complex patterns, making it more robust when handling high-dimensional features. To optimize the training process, batch normalization is introduced after the fully connected layer to reduce numerical fluctuations and accelerate convergence. This method not only improves the accuracy of the classification task but also ensures that the model has good generalization performance when faced with diverse clinical data.

[0039] S407. Based on the tooth segmentation mask and the caries detection probability, integrate the spatial location information and lesion probability information to generate an initial diagnostic report containing segmentation results and classification results.

[0040] In step S407, it is necessary to explain in detail that the initial diagnostic report generation process requires the effective integration of the tooth segmentation mask and the caries detection probability, thereby providing clinicians with intuitive and comprehensive diagnostic information. Specifically, the tooth segmentation mask clearly marks the distribution of tooth regions in the image through pixel-level spatial location information, while the caries detection probability provides the classification confidence of the lesion region through the analysis of global features. To achieve seamless integration of the two, the model first spatially indexes each tooth region in the segmentation mask and matches it with the corresponding caries detection probability. This matching mechanism ensures that each tooth region can obtain its associated lesion probability information, thereby avoiding diagnostic errors caused by information fragmentation.

[0041] Furthermore, to enhance the readability and usability of the initial diagnostic report, the model incorporates a visualization module, overlaying segmentation and classification results onto the original image in the form of heatmaps or bounding boxes. This method not only visually displays the spatial distribution of lesion areas but also uses color depth or numerical labels to reflect the probability of lesions, thus helping dentists quickly locate potential caries areas. In addition, to meet the needs of different application scenarios, the model also supports generating structured text reports, detailing the probability of lesions, segmentation accuracy, and possible diagnostic suggestions for each tooth region. This multimodal output approach improves the information density of the diagnostic report.

[0042] At the technical implementation level, to ensure the accuracy and consistency of the initial diagnostic report, the model implements rigorous quality control for key steps in the integration process. For example, during the matching of segmentation masks and detection probabilities, the model uses a cross-validation mechanism to perform secondary verification of the lesion probability for each tooth region, eliminating misjudgments caused by local noise or insufficient feature representation. Simultaneously, to improve report generation efficiency, the model employs a parallel computing strategy, processing the results of segmentation and classification tasks simultaneously, thereby significantly shortening the overall runtime. This design guarantees the real-time nature of the diagnostic report.

[0043] As an optional embodiment of the present invention, optionally, in step S5, based on the initial diagnostic report, a heatmap is generated using the Grad-CAM algorithm, and the attention weights of each client are aggregated within the federated learning framework using the Paillier homomorphic encryption protocol to generate interpretable decision support data, including: S501. Based on the caries detection probability and tooth segmentation mask in the initial diagnostic report, select the target category, calculate the gradient information of the refined fusion feature corresponding to the target category through backpropagation, and generate a gradient weight matrix. In step S4501, it is necessary to explain in detail that the Grad-CAM algorithm generates a gradient weight matrix by calculating the gradient information of the refined fusion features corresponding to the target category. Specifically, during backpropagation, the model first locks the output node of the target category and calculates its derivative to obtain the gradient value at each position in the refined fusion feature map. These gradient values ​​reflect the degree of contribution of each region in the feature map to the prediction of the target category. To further refine key information, the model performs a global average pooling operation on the gradient values, thereby generating a gradient weight matrix corresponding to the number of channels in the feature map. This design can not only effectively capture the key features of the lesion area, but also preserve the integrity of spatial information at different scales.

[0044] S502. Based on the gradient weight matrix, perform global average pooling on the refined fusion features to generate a category-related feature activation map, and retain the positive activation region through the ReLU activation function to generate a preliminary heatmap. In step S502, it is necessary to explain in detail that the category-related feature activation map is generated by performing global average pooling on the refined fused features, compressing the feature response of each channel into a single value, thereby generating a one-dimensional feature vector highly correlated with the target category. This process not only significantly reduces the data dimensionality but also retains global information crucial for the classification task. Subsequently, the ReLU activation function is introduced to filter out negative activation regions, retaining only the positive activation parts, thus generating a preliminary heatmap. This design ensures that the heatmap can focus on the key features of the lesion area while eliminating interference from irrelevant background information. Furthermore, to further enhance the spatial resolution of the heatmap, the model performs bilinear interpolation upsampling after generating the preliminary heatmap to match its spatial dimensions with the original image.

[0045] S503. Based on the preliminary heat map, the image size is upsampled to the original image size by bilinear interpolation and superimposed on the original oral cavity image to generate a visual heat map. In step S503, it is necessary to explain in detail that the process of generating the visualized heatmap requires upsampling the initial heatmap to the original image size using bilinear interpolation to ensure that it matches the spatial resolution of the input data. This upsampling operation not only restores the detailed information of the heatmap but also visually aligns it precisely with the original oral image. Subsequently, the model overlays the upsampled heatmap with the original image to generate the final visualized heatmap. To enhance interpretability, heatmaps typically employ a gradient color mapping scheme, such as a transition from cool to warm tones, to reflect the importance of different regions in predicting the target category. This method can intuitively highlight the key features of the lesion area while preserving the structural information of the original image.

[0046] S504. Within the federated learning framework, each client uses the Paillier public key to encrypt the local attention weights and generate an encrypted weight vector. In step S504, it is important to explain in detail that the introduction of the Paillier homomorphic encryption protocol ensures that the local attention weights of each client within the federated learning framework can be securely aggregated without leaking the original data. Specifically, each client first encrypts the locally computed attention weight vector using the Paillier public key, generating a corresponding encrypted weight vector. This encryption method possesses additive homomorphic properties, allowing direct addition or scalar multiplication of encrypted data in ciphertext without decryption, thus protecting the data privacy of each client. Furthermore, to further improve encryption efficiency, the model employs a block processing strategy during encryption, decomposing the high-dimensional attention weight vector into multiple low-dimensional sub-vectors for separate encryption, thereby significantly reducing the computational overhead of a single encryption operation. This method ensures security while also considering the performance requirements of practical applications.

[0047] S505. Based on the encrypted weight vector, perform weight aggregation in the encrypted state on the federated learning server to generate a global attention weight distribution. In step S505, it is necessary to explain in detail that the federated learning server generates a global attention weight distribution by performing weighted aggregation of the encrypted weight vectors uploaded by each client in their encrypted state. Since the Paillier encryption algorithm supports homomorphic addition, the server can directly perform addition operations on the encrypted data without decrypting the original information. This design not only avoids the risk of sensitive data leakage but also ensures that the global model can integrate the local feature representation capabilities of each client. To enhance the robustness of the aggregation result, the model introduces a dynamic weight adjustment mechanism during the weight aggregation process, assigning different weight coefficients to each client based on its data quality or sample size. For example, clients with higher data quality or larger sample sizes are given higher weights to improve the generalization ability of the global model. Furthermore, to avoid the influence of outliers on the aggregation result, the server also employs a truncated mean algorithm, further improving the stability and reliability of the global attention weight distribution by removing extreme values.

[0048] S505: The central server receives the encrypted weight vectors from each client, aggregates the encrypted weights through homomorphic encryption addition operations, and generates an aggregated encrypted weight vector. In step S505, it is necessary to explain in detail that after receiving the encrypted weight vectors uploaded by each client, the central server directly aggregates these ciphertext data using the additive property of Paillier homomorphic encryption. This process requires no decryption, ensuring data privacy and security, while simultaneously achieving effective integration of multi-source local features by the global model. To further optimize the aggregation results, the server introduces a weight allocation strategy based on data quality assessment, dynamically adjusting the contribution ratio of each client. For example, clients whose data distribution is closer to the global features or whose annotation quality is higher will be assigned higher weight coefficients to improve the accuracy and generalization ability of the global attention weight distribution.

[0049] S506. Based on the aggregated encrypted weight vector, decryption is performed using the Paillier private key to generate plaintext aggregated attention weights. Combined with the visualized heatmap, interpretable decision support data is formed.

[0050] In step S506, it is important to explain in detail that the introduction of the Paillier private key enables the central server to securely decrypt the aggregated encrypted weight vector, thereby generating the aggregated attention weights in plaintext form. This process strictly adheres to the security specifications of homomorphic encryption protocols, ensuring that the final aggregation result can only be accessed under authorized conditions. The decrypted aggregated attention weights not only reflect the importance of local features of each client but also integrate key information from multi-source data from a global perspective. Combined with a visual heatmap, the model further maps the aggregated attention weights onto the spatial domain of the original image, thus forming interpretable decision support data. This data format can intuitively display the distribution of lesion areas and their corresponding attention weights, helping clinicians quickly understand the model's reasoning logic. In addition, to enhance the practicality of the decision support data, the model also supports interactive analysis functions, allowing users to dynamically adjust the threshold of attention weights or focus on specific areas to meet personalized diagnostic needs.

[0051] As an optional embodiment of the present invention, optionally, in step S6, based on the interpretable decision support data and combined with a clinical knowledge graph, the heatmap, attention weights, and image features are transformed into a structured clinical report using natural language generation technology, including: S601. Based on the heatmap highlight area coordinates and attention weight distribution in the interpretable decision support data, the data parsing module is used to extract the lesion location coordinate set and severity quantification value to generate initial structured data. The expression for the data parsing module is: in, Represents the set of coordinates of the lesion location. Indicates position Activation values ​​(range [0,1]) for the target category (e.g., dental caries). This represents the activation threshold (a preset hyperparameter, such as 0.7), used to filter significantly activated areas (highlighted lesion areas) in the heatmap to avoid noise interference. The severity quantification value (scalar, range [0,1]) is obtained by averaging the activation values ​​of each dimension within the lesion region using attention-weighted methods. Indicates the total number of attention heads. Indicates the first Weight values ​​for each attention head / dimension (e.g., modal importance, spatial region importance). Indicates the first Each attention head corresponds to a heatmap sub-component.

[0052] In step S601, it is necessary to explain in detail that the data parsing module achieves precise location of lesions and quantitative assessment of their severity by combining the coordinates of highlighted areas in the heatmap with the attention weight distribution. Specifically, the extraction of the lesion location coordinate set depends on the comparison of activation values ​​with preset thresholds, retaining only those significantly activated areas, thereby effectively filtering out background noise interference. Simultaneously, the calculation of the severity quantification value introduces a multi-head attention mechanism, ensuring that the assessment results comprehensively reflect the global characteristics of the lesion area by weighted averaging of weights from different dimensions. This design improves the accuracy of the initial structured data.

[0053] S602. Based on the initial structured data, and combined with the entity mapping rules of the clinical knowledge graph, the lesion location coordinates are associated with standard oral anatomy terminology, and the severity quantification value is mapped to the clinical grading standard to generate standardized structured data. In step S602, it is necessary to explain in detail that the lesion location coordinates are associated with standard oral anatomy terminology through entity mapping rules, ensuring that each lesion area can be accurately described with a medically standardized anatomical name, such as "mesial surface of the right mandibular first molar" or "occlusal surface of the left mandibular second premolar." This mapping not only improves the professionalism and readability of the data but also facilitates subsequent integration and sharing with other medical systems. Simultaneously, the severity quantification value is further refined into specific clinical levels through preset grading standards, such as "mild caries," "moderate caries," or "severe caries," thus providing doctors with a more intuitive diagnostic basis. This grading mechanism, based on the distribution range and threshold setting of the quantification value, ensures the consistency and objectivity of the assessment results. Furthermore, to enhance the flexibility of standardized structured data, the model also supports user-defined mapping rules to adapt to the specific needs of different medical institutions. This design, while ensuring data standardization, also provides greater freedom for practical applications.

[0054] S603. Based on the standardized structured data, the template filling engine of the natural language generation module is used to generate a diagnostic conclusion paragraph in combination with semantic polishing rules, thereby generating a preliminary clinical report text. In step S603, it is important to explain in detail that, in this embodiment, the natural language generation module is specifically a high-efficiency text generation system based on a template-filling engine (such as the existing Jinja2 template engine or a custom domain-specific template system), capable of transforming standardized structured data into diagnostic conclusion paragraphs that conform to clinical norms. The template-filling engine first extracts key fields, such as lesion location, severity grade, and relevant anatomical terms, from a preset medical report template and embeds them into a fixed text framework. This design not only ensures the logic and completeness of the generated text but also significantly improves report writing efficiency. Furthermore, to further optimize language expression, the module introduces semantic polishing rules, adjusting sentence structure and vocabulary selection to make the diagnostic conclusions more closely resemble the writing habits of professional doctors. For example, the system automatically expands "severe caries exists on the mesial surface of the right lower first molar" to "severe caries lesions were detected on the mesial surface of the patient's right lower first molar; timely treatment intervention is recommended." This polishing mechanism significantly enhances the readability and authority of the report.

[0055] S604. Based on the tooth segmentation mask boundary information in the image features and the original image overlay data, a three-dimensional description of the lesion boundary is generated using a geometric analysis algorithm, and the preliminary clinical report text is embedded to generate a structured clinical report.

[0056] The expression for the geometric analysis algorithm is: in, This represents a three-dimensional descriptive point set representing the lesion boundary. This represents the boundary matrix of the tooth segmentation mask. This represents the original oral imaging data. This represents the coordinates of the lesion boundary in three-dimensional space. Represents the boundary pixel coordinates, Represents depth coordinates, This represents the depth estimation function. It represents the overall transformation function of the geometric analysis algorithm, encapsulates the entire process of boundary coordinate extraction, depth estimation and 3D point set generation, and ensures a logical closed loop from input (boundary matrix + original image) to output (3D point set).

[0057] In step S604, it is necessary to explain in detail that the geometric analysis algorithm achieves accurate 3D reconstruction of the lesion boundary through deep fusion of the tooth segmentation mask boundary information and the original image data. Specifically, the algorithm first extracts the boundary matrix from the segmentation mask, combines it with the spatial information of the original image, and uses a depth estimation function to calculate the depth value of each boundary pixel, thereby generating a set of coordinate points of the lesion boundary in 3D space. This design ensures high accuracy of the 3D description. In addition, to further improve the visualization effect of the 3D description, the algorithm introduces a smoothing mechanism, using filtering techniques to remove noise interference from the boundary point set, making the generated 3D contour more closely match the actual lesion morphology. Finally, these optimized 3D descriptions are embedded in the preliminary clinical report text to form a complete structured clinical report.

[0058] Example 2 A computer-readable storage medium comprising: A memory on which computer programs are stored; A processor is configured to execute the program in the memory to implement a deep learning-based oral image recognition method as described in Embodiment 1.

[0059] It should be noted that the electronic device disclosed in this embodiment includes a processor and a memory for storing processor-executable instructions. The processor is configured to implement any of the deep learning-based oral image recognition methods described above when executing the executable instructions.

[0060] It should be noted here that the number of processors can be one or more. Furthermore, the electronic device in this embodiment may also include input devices and output devices. The processor, memory, input devices, and output devices can be connected via a bus or other means, without specific limitations herein.

[0061] As a computer-readable storage medium, the memory can be used to store software programs, computer-executable programs, and various modules, such as the program or module corresponding to the deep learning-based oral image recognition method of this disclosure. The processor executes various functional applications and data processing of the electronic device by running the software program or module stored in the memory.

[0062] Input devices can be used to receive input digital numbers or signals. These signals can be key signals related to user settings and function control of the device / terminal / server. Output devices can include display devices such as screens.

[0063] Example 3 A deep learning-based oral image recognition system includes: processor; Memory used to store processor-executable instructions; The processor is configured to implement a deep learning-based oral image recognition method when executing executable instructions.

[0064] It should be noted that the computer device includes a processor, a memory, and may also include one or more of a multimedia component, an input / output (I / O) interface, and a communication component.

[0065] The processor controls the overall operation of the computer device to complete all or part of the steps in the deep learning-based oral image recognition method described above.

[0066] Memory is used to store various types of data to support the operation of the computer device. This data may include, for example, instructions for any application or method used to operate on the computer device, as well as application-related data. Memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0067] The multimedia component may include a screen and an audio component, wherein the screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals; for example, the audio component may include a microphone for receiving external audio signals, the received audio signals may be further stored in memory or transmitted via a communication component; the audio component may also include at least one speaker for outputting audio signals.

[0068] I / O interfaces provide interfaces between the processor and other interface modules, such as keyboards, mice, buttons, etc.; these buttons can be virtual buttons or physical buttons.

[0069] The communication component is used for wired or wireless communication between the computer device and other devices; wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G or 5G, or one or more combinations thereof, and the corresponding communication component may include: Wi-Fi module, Bluetooth module, NFC module, mobile communication module.

[0070] As a preferred embodiment, the computer device may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to execute the above-described deep learning-based oral image recognition method.

[0071] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A deep learning-based oral image recognition method, characterized in that, The method includes: A federated learning framework was deployed in a multi-center medical institution to collect and spatially align a multimodal oral imaging dataset; the multimodal oral imaging dataset included oral X-ray, CBCT, and MRI images. Based on the multimodal oral image dataset, local features are extracted using a lightweight CNN branch to obtain image features. Based on the image features, a modality-adaptive weight map is generated through a spatial attention module and a channel attention module. Based on the modality adaptive weight map, cross-modal features are fused using a multi-head self-attention mechanism to generate a fused feature map, which is then upsampled to the original image size through deconvolution to form a high-resolution multimodal feature representation. Based on the high-resolution multimodal feature representation, a tooth segmentation mask is generated using a U-Net decoder. At the same time, the caries detection probability is output through global average pooling and fully connected layers to generate an initial diagnostic report containing segmentation and classification results. Based on the initial diagnostic report, a heatmap is generated using the Grad-CAM algorithm, and the attention weights of each client are aggregated within the federated learning framework using the Paillier homomorphic encryption protocol to generate interpretable decision support data. Based on the interpretable decision support data, combined with clinical knowledge graphs, natural language generation technology is used to transform heatmaps, attention weights, and image features into structured clinical reports.

2. The oral image recognition method based on deep learning as described in claim 1, characterized in that, Generating a modal adaptive weight graph includes: Based on the multimodal oral image dataset, a lightweight CNN branch is constructed using a depthwise separable convolution and inverse residual structure to independently extract local features from each modality of the multimodal oral image dataset and obtain an initial image feature map. Based on the initial image feature map, global average pooling and global max pooling operations are performed on each modality feature map through the spatial attention module to generate a spatial location importance descriptor, and then a spatial weight map is generated through convolutional layers and activation functions. Based on the initial image feature map, the multimodal feature map is stacked and globally averaged using the channel attention module to generate a channel correlation descriptor, and then a channel weight vector is generated through a fully connected layer and an activation function. Based on the spatial weight map and channel weight vector, spatial and channel information are fused through broadcast multiplication to generate intermediate weight maps for each modality. The intermediate weight maps of all modalities are then concatenated and normalized to generate the modality adaptive weight map.

3. The oral image recognition method based on deep learning as described in claim 1, characterized in that, Forming high-resolution multimodal feature representations includes: Based on the modality adaptive weight map, it is input into multiple attention heads of the multi-head self-attention mechanism. Each attention head adaptively weights different modality features to capture the complex correlation between different modality features and generate preliminary fusion features corresponding to each attention head. The preliminary fusion features generated by each attention head are spliced ​​together to integrate the cross-modal feature information captured by different attention heads, forming a comprehensive cross-modal fusion feature. Based on the comprehensive cross-modal fusion features, the interaction and fusion between cross-modal features are enhanced through feature integration and optimization operations within the multi-head self-attention mechanism, thereby generating a fusion feature map; Based on the fused feature map, it is upsampled using deconvolution to restore the spatial resolution of the fused feature map until it is upsampled to the original image size, forming a high-resolution multimodal feature representation.

4. The oral image recognition method based on deep learning as described in claim 1, characterized in that, Generating an initial diagnostic report that includes segmentation and classification results includes: Based on the high-resolution multimodal feature representation, the U-Net decoder structure is used to perform layer-by-layer upsampling and convolution operations on the feature map to restore spatial detail information and integrate multi-scale features, thereby generating a layer-by-layer decoded feature map. Based on the layer-by-layer decoded feature map, channel splicing is performed with the corresponding encoder layer features through skip connections to fuse low-level texture information and high-level semantic information, thereby generating a fused layer feature map. Based on the fused hierarchical feature map, convolutional kernels are used to refine features and enhance edges, eliminating the semantic gap caused by splicing and generating refined fused features. Based on the refined fusion features, the number of channels is adjusted to the category dimension through convolution, and a tooth segmentation mask is generated by the Sigmoid activation function; Based on the refined fusion features, high semantic level feature maps are selected for global average pooling to compress the spatial dimension and extract global feature vectors to generate a caries detection feature representation. Based on the dental caries detection feature representation, a linear transformation and nonlinear activation are performed through a fully connected layer, and a Softmax classifier outputs the classification probability distribution of dental caries and non-dental caries, generating the dental caries detection probability. Based on the tooth segmentation mask and the caries detection probability, spatial location information and lesion probability information are integrated to generate an initial diagnostic report containing segmentation results and classification results.

5. The oral cavity image recognition method based on deep learning as described in claim 1, characterized in that, Generating interpretable decision support data includes: Based on the caries detection probability and tooth segmentation mask in the initial diagnostic report, a target category is selected, and the gradient information of the refined fusion feature corresponding to the target category is calculated through backpropagation to generate a gradient weight matrix. Based on the gradient weight matrix, global average pooling is performed on the refined fusion features to generate a category-related feature activation map, and the positive activation region is retained by the ReLU activation function to generate a preliminary heatmap. Based on the preliminary heatmap, the image size is upsampled to the original image size using bilinear interpolation and then overlaid on the original oral cavity image to generate a visual heatmap. Within the federated learning framework, each client uses the Paillier public key to encrypt the local attention weights, generating an encrypted weight vector. The central server receives the encrypted weight vectors from each client, aggregates the encrypted weights through homomorphic encryption addition, and generates an aggregated encrypted weight vector. Based on the aggregated encrypted weight vector, the plaintext aggregated attention weight is generated by decryption using the Paillier private key. Combined with the visualized heatmap, interpretable decision support data is formed.

6. The oral image recognition method based on deep learning as described in claim 1, characterized in that, The use of natural language generation technology to transform heatmaps, attention weights, and image features into structured clinical reports includes: Based on the heatmap highlight area coordinates and attention weight distribution in the interpretable decision support data, the data parsing module is used to extract the lesion location coordinate set and severity quantification value to generate initial structured data. Based on the initial structured data, and combined with the entity mapping rules of the clinical knowledge graph, the lesion location coordinates are associated with standard oral anatomy terminology, and the severity quantification value is mapped to the clinical grading standard to generate standardized structured data. Based on the standardized structured data, the template filling engine of the natural language generation module is used to generate diagnostic conclusion paragraphs in combination with semantic polishing rules, thus generating preliminary clinical report text. Based on the tooth segmentation mask boundary information in the image features and the original image overlay data, a three-dimensional description of the lesion boundary is generated using a geometric analysis algorithm, which is then embedded into the preliminary clinical report text to generate a structured clinical report.

7. The oral image recognition method based on deep learning as described in claim 6, characterized in that, The expression for the data parsing module is: in, Represents the set of coordinates of the lesion location. Indicates position Activation value for the target category, Indicates the activation threshold. Indicates a quantitative value representing the severity. Indicates the total number of attention heads. Indicates the first Weight values ​​for each attention head / dimension. Indicates the first Each attention head corresponds to a heatmap sub-component.

8. The oral cavity image recognition method based on deep learning as described in claim 6, characterized in that, The expression for the geometric analysis algorithm is: in, This represents a three-dimensional descriptive point set representing the lesion boundary. This represents the boundary matrix of the tooth segmentation mask. This represents the original oral imaging data. This represents the coordinates of the lesion boundary in three-dimensional space. Represents the boundary pixel coordinates, Represents depth coordinates, This represents the depth estimation function. This represents the overall transformation function of the geometric analysis algorithm.

9. A computer-readable storage medium, characterized in that, include: A memory on which computer programs are stored; A processor for executing the program in the memory to implement a deep learning-based oral image recognition method according to any one of claims 1 to 8.

10. A deep learning-based oral image recognition system, characterized in that, The system includes: processor; Memory used to store processor-executable instructions; The processor is configured to implement the deep learning-based oral image recognition method according to any one of claims 1 to 8 when executing the executable instructions.

Images