Dental caries analysis method and system based on static data and panoramic image

Abstract

The invention discloses a decayed tooth analysis method and system based on static data and a panoramic film image. The method comprises the following steps: acquiring image data and static data of an oral panoramic film of a user; the static data is used for reflecting at least one kind of information of dietary habits, oral hygiene, social economic factors, medical history and family history of the user; generating a weighted fusion feature map according to the static data and the image data through a convolutional neural network model based on an attention mechanism; and carrying out decayed tooth analysis according to the fused feature map. Through the convolutional neural network based on the attention mechanism, the panoramic film data and the static data are effectively combined, the accuracy of decayed tooth analysis can be improved, and a basis is provided for prevention of early decayed teeth. The problem that in the prior art, a deep learning model used for decayed tooth analysis is limited to a single data source, and risk early warning is difficult to accurately carry out is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical image analysis and artificial intelligence, and in particular to a caries analysis method and system based on static data and panoramic film images. BACKGROUND

[0002] Early childhood caries (ECC) is one of the most common chronic oral diseases in infants and preschool children, with the characteristics of early onset, rapid progression and extensive destruction. If not timely intervention, not only will damage the structure and function of children's teeth, affect chewing, pronunciation and appearance, but also may cause pain, local infection and even systemic diseases, which will have a serious impact on the growth and development of children, and the quality of life and children's psychology.

[0003] Traditional ECC analysis mainly relies on clinical examination by pediatric dentists, but clinical examination largely depends on the professional knowledge and subjective experience of doctors, and it is easy to miss diagnosis for extremely small or hidden early demineralization lesions (white spots).

[0004] In recent years, with the rapid development of artificial intelligence technology, deep learning models represented by convolutional neural networks (CNN) have made breakthrough progress in medical image analysis. CNN can automatically learn and extract complex features from images. In the field of dentistry, some studies have begun to apply CNN to analyze dental X-ray films to assist in the diagnosis of caries, periodontal disease and periapical disease, etc.

[0005] However, deep learning models that rely solely on image data may be limited to a single data source, resulting in incomplete information utilization and ignoring the fundamental risk factors of disease occurrence, making it difficult to accurately conduct risk warning.

[0006] Therefore, the prior art still needs to be improved and developed. SUMMARY

[0007] The technical problem to be solved by the present application is to provide a caries analysis method and system based on static data and panoramic film images, which aims to solve the problem that the existing deep learning model for analyzing caries is limited to a single data source and is difficult to accurately conduct risk warning.

[0008] The technical solution adopted by the present application to solve the problem is as follows:

[0009] In a first aspect, the present application provides a caries analysis method based on static data and panoramic film images, which comprises:

[0010] Obtaining image data and static data of a panoramic radiograph of a user; the static data is used to reflect at least one of information of dietary habits, oral hygiene, socio-economic factors, medical history and family history of the user;

[0011] Generating a weighted fusion feature map according to the static data and the image data through a convolutional neural network model based on an attention mechanism;

[0012] Performing caries analysis according to the fusion feature map.

[0013] In an implementation manner, the static data is obtained based on a questionnaire survey and / or an electronic health record.

[0014] In an implementation manner, generating a weighted fusion feature map according to the static data and the image data through a convolutional neural network model based on an attention mechanism comprises:

[0015] Preprocessing the static data, comprising: converting data of different dimensions in the static data into static data features respectively; and splicing all the static data features to obtain a static data long vector;

[0016] Preprocessing the image data, comprising: performing a standardization operation on the image data to obtain processed image data; the standardization operation comprises at least one of size normalization, pixel value normalization and data enhancement;

[0017] Inputting the static data long vector and the processed image data into the convolutional neural network model to obtain the fusion feature map.

[0018] In an implementation manner, the convolutional neural network model comprises:

[0019] A dual-flow feature extraction module is configured to extract features from the processed image data to obtain an image feature map, and extract features from the static data long vector to obtain a risk feature vector;

[0020] An attention fusion module is configured to generate the fusion feature map according to the image feature map and the risk feature vector.

[0021] In an implementation manner, generating the fusion feature map according to the image feature map and the risk feature vector comprises:

[0022] Generating an attention map according to the risk feature vector through an attention generation network in the attention fusion module; each value in the attention map represents a weight of a corresponding position in the image feature;

[0023] The attention map and the image feature map are subjected to an element-by-element weighting operation to obtain the fusion feature map.

[0024] In an implementation, the caries analysis is performed according to the fusion feature map, including:

[0025] The fusion feature map is input into a prediction module; the prediction module includes a global average pooling layer, a fully connected layer, and an activation function layer.

[0026] The prediction module calculates a caries probability of the user based on the fusion feature map.

[0027] In an implementation, the caries analysis is performed according to the fusion feature map, including:

[0028] The fusion feature map is input into a positioning module.

[0029] The positioning module generates a positioning image according to the fusion feature map; the positioning image is used to reflect a predicted caries position, range, and confidence level, and the image form of the positioning image includes a heat map or a segmentation map.

[0030] In a second aspect, an embodiment of the present application further provides a caries analysis system based on static data and panoramic film images, including:

[0031] A data acquisition module is configured to acquire image data and static data of a panoramic film of a user; the static data is used to reflect at least one of the following information of the user: dietary habits, oral hygiene, socio-economic factors, medical history, and family history.

[0032] An attention module is configured to generate a weighted fusion feature map according to the static data and the image data by using a convolutional neural network model based on an attention mechanism.

[0033] A data analysis module is configured to perform caries analysis according to the fusion feature map.

[0034] In a third aspect, an embodiment of the present application further provides a terminal, including a memory and one or more processors; the memory stores one or more programs; the programs contain instructions for executing any of the above-mentioned caries analysis methods based on static data and panoramic film images; and the processors are configured to execute the programs.

[0035] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores a plurality of instructions; the instructions are adapted to be loaded and executed by a processor to implement the steps of any of the above-mentioned caries analysis methods based on static data and panoramic film images.

[0036] The beneficial effects of the present application: the embodiment of the present application obtains the image data and static data of the user's oral panoramic film; the static data is used to reflect at least one information of the user's eating habits, oral hygiene, socio-economic factors, medical history and family history; a convolutional neural network model based on an attention mechanism is used to generate a weighted fusion feature map according to the static data and the image data; and caries analysis is performed according to the fusion feature map. The present application effectively combines panoramic film data and static data based on a convolutional neural network based on an attention mechanism, which can improve the accuracy of caries analysis and provide a basis for early caries prevention. The problem that the deep learning model for analyzing caries in the prior art is limited to a single data source and is difficult to accurately perform risk warning is solved. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments described in the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0038] Figure 1 is a basic flow diagram of the caries analysis method based on static data and panoramic film image provided by the embodiment of the present application.

[0039] Figure 2 is a whole working flow and structure diagram of the convolutional neural network model based on an attention mechanism provided by the embodiment of the present application.

[0040] Figure 3 is a basic module diagram of the caries analysis system based on static data and panoramic film image provided by the embodiment of the present application.

[0041] Figure 4 is a principle block diagram of the terminal provided by the embodiment of the present application. DETAILED DESCRIPTION

[0042] The present application discloses a caries analysis method and system based on static data and panoramic film image. In order to make the purpose, technical scheme and effect of the present application more clear and explicit, the present application will be further described in detail with reference to the drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0043] As those skilled in the art will appreciate, unless otherwise specifically stated herein, the use of the singular herein, such as the use of "one" to describe a particular feature, indicates that the feature is inseparable from the other features described herein. Unless otherwise specifically stated herein, the use of the singular herein, such as the use of "one" to describe a particular feature, indicates that the feature is inseparable from the other features described herein. It should further be understood that the word "comprising" used in the specification of the application indicates the presence of the described features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or there can be intervening elements. In addition, "connected" or "coupled" as used herein can include wireless connection or wireless coupling. The phrase "and / or" as used herein includes all or any one of the associated listed items, and all combinations thereof.

[0044] As those skilled in the art will appreciate, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and unless specifically defined as such, should not be interpreted in an idealized or overly formal sense.

[0045] To overcome the above-mentioned defects of the prior art, the present application provides a caries analysis method based on static data and panoramic film images. The method acquires image data and static data of a user's oral panoramic film. The static data is used to reflect at least one of the user's eating habits, oral hygiene, socio-economic factors, medical history and family history. A convolutional neural network model based on attention mechanism is used to generate a weighted fusion feature map according to the static data and the image data. The fusion feature map is used for caries analysis. The present application effectively combines panoramic film data and static data based on a convolutional neural network with an attention mechanism, which can improve the accuracy of caries analysis and provide a basis for early caries prevention. The problem of the prior art that the deep learning model for analyzing caries is limited to a single data source and cannot accurately perform risk warning is solved.

[0046] As shown in Figure 1 the method comprises:

[0047] Step S100, acquiring image data and static data of a user's oral panoramic film; the static data is used to reflect at least one of the user's eating habits, oral hygiene, socio-economic factors, medical history and family history.

[0048] Specifically, the occurrence of caries is a complex process involving multiple risk factors. The risk assessment of ECC should consider multiple factors, which can be summarized as static data, mainly including:

[0049] Dietary habits: such as the frequency of intake of sugary foods and beverages, the habit of feeding with a bottle at night (especially sugary liquids), which are key factors leading to acid production by dental plaque and then eroding the enamel; oral hygiene habits: including the frequency of brushing teeth, whether fluoride toothpaste is used, the correctness of brushing method, and whether parents effectively supervise and help, which are directly related to the efficiency of dental plaque removal;

[0050] Socioeconomic factors: family income, parents' education level and oral health awareness determine the family's dietary structure, the importance of oral care, and the convenience of obtaining professional dental services (such as regular check-ups and fluoride application);

[0051] Medical and genetic history: children's medical history, long-term use of certain drugs (such as drugs that can cause dry mouth, which can reduce the buffering and cleaning ability of saliva), fluoride exposure history (such as the fluoride content of drinking water), and the caries history of family members (parents, siblings), which reveal the individual's susceptibility to caries from physiological, environmental and genetic perspectives.

[0052] In practical application scenarios, static data can be obtained through questionnaire surveys and / or electronic health records. This embodiment uses the image data of the user's oral panoramic film and the static data as analysis data together, avoiding the limitations of a single data source, thereby improving the accuracy of caries analysis.

[0053] Step S200, generating a weighted fusion feature map according to the static data and the image data through a convolutional neural network model based on an attention mechanism.

[0054] Specifically, the embodiment needs to use a convolutional neural network model (CNN) based on an attention mechanism, which is designed as a dual-flow input structure, one receiving and processing the image data of the oral panoramic film, and the other receiving and processing the multi-source heterogeneous static data obtained from the questionnaire survey or electronic health record. And using the attention fusion method, the individualized caries risk information contained in the structured static data is converted into a dynamic and guiding weight signal / attention signal, and acts on the extraction process of the image features by the convolutional neural network, thereby guiding the model to focus the computing resources on the specific tooth area of the panoramic film associated with high-risk factors (such as the upper anterior teeth area related to the habit of night feeding, or the molar sulcus area related to high-sugar diet), enhancing the sensitivity to early micro-lesion features, and finally obtaining image features fused with risk information, i.e. the weighted fusion feature map.

[0055] In an implementation, a weighted fusion feature map is generated according to the static data and the image data by a convolutional neural network model based on an attention mechanism, including:

[0056] The static data is preprocessed, including: converting data of different dimensions in the static data into static data features respectively; and splicing all the static data features to obtain a static data long vector.

[0057] The image data is preprocessed, including: performing a standardization operation on the image data to obtain processed image data; and the standardization operation includes at least one of size normalization, pixel value normalization, and data enhancement.

[0058] The static data long vector and the processed image data are input into the convolutional neural network model to obtain the fusion feature map.

[0059] The system corresponding to the embodiment includes a data input and preprocessing module, which is specifically configured to perform image data processing and static data processing. Then, the static data long vector obtained after preprocessing and the processed image data are input into a convolutional neural network model for fusion.

[0060] For image data processing: the input original image is image data of a user's panoramic oral film (for example, a standard child's panoramic oral film (Orthopantomogram, OPG) digital image), and a series of standardization operations are performed on the input original image to eliminate differences and enhance features. The standardization operations specifically include but are not limited to: size normalization, which is used to scale images of different sources and different resolutions to a uniform size (for example, 512x1024 pixels); pixel value normalization, which is used to normalize pixel values from a range to a range of or [-1, 1] to facilitate stable training of the neural network; and data enhancement, which is used to expand the training data set and improve the generalization ability of the model by techniques such as random rotation, translation, scaling, and contrast adjustment, to prevent overfitting.

[0061] For static data processing: the input data is structured static data, which is derived from a questionnaire survey or an electronic health record (EHR) and contains information of multiple dimensions such as the aforementioned dietary habits, oral hygiene habits, socio-economic factors, medical history, and family history. The heterogeneous static data is converted so that it can be processed by the neural network. Specifically, the operations include but are not limited to: for categorical variables (such as tooth brushing frequency: "once a day" and "twice a day"), one-hot encoding is used to convert them into binary vectors; for numerical variables (such as annual household income and age), standardization or normalization is used to make them on a similar numerical scale as other features; and all the processed static data features are spliced into a long vector as the input of the static data stream.

[0062] In an implementation, the convolutional neural network model comprises:

[0063] a dual-stream feature extraction module configured to extract features from the processed image data to obtain an image feature map and extract features from the static data long vector to obtain a risk feature vector;

[0064] an attention fusion module configured to generate the fusion feature map according to the image feature map and the risk feature vector.

[0065] Specifically, as shown in Figure 2 the overall workflow and structure of the convolutional neural network model based on the attention mechanism are shown. The dual-stream feature extraction module in this embodiment adopts a parallel dual-stream structure to process data of two different modalities respectively:

[0066] The image feature extraction stream (i.e., the processed image data) adopts a deep convolutional neural network (CNN) as the backbone network, such as ResNet, VGGNet or EfficientNet, etc. The CNN network receives the preprocessed panoramic image, extracts hierarchical spatial visual features from low-level (edge, texture) to high-level (tooth shape, crown structure) through multiple convolution layers, pooling layers and activation functions. The output is an image feature map, which is a three-dimensional tensor containing rich semantic information of the image at different spatial positions.

[0067] The static data feature extraction stream (i.e., the static data long vector) adopts a multi-layer perceptron network. The MLP receives the encoded static data long vector, maps it to a low-dimensional, dense risk feature vector through several fully connected layers and nonlinear activation functions (such as ReLU). This vector is an abstract and condensed representation of all risk factors in static data, capturing the nonlinear relationship between different factors and representing the comprehensive caries risk profile of the user (e.g., a child individual).

[0068] Then, the attention fusion module is used to convert the individualized caries risk information contained in the structured static data into a dynamic and guiding weight signal / attention signal, and apply it to the image feature extraction process of the convolutional neural network.

[0069] In another implementation, for the image feature extraction stream, the CNN backbone network adopted is not limited to ResNet, but can be any advanced CNN architecture, such as Inception series, DenseNet, SENet, or even a visual model based on Transformer (Vision Transformer, ViT).

[0070] In another implementation, for the static data feature extraction flow, in addition to using MLP, a more complex model such as Factorization Machines or AutoInt network can also be used to better automatically learn high-order cross features between static risk factors.

[0071] In an implementation, the fusion feature map is generated according to the image feature map and the risk feature vector, including:

[0072] An attention map is generated according to the risk feature vector by an attention generation network in the attention fusion module; each value in the attention map represents a weight of a corresponding position in the image feature;

[0073] The attention map and the image feature map are subjected to an element-wise weighting operation to obtain the fusion feature map.

[0074] Specifically, the attention map in the embodiment corresponds to an attention signal, which is a two-dimensional attention map (for example, a two-dimensional spatial attention map) corresponding to the spatial dimension of the image feature map. The embodiment adopts a static data adjustment convolutional neural network to extract the image feature, mainly including an element-wise weighting operation (for example, element-wise multiplication) of the attention map and the original image feature map, so as to convert the static risk information into spatial weights and use it to filter the visual features.

[0075] For example, the attention fusion module can realize intelligent and deep fusion of two modal information. The module is specifically used for: inputting the risk feature vector into a small attention generation network (usually composed of one or two convolutional layers or fully connected layers), and the function of the attention generation network is to "decode" or "up-sample" the abstract risk vector to generate a two-dimensional attention map. The size of the attention map is exactly the same as the spatial size of the image feature map output by the module B1. Each value in the attention map is between 0 and 1, representing a weight, and the geographical position of the value corresponds one-to-one to the position on the image feature map.

[0076] Specifically, the generation of the attention map adopts an adaptive weight distribution algorithm, as follows:

[0077] A(i,j) = softmax(tanh(W s × S + W i × F img (i,j) + b));

[0078] Wherein, A(i,j) represents the adaptive attention weight at position (i,j), S is the static data risk feature vector, Fimg (i,j) is the feature value of the corresponding position in the image feature map, W s and W i are the learning weight matrices of static data and image data respectively, b is the bias term, tanh is the hyperbolic tangent activation function, and softmax ensures weight normalization.

[0079] Then feature weighting is performed, specifically including: element-wise multiplication of the attention map and the image feature map. The essence of this operation is: using the attention map as a "weight mask" to weight the image feature map. The region with a higher value in the attention map will "enlarge" or "retain" the corresponding image feature; the region with a lower value will "suppress" or "ignore" the corresponding image feature. Finally, a fused feature map is generated, i.e., the fusion feature map is obtained. The fusion feature map not only contains the original visual information, but also is "intelligently guided" by the static risk data, highlighting the visual features of high-risk areas.

[0080] Specifically, the element-wise weighting operation of the attention map and the image feature map to obtain the fusion feature map is as follows:

[0081] F fused (i,j) = a x A(i,j) x F img (i,j) + β x G(S) + γ x Cross_Attention(F img (i,j), S);

[0082] Where F fused (i,j) represents the feature value at position (i,j) in the fusion feature map, a, β, γ are dynamic balance factors, G(S) is a global representation function of static features, Cross_Attention is a cross-modal attention mechanism, realizing deep interactive fusion of static data and image data.

[0083] It should be noted that the existing attention mechanism is usually self-attention within the model. The embodiment is a dynamic attention generation mechanism based on external risk factors. Specifically, the attention of the embodiment is generated and driven by external, image content-independent, and personalized static risk data (such as diet and hygiene habits). This makes the model analysis process have dynamic and individualized adaptability, and is no longer a static tool that uses the same analysis logic for all images, but an intelligent analysis process that can adjust its "observation focus" in real time according to the user's unique "medical history" and "background", and can realize personalized and accurate analysis.

[0084] And, existing multimodal fusion methods usually occur at the end of feature extraction (late fusion). But the embodiment adopts a deep fusion rather than a late fusion architecture. The deep fusion strategy injects risk information (in the form of attention maps) at the middle stage of image feature extraction. This approach allows risk information to directly intervene and modulate the extraction process of visual features, not affecting the decision at the end, but affecting the "observation" in the middle. This makes the fusion more profound and effective, and can explore subtle image features related to risk that may have been lost in late fusion.

[0085] In another implementation, in addition to spatial attention, the attention mechanism adopted by the attention fusion module of the embodiment can also include:

[0086] Channel attention: The risk feature vector can be used to generate a channel attention vector for weighting different feature channels of the image feature map. For example, some risk factors may be more related to texture features, while others are more related to contour features. Spatial and channel attention can be used simultaneously to form a hybrid attention module in parallel or in series.

[0087] Cross-attention: A more complex cross-attention mechanism in the Transformer style can be used. The risk feature vector is used as "query", and the features of each spatial position of the image feature map are used as "key" and "value". The similarity between the query and the key is calculated to determine how to weight and aggregate the value.

[0088] In another implementation, in addition to the deep fusion strategy of fusion in the middle of feature extraction, the fusion strategy adopted by the attention fusion module of the embodiment can also include:

[0089] Early fusion: Instead of complex attention fusion in the middle of the network, the two types of information are combined at the very front end of data input. Implementation: All static data is encoded into a vector. This vector is "visualized" in some way, for example, it is copied and tiled into a single-channel layer with the same size as the panoramic image. The "risk layer" is spliced with the R, G, B (or grayscale) channels of the original panoramic image to form an input tensor with additional channels (such as 4 channels). Then this fused tensor is input into a standard, single CNN network for end-to-end training and analysis;

[0090] Late fusion: the image stream (i.e. the video data of the user's oral panoramic film) and the static data stream (i.e. the static data of the user) can be allowed to make preliminary predictions independently, and then the two prediction results are fused. For example, the probability of caries output by the CNN based on the image stream and the risk score output by the MLP based on the static data stream are finally fused through a weighted algorithm or a small neural network.

[0091] Multi-stage fusion: static data guided attention can be injected multiple times at different depth levels of the CNN (e.g. shallow, middle, deep) to achieve more fine-grained cross-modal information modulation.

[0092] Step S300, performing caries analysis according to the fusion feature map.

[0093] Specifically, the fusion feature map of the embodiment fuses the risk information provided by the static data and the image features provided by the oral panoramic film, so that the user's caries probability and the suspected caries position and range of the user can be accurately analyzed by using the fusion feature map.

[0094] In an implementation manner, the caries analysis according to the fusion feature map comprises:

[0095] The fusion feature map is input into a prediction module; the prediction module comprises a global average pooling layer, a fully connected layer and an activation function layer;

[0096] The prediction module calculates the caries probability of the user based on the fusion feature map.

[0097] Specifically, after obtaining the fusion feature map, the final data analysis task can be performed. One type of task is classification prediction or probability prediction: the fusion feature map is input into a classification head (usually composed of a global average pooling layer and a fully connected layer), and finally a Sigmoid or Softmax activation function is used to output one or more probability values representing the overall probability of caries. For example, output "the probability of having ECC is 95%".

[0098] In another implementation manner, the caries analysis according to the fusion feature map comprises:

[0099] The fusion feature map is input into a positioning module;

[0100] The positioning module generates a positioning image according to the fusion feature map; the positioning image is used to reflect the predicted caries position, range and confidence, and the image form of the positioning image comprises a heat map or a segmentation map.

[0101] Specifically, another task type is the location prediction: in order to provide more detailed analysis information, the fused feature map is input to a target location / segmentation head (for example, using an upsampling decoder structure similar to that in U-Net). The output of this module is a heat map or segmentation map of the same size as the original input image, and the highlighted area in the map accurately indicates the location, range and confidence of the caries predicted by the model, realizing pixel-level target location. In short, the caries analysis result not only includes the classification judgment of the overall caries probability, but also includes a positioning map (such as a heat map or a segmentation map) indicating the suspected caries location and range on the panoramic image.

[0102] Variations of the present application:

[0103] 1. Unified modeling scheme based on Transformer: abandon the heterogeneous architecture of CNN+MLP, and use the current most advanced Transformer model to uniformly process image and structured data. In other words, the powerful global relationship modeling capability of Transformer can be used to realize the deep fusion of cross-modal information.

[0104] For example, for image segmentation: the panoramic image is divided into a series of non-overlapping small image blocks, and each image block is linearly mapped to an "image token";

[0105] For data segmentation: each static risk factor (such as "night feeding habit: yes") is also regarded as an independent "data token";

[0106] All image tokens and data tokens are spliced into a sequence and input into a large multi-modal Transformer encoder; the self-attention mechanism inside the Transformer will automatically learn the complex relationships between all tokens in the sequence, including the relationships between image blocks, between image blocks and risk factors, and between risk factors.

[0107] 2. Scheme based on graph neural network (GNN): abstract the oral environment as a "graph" structure instead of a pixel matrix, which better reflects the spatial relationship between teeth and individualized attributes. In other words, the geometry and topology of the problem are explicitly encoded into the model, shifting to object (tooth) and relationship-based graph analysis.

[0108] For example, for nodes: each tooth is defined as a node in the graph. The initial features of each node can be composed of two parts: one part is the visual features extracted from the image block of the tooth cut out from the panoramic image through a small CNN; the other part is the specific risk factor applicable to the tooth (if applicable) or the global risk factor.

[0109] Edges: Edges are created between adjacent dental nodes, representing their adjacency in the dental arch.

[0110] Graph processing: The constructed graph is input into a graph neural network (GNN). GNNs allow each dental node to aggregate information from its neighbors through a “message passing” mechanism. After several rounds of passing, each node contains contextual information about its local oral environment.

[0111] Finally, each node is classified (predicting the probability of caries for that tooth, and thus achieving classification), or the entire graph is classified (predicting the probability of the child having ECC).

[0112] 3. Using condition normalization instead of attention multiplication: Instead of generating a spatial “attention map” to multiply with the feature map, the risk data is used to “regulate” the behavior of the CNN network layer.

[0113] As an example, a technique called FiLM (Feature-wise Linear Modulation) is used. The static risk data is encoded into a risk feature vector using an MLP, but instead of being used to generate an attention map, it is input into two different linear layers to generate a pair of scaling factor (gamma) and offset factor (beta). After each (or some) convolutional layer of the CNN, the resulting image feature map is batch normalized, and then linearly transformed with the pair of gamma and beta: output = gamma * (normalized feature) + beta. In this scheme, the risk data regulates the features, but does not generate a spatial attention map, nor does it perform element-wise multiplication. It achieves risk guidance by changing the parameters of the normalization layer to affect the distribution of the features.

[0114] 4. Using a gated fusion mechanism instead of attention weighting: Instead of using risk data to “amplify / diminish” the image features, it is used as a “switch” or “valve” to determine the weight of the image information and risk information themselves.

[0115] For example, the image feature map F_img and the risk feature vector V_risk are also extracted. The risk vector V_risk is passed through a sigmoid activation function to generate one or a set of gating values g (between 0 and 1). The risk vector V_risk is also passed through a linear layer to match its dimension with the image feature map F_img, resulting in F_risk. The final fused feature F_fused is calculated by the gating formula: F_fused = g*F_img + (1-g)*F_risk. This formula uses weighted summation rather than multiplication weighting. It fuses features from two sources, rather than modulating one with the other. The gating value g determines whether the final output relies more on the image features or the risk features.

[0116] 5. Late fusion ensemble learning: Give up any form of deep fusion in the network, train two models completely independently, and finally fuse them in the decision layer. In other words, the two models are completely decoupled at the feature extraction level, and only "vote" or "weight" the final prediction results.

[0117] For example: Model 1: Train a pure CNN, only use panoramic image as input, output a caries probability P_img.

[0118] Model 2: Train an independent machine learning model (such as XGBoost, LightGBM or MLP), only use static risk data as input, output a caries risk score S_risk.

[0119] Final decision: Take P_img and S_risk as two features, input into a very simple final classifier (such as logistic regression), or directly use a preset weight formula P_final = w*P_img + (1-w)*S_risk to get the final result.

[0120] The advantages of the present application are:

[0121] 1. Through intelligent multi-modal information fusion, the following can be achieved:

[0122] Significant improvement in analysis performance: By combining direct evidence from images with background information from risk factors, the accuracy and recall rate of early childhood caries detection can be greatly improved, effectively reducing missed and misdiagnosed cases due to blurred image features;

[0123] Enhance early detection capability: Attention mechanism makes the model more sensitive to early and subtle image features (such as enamel demineralization white spots) corresponding to high-risk factors, achieving from "clinically visible" to "risk warning", and truly achieving early detection, early analysis and early treatment;

[0124] Personalized precision analysis: Breakthrough the traditional "one-size-fits-all" image analysis mode, can be personalized according to each child's unique risk profile image interpretation, to provide more comprehensive, more depth of data analysis support for clinicians, for subsequent personalized intervention program development provides a reliable basis.

[0125] 2. Through the double-flow architecture and double output, the following can be achieved:

[0126] Maximization of information utilization: The double-flow architecture ensures that two different types of data are fully extracted in the most suitable network structure, avoiding information loss caused by simple splicing;

[0127] Provide comprehensive analysis report: The double output design (overall caries probability + suspicious caries site positioning map) provides a complete data analysis information chain from macro risk assessment to micro suspicious caries site positioning for doctors. Provides direct, visual basis for treatment planning. This makes the output of the present application much more clinically useful than the single classification result of the prior art.

[0128] 3. Through the attention fusion module, the following can be achieved:

[0129] Dynamic and personalized feature focusing: The attention of traditional CNN to all image regions is static and determined by data distribution. The attention map in the present application is dynamically generated according to each input individual static data. For example, when the "night feeding habit" in the static data is "yes", the attention generation network will learn to generate an attention map with higher weight in the upper jaw region; when "high sugar diet" is "yes", the attention map may focus on the sulcus region of the molar. This makes the model from a "one-size-fits-all" inspector to an intelligent analysis expert who can "check according to the medical history";

[0130] Significantly improve the detection rate of early lesions (sensitivity): Early caries (such as enamel white spot) shows extremely weak density changes on X-ray film, which is easily ignored as noise by standard CNN. However, when a child's static data shows that he is a high-risk individual for caries, the attention mechanism of the present application will "force" the model to increase the sensitivity of the image features in the corresponding tooth region, so as to capture the weak signals that are easily missed by conventional methods. This effect directly solves the pain point of the prior art that it is not sensitive to early lesions;

[0131] Effectively reduce the false positive rate (specificity): There are often artifacts or normal anatomical variations in panoramic films, which may be misjudged as caries or other lesions. For a child whose static data shows low risk, the attention map generated by him has low weight in most areas, which will inhibit the model from overreacting to these ambiguous artifact regions, thereby reducing false positives. This effect directly improves the reliability of data analysis.

[0132] Enhance the interpretability of the model: the attention map output itself is a very clinically valuable byproduct. By visualizing it and superimposing it on the original image, the doctor can clearly see which tooth regions the model has focused on and thus made the final diagnosis. This greatly alleviates the "black box" problem of deep learning and enhances the doctor's trust in the model results.

[0133] Based on the above embodiment, the application also provides a caries analysis system based on static data and panoramic film images, as shown in Figure 3 The system comprises:

[0134] A data acquisition module 01 is configured to acquire image data and static data of a panoramic film of a user's oral cavity; the static data is used to reflect at least one of the following information: dietary habits, oral hygiene, socio-economic factors, medical history, and family history;

[0135] An attention module 02 is configured to generate a weighted fusion feature map according to the static data and the image data by using a convolutional neural network model based on an attention mechanism;

[0136] A data analysis module is configured to perform caries analysis according to the fusion feature map.

[0137] Further, the attention module 02 comprises:

[0138] An image feature extraction flow (such as a CNN) is configured to extract an image feature map from the image data;

[0139] A static data feature extraction flow (such as an MLP) is configured to extract a risk feature vector from the static data;

[0140] An attention generation module is configured to receive the risk feature vector and generate an attention map (such as a spatial attention map);

[0141] A feature fusion module is configured to weight the image feature map using the attention map to generate a fusion feature map.

[0142] This scheme protects the preferred system architecture for implementing the application, i.e., the overall design of "dual-flow input + attention fusion".

[0143] Based on the above embodiment, the application also provides a terminal, and the principle block diagram thereof can be as shown in Figure 4The terminal includes a processor, a memory, a network interface, and a display screen connected through a system bus. The processor of the terminal is configured to provide computing and control capabilities. The memory of the terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The network interface of the terminal is configured to communicate with external terminals through a network connection. The computer program is executed by the processor to implement the method for caries analysis based on static data and panoramic image. The display screen of the terminal can be a liquid crystal display screen or an electronic ink display screen.

[0144] Those skilled in the art can understand that Figure 4 The principle block diagram shown in the figure is only a block diagram of part of the structure related to the present application, and does not constitute a limitation on the terminal to which the present application is applied. The specific terminal can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0145] In an implementation manner, the memory of the terminal stores more than one program, and is configured to execute the more than one program by more than one processor, which includes instructions for performing the method for caries analysis based on static data and panoramic image.

[0146] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0147] In summary, the application discloses a caries analysis method and system based on static data and panoramic film images, the method obtains image data and static data of a panoramic film of a user's oral cavity, the static data is used to reflect at least one of the following information: eating habits, oral hygiene, socio-economic factors, medical history and family history of the user, a convolutional neural network model based on an attention mechanism is used to generate a weighted fusion feature map according to the static data and the image data, and caries analysis is performed according to the fusion feature map. The application effectively combines the panoramic film data and the static data based on the convolutional neural network based on the attention mechanism, can improve the accuracy of caries analysis, and provides a basis for early caries prevention. The problem that a deep learning model for analyzing caries in the prior art is limited to a single data source and is difficult to accurately perform risk early warning is solved.

[0148] It should be understood that the application is not limited to the above examples, and can be improved or changed according to the above description for those skilled in the art, and all these improvements and changes shall belong to the protection scope of the appended claims of the application.

Claims

1. A method for caries analysis based on static data and panoramic radiographic images, characterized by, The method comprises: acquiring image data and static data of a panoramic film of a user's oral cavity; the static data is used to reflect at least one of the user's dietary habits, oral hygiene, socio-economic factors, medical history and family history; generating a weighted fusion feature map according to the static data and the image data through a convolutional neural network model based on an attention mechanism; carrying out dental caries analysis according to the fusion feature map.

2. The method of caries analysis based on static data and panoramic radiographic images according to claim 1, characterized in that, The static data is obtained based on a questionnaire survey and / or electronic health records.

3. The method of caries analysis based on static data and panoramic radiographs according to claim 1, characterized in that, The generating of the weighted fusion feature map according to the static data and the image data through the convolutional neural network model based on the attention mechanism comprises: preprocessing the static data, including: converting data of different dimensions in the static data into static data features respectively; and splicing all the static data features to obtain a static data long vector; preprocessing the image data, including: performing a standardization operation on the image data to obtain processed image data; the standardization operation includes at least one of size normalization, pixel value normalization and data enhancement; inputting the static data long vector and the processed image data into the convolutional neural network model to obtain the fusion feature map.

4. The method of caries analysis based on static data and panoramic radiographic images according to claim 3, characterized in that, The convolutional neural network model comprises: a double-flow feature extraction module, configured to extract features from the processed image data to obtain an image feature map, and extract features from the static data long vector to obtain a risk feature vector; an attention fusion module, configured to generate the fusion feature map according to the image feature map and the risk feature vector.

5. The method of caries analysis based on static data and panoramic radiographic images according to claim 4, characterized in that, The generating of the fusion feature map according to the image feature map and the risk feature vector comprises: generating an attention map according to the risk feature vector through an attention generation network in the attention fusion module; each value in the attention map represents a weight of a corresponding position in the image feature; performing an element-by-element weighting operation on the attention map and the image feature map to obtain the fusion feature map.

6. The method of caries analysis based on static data and panoramic radiographs according to claim 1, wherein, The dental caries analysis according to the fusion feature map comprises: inputting the fusion feature map into a prediction module; the prediction module comprises a global average pooling layer, a fully connected layer and an activation function layer; calculating a dental caries probability of the user based on the fusion feature map through the prediction module.

7. The method of caries analysis based on static data and panoramic radiographs according to claim 1, wherein, The dental caries analysis according to the fusion feature map comprises: inputting the fusion feature map into a positioning module; generating a positioning image according to the fusion feature map through the positioning module; the positioning image is used to reflect a predicted dental caries position, range and confidence, and the image form of the positioning image includes a heat map or a segmentation map.

8. A caries analysis system based on static data and panoramic radiographic images, characterized by, The system comprises: a data acquisition module, configured to acquire image data and static data of a panoramic film of a user's oral cavity; the static data is used to reflect at least one of the user's dietary habits, oral hygiene, socio-economic factors, medical history and family history; an attention module, configured to generate a weighted fusion feature map according to the static data and the image data through a convolutional neural network model based on an attention mechanism; A data analysis module is configured to perform caries analysis based on the fused feature map.

9. A terminal, characterized by comprising: The terminal comprises a memory and one or more processors; the memory stores one or more programs; the programs contain instructions for executing the caries analysis method based on static data and panoramic image according to any one of claims 1 to 7; the processor is configured to execute the programs.

10. A computer readable storage medium having stored thereon a plurality of instructions, the plurality of instructions comprising: The instructions are adapted to be loaded and executed by the processor to implement the steps of the caries analysis method based on static data and panoramic image according to any one of claims 1 to 7.

Images