Pulp state recognition model training method, electronic device, and storage medium

Abstract

The application provides a pulp state recognition model training method, an electronic device and a storage medium, and relates to the technical field of image processing. The method comprises the following steps: obtaining a plurality of pulp section images under a microscope in a history operation, using a feature encoder to extract local features and global features corresponding to each pulp section image, performing bilinear interaction processing on the local features and the global features corresponding to each pulp section image, obtaining a fusion feature vector corresponding to each pulp section image, and the fusion feature vector contains correlation information of the local features and the global features, thereby realizing deep fusion of global and local information; and then using the fusion feature vector corresponding to each pulp section image and an actual recognition result to iteratively train an initial classifier, and taking an initial pulp state model at the end of iteration as a target pulp state recognition model, thereby solving the problem that an existing AI model cannot recognize and process the pulp section images under the microscope in a pulp amputation operation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a pulp state recognition model training method, an electronic device and a storage medium. BACKGROUND

[0002] With the continuous development of Artificial Intelligence (AI) technology, AI has been widely used in many aspects of the medical field. In medical image processing, AI intelligent models can be used to recognize and process medical image images (such as CT scan images, magnetic resonance images) to give auxiliary recognition results.

[0003] In a pulp amputation, the pulp state needs to be identified to help the doctor perform subsequent operations. In the prior art, the pulp state is generally identified according to the experience of the doctor, and there are problems of inconsistent judgment standards and difficult to guarantee the recognition accuracy. Therefore, in the pulp amputation, how to use an AI intelligent model to assist in real-time recognition of the pulp state to improve the recognition accuracy and efficiency is a problem worth solving. SUMMARY

[0004] The present application aims to solve the technical problems in the prior art by providing a pulp state recognition model training method, an electronic device and a storage medium.

[0005] To achieve the above-mentioned purpose, the technical solutions adopted by the embodiments of the present application are as follows:

[0006] In a first aspect, the embodiments of the present application provide a pulp state recognition model training method, which comprises:

[0007] Obtaining a plurality of pulp section images under a microscope in a history operation, and determining the actual recognition results of each of the pulp section images, the actual recognition results comprising: suitable for pulp capping or not suitable for pulp capping;

[0008] Inputting each of the pulp section images into a pre-constructed initial pulp state recognition model, the initial pulp state recognition model comprising: a pre-trained feature encoder and an initial classifier;

[0009] Extracting local features and global features corresponding to each of the pulp section images by the feature encoder, and performing bilinear interaction processing on the local features and the global features corresponding to each of the pulp section images to obtain a fusion feature vector corresponding to each of the pulp section images, the fusion feature vector containing associated information of the local features and the global features;

[0010] outputting a predicted recognition result according to a fusion feature vector corresponding to each of the pulp section images by the initial classifier;

[0011] training the initial classifier iteratively according to the predicted recognition result and the actual recognition result, and taking the initial pulp state model at the end of the iteration as a target pulp state recognition model; wherein a parameter freezing strategy is adopted in the iterative training process, the network parameters of the feature encoder are kept unchanged, and the network parameters of the initial classifier are iteratively updated.

[0012] Optionally, the initial pulp state recognition model further comprises a position encoder pre-trained;

[0013] The feature encoder extracts local features and global features corresponding to each of the pulp section images, comprising:

[0014] The position encoder encodes the pulp section images to obtain high-dimensional feature vectors corresponding to the pulp section images;

[0015] The feature encoder extracts intermediate local features and global features corresponding to the pulp section images by inputting the high-dimensional feature vectors corresponding to the pulp section images into the feature encoder;

[0016] The intermediate local features are input into a pre-constructed local attention learning module to obtain local features corresponding to the pulp section images.

[0017] Optionally, the local attention learning module comprises a multi-layer perceptron network, a normalization layer and a weighted aggregation layer.

[0018] The local attention learning module comprises a multi-layer perceptron network, a normalization layer and a weighted aggregation layer.

[0019] The intermediate local features are input into the multi-layer perceptron network to obtain an importance score of each feature vector in the intermediate local features;

[0020] The normalization layer normalizes the importance scores of each feature vector to obtain attention weights corresponding to each feature vector;

[0021] The weighted aggregation layer performs weighted summation on the attention weights corresponding to each feature vector to obtain local features corresponding to the pulp section images.

[0022] Optionally, the local attention learning module comprises a multi-layer perceptron network, a normalization layer and a weighted aggregation layer.

[0023] projecting the local feature and the global feature to a low-dimensional space respectively by a pre-constructed linear projection layer to obtain a low-dimensional global feature and a low-dimensional local feature;

[0024] performing element-wise multiplication on the low-dimensional global feature and the low-dimensional local feature to capture high-order correlation to obtain a fusion feature vector corresponding to the pulp section image.

[0025] Optionally, the method further comprises:

[0026] performing feature dimension reduction on the fusion feature vector corresponding to each pulp section image to obtain a low-dimensional fusion feature vector corresponding to each pulp section image;

[0027] performing normalization processing and nonlinear processing on the low-dimensional fusion feature vector corresponding to each pulp section image to obtain a nonlinear feature vector corresponding to each pulp section image;

[0028] inputting the nonlinear feature vector corresponding to each pulp section image into the initial classifier to obtain a predicted recognition result corresponding to each pulp section image;

[0029] determining loss information of the initial pulp state recognition model according to the predicted recognition result and the actual recognition result corresponding to each pulp section image, and iteratively correcting the initial classifier according to the loss information until the corrected initial classifier meets a preset convergence condition to obtain the classifier.

[0030] Optionally, the method further comprises:

[0031] obtaining a pre-constructed joint loss function, wherein the joint loss function is obtained by weighted fusion based on a classification loss optimization function and a feature discrimination optimization function;

[0032] inputting the predicted recognition result and the actual recognition result corresponding to each pulp section image into the joint loss function to obtain the loss information of the initial pulp state recognition model.

[0033] Optionally, the method further comprises:

[0034] obtaining an original pulp section image under a current intraoperative microscope;

[0035] perform image enhancement processing on the original pulp section image using a preset at least one image enhancement transformation strategy to generate an enhanced image set, the enhanced image set including: a pulp section transformed image under each of the image enhancement transformation strategies;

[0036] respectively pre-process each of the pulp section transformed images to obtain a pre-processed image corresponding to each of the pulp section transformed images;

[0037] input the pre-processed image corresponding to each of the pulp section transformed images into the target pulp state recognition model to obtain a predicted recognition result corresponding to each of the pulp section transformed images;

[0038] determine a predicted recognition result of the original pulp section image according to the predicted recognition result corresponding to each of the pulp section transformed images.

[0039] Optionally, the determining the predicted recognition result of the original pulp section image according to the predicted recognition result corresponding to each of the pulp section transformed images includes:

[0040] perform weighted processing on the predicted recognition result corresponding to each of the pulp section transformed images, and take the weighted result as the predicted recognition result of the original pulp section image.

[0041] In a second aspect, an electronic device is provided, and the electronic device includes a memory configured to store one or more programs; and a processor. When the one or more programs are executed by the processor, the above-described pulp state recognition model training method is implemented.

[0042] In a third aspect, a computer-readable storage medium is provided, and the computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the above-described pulp state recognition model training method is implemented.

[0043] The application has the following beneficial effects:

[0044] The embodiment of the present application provides a dental pulp state recognition model training method, an electronic device and a storage medium, in the present application, a plurality of dental pulp section images under a microscope in a history operation are collected, and a feature encoder is used to extract local features and global features corresponding to each dental pulp section image, and the local features and the global features corresponding to each dental pulp section image are subjected to double linear interaction processing, to obtain a fusion feature vector corresponding to each dental pulp section image, the fusion feature vector contains associated information of the local features and the global features, realizing deep fusion of global and local information, solving the problem that a traditional ViT classification method is easy to lose key local detail information; then, the fusion feature vector corresponding to each dental pulp section image and an actual recognition result are used to iteratively train an initial classifier, and the initial dental pulp state model at the end of iteration is taken as a target dental pulp state recognition model, the target dental pulp state recognition model can be used to perform instant recognition on the dental pulp section image under the microscope in the operation, ensuring the recognition efficiency and accuracy, and solving the problem that an existing AI model cannot perform instant recognition processing on the dental pulp section image under the microscope in the dental pulp cutting operation.

[0045] In order to make the above objectives, characteristics and advantages of the present application more apparent, the following preferred embodiments are specifically described below, and the accompanying drawings are referred to, and the detailed description is as follows. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments, and it should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation to the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0047] Figure 1 A flowchart of a dental pulp state recognition model training method provided by the embodiment of the present application is shown in the figure.

[0048] Figure 2 A structure diagram of a dental pulp state recognition model provided by the embodiment of the present application is shown in the figure.

[0049] Figure 3 A structure diagram of another dental pulp state recognition model provided by the embodiment of the present application is shown in the figure.

[0050] Figure 4 A flowchart of another dental pulp state recognition model training method provided by the embodiment of the present application is shown in the figure.

[0051] Figure 5 A schematic diagram of position encoding processing of a dental pulp section image by a position encoder provided by the embodiment of the present application is shown in the figure.

[0052] Figure 6A flowchart of another method for training a dental pulp state recognition model is provided in the embodiments of the present application.

[0053] Figure 7 A flowchart of another method for training a dental pulp state recognition model is provided in the embodiments of the present application.

[0054] Figure 8 A flowchart of another method for training a dental pulp state recognition model is provided in the embodiments of the present application.

[0055] Figure 9 A flowchart of another method for training a dental pulp state recognition model is provided in the embodiments of the present application.

[0056] Figure 10 A flowchart of another method for training a dental pulp state recognition model is provided in the embodiments of the present application.

[0057] Figure 11 A flowchart of another method for training a dental pulp state recognition model is provided in the embodiments of the present application.

[0058] Figure 12 A flowchart of another method for training a dental pulp state recognition model is provided in the embodiments of the present application. DETAILED DESCRIPTION

[0059] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the following will be a clear and complete description of the technical solutions in the embodiments of the present application with reference to the accompanying drawings. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed in various different configurations.

[0060] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative labor are within the scope of protection of the present application.

[0061] The following will be a detailed description of some embodiments of the present application with reference to the accompanying drawings. The following embodiments and features in the embodiments can be combined with each other without conflict.

[0062] The implementation principles and corresponding beneficial effects of the training steps of the dental pulp state recognition model provided by the present application will be described as follows through a plurality of specific embodiments.

[0063] In an embodiment, reference is made to Figure 1As shown, a dental pulp state recognition model training method is provided. Optionally, the execution subject of the method can be a terminal device with data processing function such as a personal computer, a notebook computer, a smart phone, and a tablet computer.

[0064] It should be understood that in other embodiments, the order of some steps of the dental pulp state recognition model training method can be exchanged according to actual needs, or some steps can be omitted or deleted. Figure 1 As shown, the method comprises:

[0065] S101, acquiring a plurality of dental pulp section images under a historical intraoperative microscope, and determining actual recognition results of the dental pulp section images.

[0066] The actual recognition results include: suitable pulp capping or unsuitable pulp capping. The suitable pulp capping or unsuitable pulp capping is the dental pulp state of a diseased tooth.

[0067] The suitable pulp capping refers to that the dental pulp section is red soft tissue rich in blood, the morphology is continuous and uniform, the surrounding is healthy dentin, and the surface is not covered with purulent secretion and dentin fragments.

[0068] The unsuitable pulp capping refers to that the dental pulp section is light yellow, pale, dark gray, or dark red in color, or the bleeding has not stopped, or the surface is covered with purulent secretion or dentin fragments, or the tissue texture is loose and discontinuous.

[0069] It should be noted that the dental pulp section image in the present application is a microscopic image under a microscope. The microscopic image mainly focuses on the surface state of the tissue, contains visual features such as color and texture, and the image details are rich, the illumination changes greatly, and the image processing is difficult. It is completely different from other medical image images (without color information, preoperative shooting). At the same time, the existing AI model is mostly based on preoperative medical images (such as CBCT, intraoral scan), rather than real-time microscopic images under an intraoperative microscope. Therefore, the existing AI model cannot perform instant recognition processing on the dental pulp section image under the intraoperative microscope during the dental pulp amputation to assist in identifying the dental pulp state of the diseased tooth.

[0070] In an implementable manner, a plurality of dental pulp section images under a historical intraoperative microscope are collected, such as at least 100 high-definition magnified dental pulp section images. The collection criteria are: age 14-60 years old, no systemic disease history, mature permanent teeth with closed apical foramen, deep caries in the crown, and dental pulp exposure during the removal of the caries.

[0071] Exclusion criteria: other dental diseases such as crown cracks, teeth with severe calcification of the pulp cavity shown in the preoperative film, severely contaminated dental pulp section, and image blur that cannot be identified.

[0072] Photographic parameter details: Zeiss E200 microscope, Ossun microscope, shooting resolution 3840*2160, magnification 20-40 times, cold light source, and shooting angle perpendicular to the dental pulp section as much as possible.

[0073] Among them, mainly by artificial means (such as by 2 dental pulp experts with clinical experience > 10 years), each dental pulp section image collected is independently labeled, dental pulp section images with differences are introduced to the third dental pulp expert (clinical experience > 15 years) for arbitration, and the final actual recognition result is determined after 3 people discuss and reach an agreement, and the dental pulp section position is labeled by labelme, accurate to the pixel level, for later data processing.

[0074] Optionally, after collecting a plurality of dental pulp section images under the microscope in the history operation, each dental pulp section image needs to be processed as follows, and the specific process is as follows:

[0075] (1) Data preprocessing:

[0076] Crop: based on the dental pulp section rectangular frame labeled by experts, crop the area containing the complete dental pulp section (exclude irrelevant background);

[0077] Uniform size: resize the cropped image to 448x448 pixels (use bicubic interpolation);

[0078] Standardization: Z-score standardization is performed on the image pixel value, and the formula is x'=(x-μ) / σ, wherein μ is the pixel mean of all images in the training set, and σ is the pixel standard deviation of all images in the training set (to eliminate the influence of different microscope light source differences).

[0079] (2) To improve the generalization ability of the model, the following enhancement is performed on the training set images using data enhancement strategy, and the specific process is as follows:

[0080] Random crop: crop a 448x448 pixel area from the center of the original image (if the original image size is insufficient, up-sample to 518x518);

[0081] Random rotation: random rotation within ±15°;

[0082] Brightness / contrast adjustment: brightness randomly changes within ±20%, and contrast randomly changes within ±15%;

[0083] Horizontal flip: horizontal flip with a probability of 50%;

[0084] Gaussian noise: add Gaussian noise with a standard deviation ≤0.01 (simulate microscope imaging noise).

[0085] (3) Data set division

[0086] The stratified sampling method is used to divide into a training set (80%): for training; a test set (20%): independent of the training process, for model performance evaluation, in a ratio of 8:2.

[0087] S102, input each pulp section image into the pre-constructed initial pulp state recognition model, extract the local features and global features corresponding to each pulp section image by the feature encoder, and perform bilinear interaction processing on the local features and global features corresponding to each pulp section image to obtain the fusion feature vector corresponding to each pulp section image.

[0088] The initial pulp state recognition model includes a pre-trained feature encoder and an initial classifier. Figure 2 As shown, the initial pulp state recognition model is composed of a feature encoder and an initial classifier. The initial classifier can be a linear classifier, such as a logistic regression, a Bayesian classifier, etc.

[0089] The fusion feature vector contains the association information of the local features and the global features.

[0090] In an implementable manner, the present application adopts a visual model (ViT and DINOv2) based on the Transformer architecture to realize the binary classification judgment of pulp capping and pulp suitability.

[0091] In an implementable manner, the present application adopts a visual Transformer (ViT) architecture visual model as the initial pulp state recognition model, which includes a feature encoder and an initial classifier. The ViT after self-supervised learning is used as the feature encoder (DINOv2), which uses a contrastive learning strategy to learn the representation of the field image through different enhanced views of the image, and the student network learns the output distribution of the teacher network to improve the feature extraction capability.

[0092] Therefore, the self-attention mechanism of the DINOv2 feature encoder can be used to model global dependencies and capture spatial correlations between different regions of the pulp section (such as boundary features of healthy pulp tissue and inflammatory areas, correlation between blood vessel distribution patterns and tissue color, texture changes between bleeding areas and surrounding tissues, and other complex features).

[0093] In this embodiment, the local features LocalFeatures and the global features Global Features corresponding to each pulp section image are extracted by the DINOv2 feature encoder, and the local features LocalFeatures and the global features Global Features corresponding to each pulp section image are subjected to bilinear interaction processing to obtain a fusion feature vector corresponding to each pulp section image, realizing deep fusion of global and local information and solving the problem that the traditional ViT classification method only uses [CLS] tokens for classification, which easily loses key local detail information.

[0094] Alternatively, a convolutional neural network (such as ResNet-50, EfficientNet-B4) can be used instead of a Transformer architecture, or other pre-trained models (such as CLIP, ConvNeXt) can be used to build a binary classification model. The core is to extract microscopic image features through deep learning to realize pulp state recognition.

[0095] S103, outputting a predicted recognition result according to the fusion feature vector corresponding to each pulp section image by the initial classifier, iteratively training the initial classifier according to the predicted recognition result and the actual recognition result, and taking the initial pulp state model at the end of iteration as a target pulp state recognition model.

[0096] In the iterative training process, a parameter freezing strategy is adopted to keep the network parameters of the feature encoder unchanged and iteratively update the network parameters of the initial classifier.

[0097] Alternatively, in this application, the model is trained using the transfer learning paradigm and supervised learning method, and the end-to-end training is performed using the expert-labeled pulp image dataset.

[0098] Illustratively, the training environment is as follows: software: PyTorch 2.0 framework, Python 3.9; hardware: NVIDIA RTX 3090 GPU (24GB video memory), 128GB memory.

[0099] In this embodiment, the specific process of model training is as follows: the fusion feature vector corresponding to each pulp section image can be input to the initial classifier to obtain the predicted recognition result output by the initial classifier, and the initial classifier is iteratively trained using the predicted recognition result and the actual recognition result of each pulp section image. The initial pulp state model at the end of iteration is taken as a target pulp state recognition model, which can be used to perform real-time recognition of the pulp section image under the microscope during the operation, ensuring the recognition efficiency and accuracy, and solving the problem that the existing AI model cannot perform real-time recognition of the pulp section image under the microscope during the pulp sectioning operation.

[0100] Optionally, the pre-trained DINOv2 feature encoder is used as a fixed feature encoder, which learns hierarchical visual representations on large-scale natural image datasets that can effectively capture texture, morphology, and color features of dental pulp tissue. By using the parameter freezing mechanism, the encoder weights remain unchanged, avoiding the catastrophic forgetting phenomenon on small-scale medical datasets.

[0101] During training, the network parameters of the DINOv2 feature encoder remain unchanged, and gradient backpropagation only updates the weight parameters and bias parameters of the initial classifier. For example, the batch size is set to 8, the total training epoch is 100, the AdamW optimizer is used for parameter update, the learning rate is set to 1.5*10 -4 , the weight decay coefficient is 0.05, the loss function uses cross-entropy loss to calculate the difference between the model prediction probability and the true label, the learning rate uses StepLR scheduling strategy, and the learning rate is reduced to 0.1 times every 10 epochs, Dropout regularization is applied to the classification head during training, and the dropout rate is set to 0.3 to prevent overfitting.

[0102] Optionally, the final trained target dental pulp state recognition model can be deployed to mobile devices, cloud servers, or edge devices, etc. Model compression techniques such as model quantization, knowledge distillation, and neural network pruning can be used to adapt to resource-constrained devices. Different communication protocols such as RESTful API, gRPC, and WebSocket can be used for interface implementation to support integration with existing medical information systems. Relational databases, non-relational databases, or distributed storage systems can be used for data storage to ensure secure management and fast retrieval of large-scale clinical data.

[0103] In summary, the embodiment of the present application provides a dental pulp state recognition model training method. In the present application, a plurality of dental pulp section images under the microscope during the operation are collected, and feature encoders are used to extract local features and global features corresponding to each dental pulp section image. The local features and global features corresponding to each dental pulp section image are bilinearly interacted to obtain a fusion feature vector corresponding to each dental pulp section image. The fusion feature vector contains the association information of the local features and global features, realizing the deep fusion of global and local information, and solving the problem that the traditional ViT classification method easily loses key local detail information. Then, the fusion feature vector corresponding to each dental pulp section image and the actual recognition result are used to iteratively train an initial classifier, and the initial dental pulp state model at the end of iteration is used as a target dental pulp state recognition model. The target dental pulp state recognition model can be used to instantly recognize the dental pulp section image under the microscope during the operation, ensuring the recognition efficiency and accuracy, and solving the problem that existing AI models cannot instantly recognize the dental pulp section image under the microscope during the dental pulp cutting operation.

[0104] Optionally, referring to Figure 3 As shown in the figure, the initial pulp state recognition model further includes a pre-trained position encoder.

[0105] Exemplarily, the ViT-Base architecture can be adopted as the position encoder, containing 12 Transformer blocks, each block having a 768-dimensional hidden layer and 12 attention heads.

[0106] Optionally, referring to Figure 4 As shown in the figure, the step S102 includes:

[0107] S201, position encoding the pulp cross-section image by the position encoder to obtain a high-dimensional feature vector corresponding to the pulp cross-section image.

[0108] S202, inputting the high-dimensional feature vector corresponding to the pulp cross-section image into the feature encoder to extract intermediate local features and global features corresponding to the pulp cross-section image by the feature encoder.

[0109] S203, inputting the intermediate local features into the pre-constructed local attention learning module to obtain local features corresponding to the pulp cross-section image.

[0110] In an implementable manner, the position encoder of the ViT-Base architecture can be selected to perform position encoding processing on the pulp cross-section image. As shown in the figure, Figure 5 As shown in the figure, the size of the input pulp cross-section image is 448×448×3, each pulp cross-section image can be divided into 1024 14×14×3 patches, each 14×14×3 patch is transformed into a 1×1×768 feature through convolution operation, and then is tiled into a 1×768 feature, and finally the Patch Embedding size is 1024×768. Therefore, different rows in the Patch Embedding represent patches at different positions, and different columns represent pixel points at different positions in the patch.

[0111] Here, the DINOv2-ViT-Base architecture is selected, which adopts a 448×448 pixel input image, uses a 14×14 pixel image block for segmentation, generates 1024 image blocks (32×32=1024), and maps them into 768-dimensional feature vectors through a linear projection layer. The parameter matrix dimension of the linear transformation layer is (14×14×3)×768, and after adding a classification token, 1025 input sequences are formed for classification, wherein 1024×768 is the intermediate local feature, and 1×768 is the global feature.

[0112] The architecture adds two-dimensional sine-cosine position encoding to each image patch and classification token, with a dimension of 768, and the position encoding formula is shown in the following formula (1) - formula (2):

[0113] (1)

[0114] (2)

[0115] wherein pos is a position index, i is a dimension index, and the dimension index is used to ensure that the model can understand the spatial position relationship.

[0116] In the embodiment, continuing to refer to the initial pulp state recognition model shown in Figure 3 , the initial pulp state recognition model further includes a local attention learning module, so that the intermediate local features (i.e., 1024x768) can be input to the pre-constructed local attention learning module to obtain the local features corresponding to the pulp cross-sectional image.

[0117] Optionally, continuing to refer to the local attention learning module shown in Figure 3 , the local attention learning module includes a multi-layer perceptron network, a normalization layer, and a weighted aggregation layer.

[0118] For example, the multi-layer perceptron network is a two-layer multi-layer perceptron (MLP) network, and the normalization layer is a Softmax function.

[0119] Optionally, referring to the above step S203 shown in Figure 6 , the step S203 includes:

[0120] S301, inputting the intermediate local features to the multi-layer perceptron network to obtain the importance scores of each feature vector in the intermediate local features.

[0121] S302, normalizing the importance scores of each feature vector by the normalization layer to obtain the attention weights corresponding to each feature vector.

[0122] S304, performing weighted summation on the attention weights corresponding to each feature vector by the weighted aggregation layer to obtain the local features corresponding to the pulp cross-sectional image.

[0123] In the embodiment, 1) local attention weight learning: the intermediate local features can be input to a two-layer multi-layer perceptron (MLP) network, and for each patch feature vector in the intermediate local features, the importance score of the patch feature vector is learned by the two-layer multi-layer perceptron (MLP) network, as shown in the following formula (3):

[0124] (3)

[0125] wherein, is the first layer weight matrix, is the second layer weight vector, and is the bias parameter, and tanh is used as the activation function to ensure numerical stability.

[0126] The attention scores of all patches are then normalized by Softmax to obtain the normalized attention weights as shown in the following equation (4):

[0127] (4)

[0128] This process automatically learns to identify key regions in the pulp section image, such as bleeding areas, abnormal tissue color areas, and clinically important parts such as inflammation infiltration boundaries.

[0129] 2) Weighted local feature aggregation: based on the learned attention weights, the weighted sum of all patch features is performed to generate the local feature corresponding to the pulp section image, i.e. the integrated local focus feature vector, as shown in the following equation (5):

[0130] (5)

[0131] Therefore, the key information scattered in 1024 patches can be aggregated into a single vector representation.

[0132] Optionally, referring to Figure 7 the specific processing process of the bilinear interaction in step 102, including:

[0133] S401, by the pre-constructed linear projection layer, respectively project the local feature and the global feature into the low-dimensional space to obtain the low-dimensional global feature and the low-dimensional local feature.

[0134] S402, element-wise multiplication is performed on the low-dimensional global feature and the low-dimensional local feature to capture high-order correlation, and a fusion feature vector corresponding to the pulp section image is obtained.

[0135] In an implementable manner, to avoid dimension disaster and computational complexity in high-dimensional space, first, the shared linear projection layer is used to project the local feature and the global feature into the low-dimensional space, which is specifically shown in the following equations (6)-(7):

[0136] (6)

[0137] (7)

[0138] wherein, is biased, is a low-dimensional local feature, is a low-dimensional global feature.

[0139] Then, the low-dimensional global feature and the low-dimensional local feature are multiplied element by element (Hadamard product) to capture high-order correlations, specifically as shown in the following formula (8):

[0140]

[0141] wherein, is a fusion feature vector corresponding to the pulp cross-section image, that is, the associated information of the fused global and local features.

[0142] Optionally, as shown in Figure 8 , the step S103 includes:

[0143] S501, respectively, the fusion feature vector corresponding to each pulp cross-section image is subjected to feature dimension reduction to obtain a low-dimensional fusion feature vector corresponding to each pulp cross-section image.

[0144] S502, respectively, the low-dimensional fusion feature vector corresponding to each pulp cross-section image is subjected to normalization processing and nonlinear processing to obtain a nonlinear feature vector corresponding to each pulp cross-section image.

[0145] S503, the nonlinear feature vector corresponding to each pulp cross-section image is input into an initial classifier to obtain a predicted recognition result corresponding to each pulp cross-section image.

[0146] S504, according to the predicted recognition result and the actual recognition result corresponding to each pulp cross-section image, the loss information of the initial pulp state recognition model is determined, and the initial classifier is iteratively corrected according to the loss information until the modified initial classifier meets the preset convergence condition, and the classifier is obtained.

[0147] In an implementable manner, as shown in Figure 3 , the fusion feature vector corresponding to the pulp cross-section image is subjected to feature dimension reduction processing, that is, the 768-dimensional fusion feature vector is reduced to a 512-dimensional fusion feature vector, and the 512-dimensional fusion feature vector is reduced to a 2-dimensional fusion feature vector, that is, a low-dimensional fusion feature vector.

[0148] Then, the low-dimensional fusion feature vector corresponding to each pulp cross-section image is subjected to L2 normalization processing to ensure the numerical stability of the features, and then the nonlinearity is introduced through ReLU activation. Specifically, as shown in the following formula (9) - formula (10):

[0149] (9)

[0150] (10)

[0151] wherein, is a normalized feature vector, is a nonlinear feature vector.

[0152] The nonlinear feature vector corresponding to each dental pulp cross-sectional image is input to the initial classifier, mapped to a 2-dimensional logits vector by a linear classifier, specifically as shown in the following formula (11):

[0153] (10)

[0154] wherein, is a classification weight matrix, is a classification bias, and logits are converted to a probability distribution by a Softmax function, specifically as shown in the following formula (12) - formula (13):

[0155] (12)

[0156] (13)

[0157] Therefore, the predicted recognition result corresponding to each dental pulp cross-sectional image is obtained, and the predicted recognition result corresponding to each dental pulp cross-sectional image is used to calculate the loss information of the initial dental pulp state recognition model at the current iteration round, and the network parameters of the initial classifier are iteratively corrected according to the loss information, until the corrected initial classifier meets the preset convergence condition, and the classifier is obtained.

[0158] Optionally, as shown in Figure 9 the above step S504 comprises:

[0159] S601, obtaining a pre-constructed joint loss function.

[0160] wherein the joint loss function is obtained by weighted fusion based on a classification loss optimization function and a feature distinction optimization function.

[0161] S602, input the predicted recognition result corresponding to the dental pulp cross-sectional image and the actual recognition result to the joint loss function, to obtain the loss information of the initial dental pulp state recognition model.

[0162] In an implementable manner, for the problems of class imbalance (e.g., too high proportion of "inappropriate" samples) and sample difficulty imbalance (a large number of simple samples with obvious features and a small number of difficult samples with ambiguous boundaries) in the pulp status identification task, the application proposes to use a classification loss optimized Focal Loss function as a basic loss function to dynamically suppress the weight of simple samples and strengthen the learning of difficult samples. Meanwhile, to solve the problem of "similar and different sample feature confusion" (e.g., mild lesion pulp and normal hyperemia pulp) in the pulp image, a contrast loss Contrastive Loss is introduced to construct a joint loss function, which further improves the model's ability to distinguish fine-grained features of the pulp, as follows:

[0163] Wherein, (1) for the sample difficulty imbalance problem existing in the pulp image data set, the application uses a classification loss optimized Focal Loss function to replace the traditional cross-entropy loss function, which dynamically suppresses the learning of simple samples and strengthens the learning of difficult samples by setting the model's prediction probability for the true class as , then the classification loss optimized Focal Loss function is defined as formula (14) shown below:

[0164] (14)

[0165] When the real label is 1, ; when the real label is 0, , wherein is the positive class probability output by the model. γ is a focusing parameter that controls the degree of weight suppression for simple samples.

[0166] In the application, γ = 2.0, is a class balance parameter used to handle class imbalance problems. The application dynamically calculates it according to the ratio of "appropriate pulp covering" and "inappropriate pulp covering" samples in the training data, using the following formula (15) - formula (16) shown below:

[0167] (15)

[0168] (16)

[0169] Wherein, and are the number of positive and negative samples, respectively.

[0170] (2) Auxiliary loss function:

[0171] In this embodiment, a feature distinction optimized Contrastive Loss function is introduced to solve the problem of similar sample feature confusion. Among them, the sample pair wherein , is a dental pulp cross-section image sample, and a corresponding true label pair is , a sample feature vector extracted by the model is is a feature dimension.

[0172] wherein the core of the feature distinction optimization Contrastive Loss function is to minimize the feature vector distance of the same sample pair, promote the aggregation of the same class features, and maximize the feature vector distance of the different sample pair, avoid the confusion of different class features, and the specific definition is shown in formula (17):

[0173] (17)

[0174] wherein, is the number of sample pairs in the training batch, is a sample pair label, =1 indicates is a same sample pair, =0 indicates a different sample pair, is the Euclidean distance square of the sample pair feature vector.

[0175] m is a margin parameter (m=2.0 is set in the present application), when the feature distance of the different sample pair is greater than m, the loss is no longer punished, avoiding invalid calculation; when the feature distance of the different sample pair is less than m, the model is punished by the square term, and the distance of the different class features is forced to be enlarged.

[0176] 3) Construct a joint loss function

[0177] is to balance the classification loss optimization Focal Loss function and the feature distinction optimization Contrastive Loss function, and to weight and fuse the two into the final training loss, and the weight coefficient λ (λ=0.3 is determined by the present application) is used to adjust the contribution degree of the auxiliary loss, and the specific definition is shown in formula (17):

[0178] (17)

[0179] Therefore, the prediction recognition result and the actual recognition result corresponding to all dental pulp cross-section image pairs can be input into the joint loss function, and the loss information of the initial dental pulp state recognition model in the current iteration round can be obtained.

[0180] Optionally, the target dental pulp state recognition model finally trained can also be evaluated, such as a multi-dimensional quantitative index system can be used to comprehensively evaluate the accuracy of the target dental pulp state recognition model in the dental pulp cap dental pulp suitability identification task, and the evaluation process can be based on an independent test set.

[0181] The classification performance evaluation adopts the confusion matrix framework to compare and analyze the model prediction results with the reference standard labeled by experts, and to count four types of judgment results: true positive (TP), true negative (TN), false positive (FP), and false negative (FN).

[0182] The core performance indicators are calculated based on the confusion matrix: the accuracy is defined as the proportion of correctly classified samples to the total samples, and the calculation formula is .

[0183] The accuracy reflects the overall correctness of the model; the sensitivity (sensitivity) is defined as the proportion of correctly identified samples in the true suitable pulp covering samples, and the calculation formula is , which measures the ability of the model to identify positive cases;

[0184] The specificity is defined as the proportion of correctly identified samples in the true unsuitable pulp covering samples, and the calculation formula is , which evaluates the ability of the model to exclude negative cases;

[0185] The precision is defined as the proportion of true suitable samples in the predicted suitable pulp covering samples, and the calculation formula is , which reflects the reliability of the model's positive prediction.

[0186] The comprehensive performance evaluation adopts F1 score as the harmonic mean of precision and recall, and the calculation formula is , where recall is equal to sensitivity, and F1 score can balance the identification ability of the model for two types of samples, providing more robust performance evaluation on class-imbalanced datasets.

[0187] The receiver operating characteristic curve analysis draws the curve of true positive rate and false positive rate at different judgment thresholds, calculates the area under the curve (AUC-ROC), and quantifies the discrimination ability of the model.

[0188] In addition, the precision-recall curve analysis is used to evaluate the performance of the model at different operating points, and the area under the curve (AUC-PR) is calculated as a supplementary evaluation indicator, which is suitable for the pulp status recognition scenario with unbalanced positive and negative samples.

[0189] The following embodiments will be used to specifically explain how to use the trained target pulp status recognition model to accurately recognize the current intraoperative microscopic pulp section image.

[0190] Optionally, as shown in Figure 10 , the method further comprises:

[0191] S701, obtaining the original pulp section image under the current intraoperative microscope.

[0192] S702, perform image enhancement processing on the original pulp section image using a pre-set at least one image enhancement transformation strategy to generate an enhanced image set.

[0193] Among them, the enhanced image set includes: the pulp section transformation image under each image enhancement transformation strategy.

[0194] Exemplarily, the image enhancement transformation strategy can include: geometric transformation type enhancement (elastic deformation, affine transformation, perspective transformation), pixel level enhancement (random erasing, Cutout, Mixup, CutMix), color space enhancement, etc. Among them, the specific parameter setting can be adjusted according to the data characteristics, such as the deformation coefficient range of elastic deformation 0.05-0.3, the erasing area ratio of random erasing 5%-15%, and the rotation angle range ±5° to ±30°. These enhancement methods can be used alone or in combination, and the variability of microscope shooting in actual clinical practice can be simulated by introducing appropriate randomness.

[0195] S703, respectively pre-process each pulp section transformation image to obtain a pre-processed image corresponding to each pulp section transformation image.

[0196] S704, input the pre-processed image corresponding to each pulp section transformation image into the target pulp state recognition model to obtain a predicted recognition result corresponding to each pulp section transformation image.

[0197] S705, determine the predicted recognition result of the original pulp section image according to the predicted recognition result corresponding to each pulp section transformation image.

[0198] In an implementable manner, the target pulp state recognition model trained as described above can be used to recognize the original pulp section image under the current intraoperative microscope, and the specific process is as follows:

[0199] The original pulp section image taken by the microscope can be recognized, and the image region containing the complete pulp section can be cropped according to the rectangular frame coordinates labeled by experts or automatic detection algorithm, and irrelevant background and instrument interference can be removed. The cropped image keeps the pulp section centered, and the boundary has a proper margin to avoid loss of key information.

[0200] Then, in order to improve the accuracy of the recognition result, it is proposed that inference fusion can be performed from multiple perspectives, that is, by using multiple image enhancement transformation strategies, the original pulp section image is processed to generate an enhanced image set. Exemplarily, based on the characteristics of the pulp microscopic image and the variability of the microscope shooting conditions, six enhancement transformations covering geometric transformation, optical transformation and noise simulation are designed:

[0201] Original pulp section image: as a baseline control, no transformation is performed;

[0202] Horizontal flip: mirror image along the horizontal mid-axis of the pulp section image, simulate the left-right difference of the microscope observation angle, get the horizontal flip image of the pulp section;

[0203] Rotation transformation: select 2 angles from the angle set {-15°, -10°, +10°, +15°} for rotation, simulate the slight tilt of the microscope head or the incomplete verticality of the pulp section, after rotation, use bilinear interpolation to fill the boundary, get the rotation transformation image of the pulp section;

[0204] Brightness adjustment: increase the overall brightness of the image by 15%, simulate the intensity fluctuation of the microscope light source, realize , and truncate to the range [0, 255], get the brightness adjustment image of the pulp section;

[0205] Contrast adjustment: reduce the contrast of the image by 15%, simulate the aging of the lens or the pollution of the light path, realize the formula , get the contrast adjustment image of the pulp section;

[0206] Gaussian noise: add Gaussian white noise with mean 0 and standard deviation σ=5, simulate the sensor noise, get the Gaussian noise image of the pulp section.

[0207] Therefore, the original pulp section image can be applied to the transformed images generated by the six transformations in turn to construct an enhanced image set . And each transformed image is uniformly preprocessed, resized to 448x448 pixels, pixel values are normalized to [0,1], and Z-score standardization uses training set statistics.

[0208] Referring to Figure 11 , input the six pulp transformed images into the target pulp state recognition model respectively, and based on the prediction recognition results of each pulp transformed image, get the prediction recognition result of the original pulp section image.

[0209] Optionally, the above step S705 comprises:

[0210] The prediction recognition results corresponding to each pulp section transformed image are weighted, and the weighted results are taken as the prediction recognition result of the original pulp section image.

[0211] In an implementable manner, the prediction recognition results of each pulp transformed image output by the target pulp state recognition model can be obtained, that is, six groups of independent probability outputs .

[0212] Wherein, represents the probability that the ith transformed image is judged as "qualified", denotes the probability of being judged as "unqualified".

[0213] To comprehensively utilize the evaluation results from different enhanced perspectives, the application adopts a weighted arithmetic average strategy to fuse the above six independent probabilities, that is, according to the contribution and reliability of the evaluation results of different enhancement transformations to the image quality, corresponding weights are given to obtain more stable and accurate final evaluation probability, and therefore, the final predicted recognition result of the original dental pulp section image is shown in formula (18) as follows:

[0214] (18)

[0215] Among them, the weight setting strategy is as follows:

[0216] Original image (P0) ): The weight is set to 1.5. Since the original image is not subjected to any distortion processing, its evaluation result is considered to be the most direct and reliable, and therefore the highest weight is given.

[0217] Horizontal flip (P1) ) and rotation transformation (P2 ): The weights are both set to 1.2. Under the premise of keeping the integrity of the core structure and content of the image, these two transformations simulate the slight change of the observation angle, and their evaluation results have high reference value.

[0218] Brightness adjustment (P3 ) and contrast adjustment (P4 ): The weights are both set to 1.0. These two transformations simulate common optical parameter changes, and their evaluation results are considered as standard weights.

[0219] Gaussian noise (P5 ): The weight is set to 0.8. Since noise addition introduces additional interference, although it helps to improve the robustness of the model, its direct reference value of the evaluation result is slightly lower than that of other transformations in the final fusion, and therefore the weight is appropriately reduced.

[0220] Therefore, by designing a threshold , when , the image is recognized as a qualified image, otherwise it is recognized as an unqualified image.

[0221] Optionally, the application can also support multi-resolution input (such as 224x224 / 512x512 pixels), be compatible with non-4K microscopes (such as 1080P images processed by super-resolution enhancement), and the output interface can provide various forms of result display, including binary classification results, continuous probability values, confidence intervals, multi-class probability distributions, etc. The visual output can use class activation mapping (CAM).

[0222] Figure 12A structural schematic diagram of an electronic device is provided in the embodiments of the present application. The electronic device can be integrated in a terminal device or a chip of the terminal device. The terminal device can be a computing device with a data processing function.

[0223] The electronic device includes a processor 1201 and a memory 1202.

[0224] The memory 1202 is configured to store a program. The processor 1201 invokes the program stored in the memory 1202 to execute the above-mentioned method embodiments. The specific implementation manners and technical effects are similar, and thus are not described herein.

[0225] Optionally, the present application further provides a program product, for example, a computer readable storage medium, including a program. The program, when executed by a processor, is configured to execute the above-mentioned method embodiments.

[0226] The above merely provides preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included in the scope of the present application.

[0227] It is obvious for those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, but can be implemented in other concrete forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be regarded as exemplary and non-limiting, the scope of the present application is defined by the appended claims rather than the above description, and all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present application. Any reference signs in the claims should not be regarded as limiting the involved claims.

Claims

1. A method for training a dental pulp state recognition model, characterized in that, The method includes: Acquire multiple pulp section images under a historical intraoperative microscope and determine the actual identification result of each pulp section image, wherein the actual identification result includes: suitable for pulp capping or unsuitable for pulp capping; Each of the aforementioned pulp cross-sectional images is input into a pre-constructed initial pulp state recognition model, which includes: a pre-trained feature encoder and an initial classifier; The feature encoder extracts local and global features corresponding to each pulp section image, and performs bilinear interactive processing on the local and global features corresponding to each pulp section image to obtain a fused feature vector corresponding to each pulp section image. The fused feature vector contains the correlation information between the local and global features. The initial classifier outputs a prediction and recognition result based on the fused feature vectors corresponding to each of the pulp cross-sectional images; The initial classifier is iteratively trained based on the predicted recognition results and the actual recognition results, and the initial pulp state model at the end of the iteration is used as the target pulp state recognition model; wherein, a parameter freezing strategy is adopted during the iterative training process to keep the network parameters of the feature encoder unchanged, and the network parameters of the initial classifier are iteratively updated.

2. The method according to claim 1, characterized in that, The initial pulp state recognition model also includes: a pre-trained position encoder; The extraction of local and global features corresponding to each pulp cross-section image by the feature encoder includes: The position encoder performs position encoding on the pulp cross-sectional image to obtain a high-dimensional feature vector corresponding to the pulp cross-sectional image; The high-dimensional feature vector corresponding to the pulp cross-section image is input into the feature encoder, and the feature encoder extracts the intermediate local features and global features corresponding to the pulp cross-section image. The intermediate local features are input into a pre-built local attention learning module to obtain the local features corresponding to the pulp cross-section image.

3. The method according to claim 2, characterized in that, The local attention learning module includes: a multilayer perceptron network, a normalization layer, and a weighted aggregation layer; The step of inputting the intermediate local features into a pre-constructed local attention learning module to obtain the local features corresponding to the pulp cross-section image includes: The intermediate local features are input into the multilayer perceptron network to obtain the importance score of each feature vector in the intermediate local features; The importance scores of each feature vector are normalized by the normalization layer to obtain the attention weights corresponding to each feature vector. The weighted aggregation layer sums the attention weights corresponding to each feature vector to obtain the local features corresponding to the pulp cross-section image.

4. The method according to claim 1, characterized in that, The step of performing bilinear interactive processing on the local and global features corresponding to each pulp cross-section image to obtain the fused feature vector corresponding to each pulp cross-section image includes: By using a pre-constructed linear projection layer, the local features and the global features are projected onto a low-dimensional space to obtain low-dimensional global features and low-dimensional local features. The low-dimensional global features and the low-dimensional local features are multiplied element-wise to capture higher-order correlations, resulting in a fused feature vector corresponding to the pulp cross-section image.

5. The method according to claim 1, characterized in that, The process involves the initial classifier outputting a predicted recognition result based on the fused feature vector corresponding to each pulp cross-section image, and iteratively training the initial classifier based on the predicted recognition result and the actual recognition result, including: The dimensionality reduction of the fusion feature vectors corresponding to each of the pulp cross-sectional images is performed to obtain the low-dimensional fusion feature vectors corresponding to each of the pulp cross-sectional images. Normalization and nonlinear processing are performed on the low-dimensional fusion feature vectors corresponding to each pulp cross-section image to obtain the nonlinear feature vectors corresponding to each pulp cross-section image. The nonlinear feature vectors corresponding to each of the pulp cross-sectional images are input into the initial classifier to obtain the prediction and recognition results corresponding to each of the pulp cross-sectional images; Based on the predicted recognition results and actual recognition results corresponding to each pulp cross-section image, the loss information of the initial pulp state recognition model is determined, and the initial classifier is iteratively corrected according to the loss information until the corrected initial classifier meets the preset convergence condition, thus obtaining the classifier.

6. The method according to claim 5, characterized in that, The step of determining the loss information of the initial pulp state recognition model based on the predicted recognition results and actual recognition results corresponding to each pulp cross-section image includes: Obtain a pre-constructed joint loss function, which is obtained by weighted fusion of a classification loss optimization function and a feature discrimination optimization function; The predicted recognition result and the actual recognition result corresponding to the pulp cross-section image are input into the joint loss function to obtain the loss information of the initial pulp state recognition model.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: Obtain the original cross-sectional image of the dental pulp under the current intraoperative microscope; The original pulp section image is enhanced using at least one pre-defined image enhancement transformation strategy to generate an enhanced image set, which includes pulp section transformation images under each of the image enhancement transformation strategies. Each of the described pulp cross-sectional transformation images is preprocessed to obtain the preprocessed image corresponding to each of the described pulp cross-sectional transformation images; The preprocessed images corresponding to each of the pulp cross-sectional transformation images are input into the target pulp state recognition model to obtain the prediction and recognition results corresponding to each of the pulp cross-sectional transformation images; Based on the prediction and recognition results corresponding to each of the described pulp section transformation images, the prediction and recognition results of the original pulp section image are determined.

8. The method according to claim 7, characterized in that, The step of determining the prediction and recognition result of the original pulp section image based on the prediction and recognition results corresponding to each of the pulp section transformation images includes: The prediction and recognition results corresponding to each of the pulp section transformation images are weighted and the weighted results are used as the prediction and recognition results of the original pulp section image.

9. An electronic device, characterized in that, include: Memory, used to store one or more programs; A processor; when the one or more programs are executed by the processor, to implement the method as described in any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-8.

Images