Tongue picture classification method, tongue picture model acquisition method and electronic equipment

Abstract

The invention relates to the technical field of image classification, and discloses a tongue picture classification method, a tongue picture classification model acquisition method and electronic equipment, and the tongue picture classification model acquisition method comprises the steps: inputting a preprocessed unlabeled tongue picture image into a neural network model for self-supervised learning; masking the unlabeled tongue picture image according to the reconstruction mask loss to obtain a pre-trained neural network model; inputting the labeled tongue picture image into a pre-trained neural network model for supervised learning; guiding supervised learning of a pre-trained neural network model based on a centroid classifier to obtain a trained neural network model; and testing the trained neural network model, and taking the neural network model meeting a test condition as a tongue picture classification model. According to the tongue picture classification model acquisition method, the unlabeled tongue picture image and the labeled tongue picture image are fully utilized to complete knowledge-assisted model learning, and the accuracy of small sample learning in a tongue picture classification task is improved.

Description

Technical Field

[0001] The present application relates to the technical field of image classification, and in particular to a tongue image classification method, a model acquisition method thereof, and an electronic device. Background Art

[0002] Research has shown that tongue image features should be sparse, regional, and visually similar. Different tongue features should be attributed to different regions, and disease diagnosis based on tongue image data often relies on the characteristics of these different regions. Currently, tongue image classification is difficult to obtain high-quality tongue images, and the classification and annotation of tongue images requires specialized and experienced personnel. This makes tongue image classification and annotation in the field of artificial intelligence (AI) high-cost and incapable of accurately classifying tongue images from even a small number of images. Summary of the Invention In view of this, the embodiments of the present application provide a tongue image classification method and a model acquisition method thereof, and an electronic device, aiming to achieve tongue image classification using a small amount of annotated tongue image data.

[0003] In a first aspect, an embodiment of the present application provides a method for obtaining a tongue image classification model, comprising: The preprocessed unlabeled tongue images are input into the neural network model for self-supervised learning; masking the unlabeled tongue image according to the reconstruction mask loss of the unlabeled tongue image; Training the neural network model based on the masked unlabeled tongue image to obtain a pre-trained neural network model; Inputting the labeled tongue image into the pre-trained neural network model for supervised learning; Training a centroid classifier based on the labeled tongue image to guide supervised learning of the pre-trained neural network model based on the centroid classifier to obtain a trained neural network model; The trained neural network model is tested, and the neural network model that meets the test conditions is used as the tongue image classification model.

[0004] In a first possible embodiment of the first aspect, the further comprising: performing image segmentation on the unlabeled tongue image to obtain a plurality of image blocks of each of the unlabeled tongue images; Random masking is performed on each of the image blocks in each of the unlabeled tongue images based on the same first mask ratio, so as to input the unlabeled tongue images after image segmentation and masking into the neural network model respectively.

[0005] In a second possible embodiment of the first aspect, inputting the preprocessed unlabeled tongue image into a neural network model for self-supervised learning includes: Obtaining image features of the unlabeled tongue image after image segmentation through the encoder of the neural network model; splicing the predefined clustering features with the image features of the unlabeled tongue image; Inputting the spliced ​​features into the independent attention mechanism and the multi-layer neural network component in sequence to obtain the cluster center of the unlabeled tongue image; The image features of the unlabeled tongue image after masking are obtained through the encoder of the neural network model.

[0006] In a third possible embodiment of the first aspect, adjusting the mask ratio of the unlabeled tongue image according to the reconstruction mask loss of the unlabeled tongue image includes: performing regional clustering on the image blocks in each of the unlabeled tongue images based on the cluster centers; Calculating the mask reconstruction loss of different regions of each of the unlabeled tongue images; The mask ratio of the image blocks in each region is determined according to the mask reconstruction loss of each region, and the image blocks in each region are re-masked based on the determined mask ratio.

[0007] In a fourth possible embodiment of the first aspect, the training of a centroid classifier based on the labeled tongue image includes: Extracting a feature vector of the labeled tongue image through the pre-trained neural network model; Taking the mean of the feature vectors of the labeled tongue images of each tongue image category as the initial centroid vector of the centroid classifier; Calculating the similarity between the cluster center obtained by the self-supervised learning of the neural network model and the initial centroid vector; The cluster center is fused with the initial centroid vector according to the similarity and a preset weight to obtain a target centroid vector.

[0008] In a fifth possible embodiment of the first aspect, guiding the supervised learning of the pre-trained neural network model according to the centroid classifier includes: Calculating the distance between the target centroid vector of the centroid classifier and the feature vector of the labeled tongue image, so as to adjust the parameters of the neural network model through back propagation according to the distance; Determining a first probability distribution of the prediction of the labeled tongue image by the centroid classifier, and determining a second probability distribution of the prediction of the labeled tongue image by the neural network model; Supervised learning is performed on the neural network model by minimizing the divergence loss between the first probability distribution and the second probability distribution.

[0009] In a sixth possible embodiment of the first aspect, determining a first probability distribution predicted by the centroid classifier for the labeled tongue image includes: Converting each of the distances into a prediction score of each of the labeled tongue images belonging to each of the tongue image categories using a probability distribution function to obtain a first probability distribution; The smoothness of the first probability distribution is controlled by a temperature parameter of the probability distribution function.

[0010] In a seventh possible embodiment of the first aspect, testing the trained neural network model includes: Fine-tuning the parameters of the neural network model based on the support set of the test task; The generalization performance of the neural network model on new samples is evaluated based on the query set of the test task.

[0011] In a second aspect, embodiments of the present application provide a tongue image classification method, comprising: The tongue image to be classified is input into the above-mentioned tongue image classification model, and the tongue image classification result is output.

[0012] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores a computer program, and the processor is configured to execute the computer program to implement the above-mentioned tongue image classification model acquisition method and tongue image classification method.

[0013] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program. When the computer program is executed on a processor, the above-mentioned tongue image classification model acquisition method and tongue image classification method are implemented.

[0014] The embodiments of the present application have the following beneficial effects: This embodiment provides a method for acquiring a tongue image classification model, including: inputting preprocessed unlabeled tongue image images into a neural network model for self-supervised learning; masking the unlabeled tongue image images according to a reconstruction mask loss to obtain a pretrained neural network model; inputting labeled tongue image images into the pretrained neural network model for supervised learning; training a centroid classifier based on the labeled tongue image images, guiding the supervised learning of the pretrained neural network model based on the centroid classifier, to obtain a trained neural network model; testing the trained neural network model, and selecting the neural network model that meets the test criteria as the tongue image classification model. This method for acquiring a tongue image classification model fully utilizes both unlabeled and labeled tongue image images to perform knowledge-assisted small-sample model learning, thereby improving the accuracy of small-sample learning in tongue image classification tasks, both with and without labeled tongue image images. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0016] Figure 1 A first flow chart of the method for obtaining a tongue image classification model according to an embodiment of the present application is shown; Figure 2 A schematic diagram of tongue image segmentation according to an embodiment of the present application is shown; Figure 3 A second flow chart of the method for obtaining a tongue image classification model according to an embodiment of the present application is shown; Figure 4 A schematic diagram of the process of obtaining cluster centers in an embodiment of the present application is shown; Figure 5 A third flow chart of the tongue image classification model acquisition method according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0017] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0018] The components of the embodiments of the present application generally described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed application, but rather merely represents selected embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0019] Hereinafter, the terms "including", "having" and their cognates used in various embodiments of the present application are intended only to indicate specific features, numbers, steps, operations, elements, components or combinations of the aforementioned items, and should not be understood as excluding the existence of one or more other features, numbers, steps, operations, elements, components or combinations of the aforementioned items or adding the possibility of one or more features, numbers, steps, operations, elements, components or combinations of the aforementioned items. In addition, the terms "first", "second", "third" and the like are only used to distinguish descriptions and should not be understood as indicating or implying relative importance.

[0020] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by those skilled in the art to which the various embodiments of the present application belong. Terms (such as those defined in generally used dictionaries) will be interpreted as having the same meaning as in the context of the relevant technical field and will not be interpreted as having an idealized meaning or an overly formal meaning unless clearly defined in the various embodiments of the present application.

[0021] The following describes some embodiments of the present application in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments may be combined with each other.

[0022] Common tongue classifications include pale tongue, red tongue, yellow tongue coating, tooth-marked tongue, and fissured tongue. Currently, the field of artificial intelligence is capable of training large amounts of labeled tongue image data to achieve tongue classification. Since labeling tongue image data requires highly experienced personnel, research on how to classify diseases using small amounts of labeled data has practical applications.

[0023] Existing methods use unsupervised learning on natural image datasets. Method 1 utilizes a self-supervised approach based on contrastive learning, enabling the model to learn universal visual representations through contrastive learning on unlabeled ImageNet images. This process involves performing augmentation operations such as random cropping, color transformation, and Gaussian blurring on the images. Representations are then generated through an encoder network and a nonlinear transformation network, and the model is optimized using a contrastive loss. This allows the model to distinguish between different image views and learn feature representations. Contrastive learning is performed based on the availability of unlabeled medical / tongue images. Specifically, if multiple images are available for each tongue image type, a novel multi-instance contrastive learning approach is employed. This approach constructs positive pairs by extracting two crops from different tongue images of the same patient, rather than from augmented views of the same image as in traditional approaches. This allows the model to learn invariant representations of the same pathology across different images, making it more adaptable to the multi-view and multi-conditional scenarios of medical images and further improving its understanding and representation of medical image features. The model weights are then initialized using pre-trained model weights, and end-to-end training is performed on a labeled tongue image dataset to adapt it to specific image classification tasks. During fine-tuning, data augmentation operations such as random color augmentation, cropping, rotation, and flipping are also performed to improve the model's generalization ability. This method is task- and dataset-specific, aiming to enable the model to learn features and classification boundaries relevant to specific medical tasks.

[0024] There is also a second way. First, the public dataset MiniImageNet is used to perform meta-learning training on the classification model, and the general knowledge of the classification task is learned from the large-scale dataset. Then, the labeled tongue image data of the source domain and a small amount of labeled data of the target domain are used to perform meta-learning training on the classification model, so that it can quickly adapt to the characteristics of the new tongue image data samples in the target domain. ResNet18 is selected as the backbone network, and a metric-based meta-learning method is used for training. The data is divided into a meta-training set and a meta-test set, and then further divided into a support set and a query set. The loss function is calculated by shortening the distance between the sample in the query set and the corresponding class centroid to update the model parameters. A large number of unlabeled samples are predicted, and the prediction results are used as pseudo labels. Three heterogeneous network structures, Resnet18, SACA-Resnet18, and E-CA2-ResNet18, are used for model ensemble learning. The ensemble learning adopts the voting method to screen out high-quality pseudo labels. Finally, the robust symmetric cross entropy is used. The network is optimized and trained using the Single Entropy (SCE) loss function and the Comparative Regularization (CTR) function to suppress the impact of noise samples on classification performance.

[0025] Existing self-supervised methods primarily employ contrastive learning or masked reconstruction. Contrastive learning relies on data augmentation, which can damage local areas of the tongue (such as the tongue coating and tongue edge), affecting the accuracy of tongue classification. Masked reconstruction employs global random masking and uses unmasked areas for reconstruction. Studies have shown that directly using global random masking reduces the model's attention to sparse pathological regions, resulting in poor pre-training performance. To overcome these issues, Li et al. proposed using a reconstruction loss to identify key masked regions, while Xu et al. used the reconstruction loss as a reward signal to identify areas requiring focused masking. While these methods have achieved promising results in medical imaging, they have limitations in their application to tongue image data. First, because pathological features in tongue image data are localized and visually similar, random masked reconstruction can result in falsely high loss in features of some regions, potentially misleading the model into focusing too much attention on non-lesional areas. Furthermore, unlike anatomical regions such as organs and tumors in medical imaging data, which typically have higher loss than air- and fluid-filled regions and warrant greater attention, the potential role of features corresponding to low-loss regions in tongue images in downstream tasks cannot be ignored.

[0026] This application, based on a VIT model (Vision Transformer) trained on natural datasets, collects publicly available tongue image datasets, removes labels, and constructs an unlabeled tongue image dataset, which serves as the data foundation for pre-training. Then, through mask reconstruction using unsupervised methods, this approach learns general visual representations while also capturing more detailed features, making it effective for distinguishing different images. Subsequently, small-sample training tasks are divided based on labeled tongue image images, and the model is trained on a small sample. Finally, the model is tested on a set of labeled tongue image images. During the model training process, a more suitable model can be selected to complete the small-sample image classification task based on the difficulty of the task.

[0027] The method for obtaining the tongue image classification model is described below with reference to some specific embodiments.

[0028] Figure 1 A flow chart of a tongue image classification model acquisition method according to an embodiment of the present application is shown. Exemplarily, the tongue image classification model acquisition method includes the following steps: S110, inputting the preprocessed unlabeled tongue image into the neural network model for self-supervised learning.

[0029] In one embodiment, the present application performs image segmentation on the unlabeled tongue image to obtain multiple image blocks for each unlabeled tongue image. In the image preprocessing, the original unlabeled tongue image is first segmented to remove irrelevant background information and reduce unnecessary interference. The segmented tongue image is further segmented into the tongue body and the tongue periphery. The pre-training data set is then expanded in a multi-view manner. The composition is as follows: Figure 2 This application further segments the unlabeled tongue image into several fixed-size image blocks (patch blocks). For example, if the image size of the unlabeled tongue image is 640x640, the size of each patch block after segmentation is 16x16.

[0030] In another embodiment, the present application performs random masking on each image block in each unlabeled tongue image based on the same first mask ratio, so as to input the unlabeled tongue image after image segmentation and masking into the neural network model respectively.

[0031] In this example, when processing multiple unlabeled tongue images, to ensure training consistency and fairness, each unlabeled tongue image is assigned the same initial mask ratio (for example, 50%), resulting in the same proportion of image blocks being masked in each image. The neural network model is the VIT model, an image processing model based on the Transformer architecture.

[0032] For example, the VIT model extracts global and local features based on the complete information of the divided image blocks (unmasked), focusing on analyzing the detailed features of each block. While acquiring the image blocks, the VIT model memorizes global features (CLS tokens), referred to here as cluster centers. Cluster centers are used to calculate subsequent block aggregation (representing local information) and category aggregation (representing the complete image information).

[0033] In one embodiment, if Figure 3 As shown, the unlabeled tongue image of this application is input into the neural network model for self-supervised learning, including the following steps: S111, obtaining image features of the unlabeled tongue image after image segmentation through the encoder of the neural network model.

[0034] In one embodiment, in a specific pre-training model process, the encoder of the VIT model is used to map the unlabeled tongue image after image segmentation, and the features of the unlabeled tongue image after image segmentation are mapped to the unlabeled tongue image. Through linear projection to the corresponding feature space The formula is summarized as follows:

[0035] in, is the feature vector of the image block, is a linear projection function that maps the input image features from the original space to the target feature space. For the Input image features, are model parameters, including weight matrix and bias vector. By adjusting the model parameters, effective conversion from the features of the input image to the target feature space can be achieved.

[0036] S112, combining the predefined clustering features with the image features of the unlabeled tongue image.

[0037] In this embodiment, the applicant initializes multiple clustering features to identify trainable clustering tag sequences and classify different parts of the image into specific categories.

[0038] S113, the spliced ​​features are sequentially input into the independent attention mechanism and the multi-layer neural network component to obtain the cluster center of the unlabeled tongue image.

[0039] In one embodiment, if Figure 4 As shown, this application combines predefined clustering features with the image feature vectors of unlabeled tongue images to form a new feature representation. This application then inputs the concatenated features into a separate attention mechanism. The core idea of ​​the attention mechanism is to enable the model to "focus" on the most important parts of the input data while ignoring the less important parts. In this application, the independent attention module is responsible for selecting the most useful features for the classification task from the concatenated features (clustering token + feature vector extracted by the VIT encoder) and performing weighted processing on the concatenated features.

[0040] The weighted features are fed into multiple Transformer Blocks for further feature transformation. This application processes the input features layer by layer by stacking multiple Transformer Blocks. TransformerBlock is used to perform multi-layer nonlinear transformations on the input features and gradually optimize the feature representation. During this process, the model continuously adjusts the parameters according to the training objectives (such as the clustering loss function) to make the final feature representation more compact and discriminative. The design of N Transformer Blocks allows the model to gradually deepen its understanding of the features, thereby improving its learning ability. The model ultimately outputs a set of optimized cluster centers. The cluster centers not only reflect the distribution characteristics of the input data, but also enhance the separability of the features through self-supervised learning. The cluster centers obtained in this way actually represent the pre-trained latent categories of unlabeled tongue images. Although there are no labels, they can be used as a weak enhancement method to affect the classification task during downstream tasks.

[0041] Among them, Transformer Block is the core component of the Transformer model. Each TransformerBlock includes layer normalization (Layer Norm), multi-head attention mechanism (Multi-Head Attention), layer normalization and multi-layer perceptron (MLP) in sequence. Layer normalization is used to standardize the input features, the multi-head attention mechanism is used to capture the global relationship between the input features, and layer normalization is used to standardize the output of the multi-head attention mechanism. The multi-layer perceptron is used to perform nonlinear transformation on the input features to further enhance the feature expression capability. Each Transformer Block also includes residual connections. The function of the residual connection is to add the layer normalization input to the output of the multi-head attention mechanism and the multi-layer perceptron respectively, helping the model to learn and optimize better. In this application, the goal of each Transformer Block is to extract more meaningful feature representations by optimizing the input features layer by layer. After being processed by multiple Transformer Blocks, these feature representations are finally used to generate optimized cluster centers.

[0042] In one embodiment, the plurality of cluster center sets are represented as:

[0043] in, is the artificially initialized clustering feature ( , dimension )gather, represents the first clustering feature, represents the second clustering feature, represents the p-th cluster feature, represents a D-dimensional real vector space, Indicates the The clustering features are D-dimensional real vector spaces.

[0044] In one embodiment, the present application performs a regional clustering operation by concatenating the features obtained from the cluster center and the corresponding feature space and calculating the similarity, which is in the following form:

[0045] Indicates that according to the input feature vector and cluster center set , calculate the similarity between the input feature vector and each cluster center, and assign the input feature vector to the category represented by the cluster center that is most similar to it, Represents the input feature vector Hedi The cluster center Euclidean distance squared, Indicates finding the smallest Euclidean distance squared and finding the cluster center index with the smallest distance .

[0046] In one embodiment, the present application optimizes the distribution of cluster centers through a clustering loss function during the training process. The clustering loss is expressed as:

[0047] in, represents the clustering loss function, which is used to optimize the compactness and discrimination of cluster centers. Represents the distribution of input data The expectation of (i.e., the average over the entire dataset), , represents the corresponding weight, Indicates the The sum of the intra-class distances of the cluster centers, that is, the distances between them and all image block features in the cluster to which they belong, ensures that the samples in each cluster are more concentrated, thereby improving the representativeness of the cluster centers. Represents the cluster center and The distance metric function (mixed Euclidean + cosine) is used to control the distribution between cluster centers and avoid cluster centers being too close.

[0048] S114, obtaining the image features of the unlabeled tongue image after mask processing through the encoder of the neural network model.

[0049] In one embodiment, the VIT model uses masked image blocks to predict masked content from the unmasked portions. The model learns how to infer unknown content based on known portions, helping it better understand the overall structure of the image. In this application, even without annotations, the model can learn rich feature information from these two input modes, significantly reducing its reliance on labeled data.

[0050] S120 , masking the unlabeled tongue image according to the reconstruction mask loss of the unlabeled tongue image.

[0051] Exemplarily, during the training process, the present application dynamically adjusts the masking strategy based on the performance of the model. For high-loss areas (where the model prediction error is large), the mask ratio can be increased to allow the model to pay more attention to these areas.

[0052] In one embodiment, the present application performs regional clustering of image blocks in each unlabeled tongue image based on cluster centers; calculates mask reconstruction losses of different regions of each unlabeled tongue image; determines the mask ratio of image blocks in each region based on the mask reconstruction loss of each region, and re-masks the image blocks in each region based on the determined mask ratio.

[0053] In one embodiment, after obtaining global features, the present application locates similar regions (such as the tongue body, edges, etc.) in the unlabeled tongue image by calculating the similarity between the cluster center and the features of each image block. For example, the similarity between the cluster center and the features of each image block can be calculated by cosine similarity or dot product similarity. The present application performs cluster analysis on the image blocks based on the similarity score and divides patches with the same similarity score into the same similar region. The patches in each similar region are initially masked with the same first mask ratio (such as 50%). For example, the masked patch is replaced with a zero vector, or the masked patch is replaced with a learnable mask marker.

[0054] In another embodiment, the present application calculates the reconstruction loss for each similar region, determines a second mask ratio based on the reconstruction loss for each similar region, and masks each image block in the similar region according to the second mask ratio. Reconstruction loss is an important metric for measuring a model's ability to recover masked data and is used to evaluate the model's recovery effect on masked patches.

[0055] In one embodiment, during the training of the few-shot model, masked patches are reconstructed. In the VIT model, the decoder reconstructs the original image, and the loss function calculates the mean squared error between the masked pixels in the original and reconstructed images. Regions with higher reconstruction loss indicate that the model has greater difficulty recovering their information. For similar regions with higher reconstruction loss, a larger second mask ratio is used to mask the image patches in similar regions. Higher loss results in a higher mask ratio being assigned.

[0056] It can be understood that this application is based on the clustering center of the unlabeled tongue image to capture similar areas from the global level of the tongue image. Through the mask strategy guided by the clustering center, the model training can better understand the overall semantic structure of the tongue image and learn a more discriminative global feature representation.

[0057] S130: Training the neural network model based on the masked unlabeled tongue image to obtain a pre-trained neural network model.

[0058] In this example, as training progresses, the model gradually identifies areas that are more difficult to reconstruct (i.e., high-loss areas) and applies more masks to these areas. This dynamic adjustment allows the model to focus more on diagnostically valuable pathological areas (such as changes in tongue coating color or the location of tooth marks). Through this training process, the model gradually learns how to infer unknown content from a small amount of known information, thus completing pre-training.

[0059] S140: Input the labeled tongue image into the pre-trained neural network model for supervised learning.

[0060] S150, training a centroid classifier based on the labeled tongue image, guiding the supervised learning of the pre-trained neural network model based on the centroid classifier, and obtaining a trained neural network model.

[0061] Exemplarily, the centroid classifier is a classification method based on the cluster center of the sample feature vector. In this application, the centroid vector of each tongue image category (that is, the average value of the feature vectors of all labeled tongue image images of the tongue image category) is calculated, and the first probability distribution of the tongue image category is determined based on the centroid vector to guide model training.

[0062] In the embodiments of the present application, supervised learning of a pre-trained neural network model includes a test task and a training task, each of which includes a support set and a query set. The support set consists of randomly extracting five tongue image categories from the labeled tongue image data, and extracting a number of labeled tongue image data from each tongue image category. The query set consists of the same number of labeled tongue image images from each tongue image category.

[0063] In one embodiment, in a small-sample learning centroid classifier method, a centroid classifier is first trained using labeled tongue images. This application extracts feature vectors of the labeled tongue images using a pre-trained VIT model, and uses the mean of the feature vectors of the labeled tongue images for each tongue category as the centroid vector of the centroid classifier.

[0064] In another embodiment, the present application generates multiple cluster centers in the pre-training stage, and combines them with the features of downstream tasks to construct enhanced feature representation to alleviate the problem of separation between feature representation and downstream tasks in self-supervised learning. In the downstream task stage, the present application calculates the similarity (such as cosine similarity or Euclidean distance) between the cluster centers obtained by self-supervised learning of the neural network model and the initial centroid vector, and fuses the cluster centers with the initial centroid vector according to the similarity and preset weights to obtain the target centroid vector, forming weighted fused cluster information. The weighted fused cluster information is used for subsequent classification judgments to alleviate the inherent defects of self-supervision and the problem of separation between the features obtained by self-supervised pre-training and downstream tasks, aiming to improve the performance of downstream classification tasks.

[0065] In one embodiment, multiple cluster centers obtained during the pre-training process are used as additional guidance to calculate the similarity between the centroid vectors of different centroid classifiers and the cluster centers. The similarity results are used as additional guidance signals. For example, the cluster centers can be incorporated into the optimized weighted fusion cluster information through weighted averaging or other combined methods. The preset weight (e.g., 0.2) serves to balance the contribution between explicit category information (provided by the centroid classifier) ​​and implicit category information (reflected by the cluster centers). By introducing a weight parameter (e.g., 0.2), the similarity results are proportionally incorporated into the optimization process of the centroid features, which not only enhances the model's ability to utilize implicit category information but also improves the flexibility and adaptability of the method.

[0066] In one embodiment, the present application calculates the distance between the centroid vector of the centroid classifier and the feature vector of the labeled tongue image, which is the Euclidean distance between the centroid vector and the feature vector of the labeled tongue image. Figure 5 As shown, the present application determines a first probability distribution of the centroid classifier's prediction of the labeled tongue image, and determines a second probability distribution of the neural network model's prediction of the labeled tongue image, and performs supervised learning on the neural network model by minimizing the divergence loss of the first probability distribution and the second probability distribution.

[0067] In one embodiment, the present application converts each distance into a predicted score of each tongue image belonging to each tongue image category through a probability distribution function to obtain a first probability distribution, and controls the smoothness of the first probability distribution through the temperature parameter of the probability distribution function. In this embodiment, the distance between the centroid vector and the tongue image with the label is calculated by using a temperature parameter. The softmax function is converted into a prediction score, for example, calculating the distance between the labeled tongue image and the centroid of "yellow fur" After that, the score is , is the distance between the labeled tongue image and the centroid vector of the zth category.

[0068] The prediction scores of the labeled tongue image belonging to each tongue image category not only reflect the probability distribution of the labeled tongue image belonging to each category, but also control the smoothness of the distribution through the temperature parameter. For example, in the classification task of 5 tongue image categories, the distances between a tongue image sample and the centroid of the 5 categories are [1.2, 0.8, 1.5, 2.0, 1.0] respectively. =1, the distance to the nearest class centroid of the tongue image sample (distance 0.8) will obtain a probability weight of about 35%, which is the highest probability weight. The concentration of the probability distribution can be changed by adjusting the temperature parameter.

[0069] In another embodiment, the present application uses the support set as input to a pre-trained VIT model to train the model's classification capabilities in small sample sizes. During the training phase, the model learns the sample features and category information in the support set to generate a predicted distribution for the samples in the support set, i.e., a second probability distribution.

[0070] In this embodiment, the prediction score plays a key role in model training. On the one hand, it serves as a supervisory signal to guide the model to learn more discriminative feature representations (bringing features of similar samples closer to their centroids). In the centroid classifier method, the supervisory signal guides model learning in two ways: first, the model calculates the distance between sample features and the centroids of each class and adjusts the network parameters through backpropagation to bring sample features closer to the centroids of similar classes (reducing the distance between similar classes) and away from the centroids of different classes (increasing the distance between different classes).

[0071] In one embodiment, the present application calculates the distance between the target centroid vector of a centroid classifier and the feature vector of a labeled tongue image, and adjusts the network parameters of a neural network model based on this distance through backpropagation. In this embodiment, the model parameters are adjusted through backpropagation to reduce the distance between the labeled tongue image and the centroid vector for similar features and increase the distance between the labeled tongue image and the centroid vector for dissimilar features, thereby aligning sample features closer to the centroid of similar features and away from the centroid of dissimilar features.

[0072] Secondly, the probability distribution output by the centroid classifier serves as a "soft label." The KL divergence loss is used to calculate the error between the first and second probability distributions. By minimizing this loss, the model learns the smoothed category relationships of the centroid classifier. As the model continuously trains toward minimizing this loss, feature learning becomes more accurate, and the weights assigned to features of different categories gradually adjust.

[0073] For example, when distinguishing damp-heat syndrome from cold-dampness syndrome in tongue image classification, the yellow and greasy tongue coating is a key characteristic for damp-heat syndrome. As training progresses, the model assigns more weight to identifying the yellow and greasy tongue coating, enabling more accurate judgment of damp-heat syndrome in new tongue image samples.

[0074] Understandably, this dual supervision not only preserves the clustering properties of similar samples (enhancing intra-class compactness), but also reflects the relative similarity between different categories (such as the distance relationship between "yellow fur" and "thick fur") through probability distribution, thereby helping the model learn more discriminative feature representations even with small sample sizes. Furthermore, this application avoids overfitting the model to a small number of samples through a smoothed probability distribution. Furthermore, the "soft labels" provided by the centroid classifier contain richer category relationship information than "hard labels," effectively alleviating optimization difficulties caused by data scarcity in small sample sizes, ultimately helping the model learn more accurate tongue image classification capabilities.

[0075] S160, testing the trained neural network model, and using the neural network model that meets the test conditions as the tongue image classification model.

[0076] In one embodiment, during the task testing phase of the model, the present application tests the small sample model by constructing a support set and query set for the test task. The present application fine-tunes the parameters of the small sample model based on the test task support set of the small sample model, wherein the support set provides a small number of labeled samples (such as N-way K-shot) for the model to quickly fine-tune parameters or calculate prototypes. In small sample learning, the small sample model has already learned the feature representation of tongue image data during the pre-training phase, such as the semantic region features of the tongue center and tongue edges. The small number of labeled samples in the support set can guide the model to adapt the features of the tongue image data to the tongue image classification task of the target dataset, so as to quickly identify similar features in the target support set.

[0077] When faced with a new small-sample task, the model is fine-tuned using a small number of labeled examples from the support set. This application uses fine-tuning on the support set to enable the model to quickly adapt to new tasks and categories, even with only a small number of labeled examples. This is because the model has already learned the feature representations of tongue image data during pre-training, and fine-tuning helps the model transfer this knowledge to the new task.

[0078] In an optional embodiment, the present application also calculates a prototype for each tongue image category based on the training task support set of the small sample model. This prototype can be the average of the features of the samples in each tongue image category. For example, after calculating the prototypes for different healthy or diseased tongue image categories, when encountering a new sample in the query set, the new sample can be classified by comparing the distance between the new sample and these prototypes. If the model can accurately determine the tongue image category of the new sample, it can be determined that the test conditions are met.

[0079] In another embodiment, the present application evaluates the generalization performance of a small-sample model on new samples based on the training task query set of the small-sample model. In the small-sample model test, the query set includes new samples that the model has not seen during training. The model's classification accuracy for the query set samples directly reflects the model's generalization performance on the target dataset. If the model can accurately classify the tongue images in the query set, it means that the model has learned the essential characteristics of the target dataset and can generalize the knowledge learned in the support set to new samples, thus meeting the test conditions.

[0080] It can be understood that this task-driven learning method simulates the data scarcity conditions in real scenarios, enabling the model to quickly adapt to new categories with only a very small number of samples (such as 1-5 images per category), and through lightweight optimization on the support set (such as fine-tuning or prototype matching) and performance verification on the query set, it ensures that the model can still effectively generalize under small sample conditions, thereby solving the practical problem of tongue image annotation costs.

[0081] The present application also provides a tongue image classification method, comprising: inputting a tongue image to be classified into the tongue image classification model of the above embodiment, and outputting a tongue image classification result.

[0082] As you can understand, this application inputs the tongue image to be classified into a trained tongue classification model. The tongue classification model then classifies the image based on the features of the tongue image and outputs the classification result. Through experiments and simulations, this application has achieved an average 1.7% improvement in tongue classification compared to similar methods (such as FEAT, LASTSHOT, and Protonet).

[0083] The present application also provides an electronic device. Exemplarily, the electronic device includes a processor and a memory, wherein the memory stores a computer program, and the processor runs the computer program to enable the electronic device to execute the above-mentioned tongue image classification model acquisition method and tongue image classification method.

[0084] The processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, including at least one of a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor, etc., and can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0085] The memory may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), and electrically erasable programmable read-only memory (EEPROM). The memory is used to store computer programs, and the processor can execute the computer programs accordingly after receiving execution instructions.

[0086] This application also provides a computer-readable storage medium for storing a computer program used in the electronic device. For example, the computer-readable storage medium may include, but is not limited to, a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, among other media capable of storing program code.

[0087] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and structure diagrams in the accompanying drawings show the possible architectures, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in an alternative implementation, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the structure diagram and / or flowchart, and the combination of boxes in the structure diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0088] In addition, the functional modules or units in the various embodiments of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0089] If a function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, or the part that contributes to the existing technology, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a smart phone, personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the method of this application.

[0090] The above is only a specific implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the protection scope of the present application.

Claims

1. A tongue image classification model acquisition method, characterized in that: include: The preprocessed unlabeled tongue images are input into the neural network model for self-supervised learning; masking the unlabeled tongue image according to the reconstruction mask loss of the unlabeled tongue image; Training the neural network model based on the masked unlabeled tongue image to obtain a pre-trained neural network model; Inputting the labeled tongue image into the pre-trained neural network model for supervised learning; Training a centroid classifier based on the labeled tongue image to guide supervised learning of the pre-trained neural network model based on the centroid classifier to obtain a trained neural network model; The trained neural network model is tested, and the neural network model that meets the test conditions is used as the tongue image classification model.

2. The tongue image classification model acquisition method according to claim 1, characterized in that: Also includes: performing image segmentation on the unlabeled tongue image to obtain a plurality of image blocks of each of the unlabeled tongue images; Random masking is performed on each of the image blocks in each of the unlabeled tongue images based on the same first mask ratio, so as to input the unlabeled tongue images after image segmentation and masking into the neural network model respectively.

3. The tongue image classification model acquisition method according to claim 2, characterized in that: The pre-processed unlabeled tongue image is input into the neural network model for self-supervised learning, including: Obtaining image features of the unlabeled tongue image after image segmentation through the encoder of the neural network model; splicing the predefined clustering features with the image features of the unlabeled tongue image; Inputting the spliced ​​features into the independent attention mechanism and the multi-layer neural network component in sequence to obtain the cluster center of the unlabeled tongue image; The image features of the unlabeled tongue image after masking are obtained through the encoder of the neural network model.

4. The method for obtaining a tongue image classification model according to claim 3, characterized in that: The adjusting the mask ratio of the unlabeled tongue image according to the reconstruction mask loss of the unlabeled tongue image includes: performing regional clustering on the image blocks in each of the unlabeled tongue images based on the cluster centers; Calculating the mask reconstruction loss of different regions of each of the unlabeled tongue images; The mask ratio of the image blocks in each region is determined according to the mask reconstruction loss of each region, and the image blocks in each region are re-masked based on the determined mask ratio.

5. The tongue image classification model acquisition method according to claim 1, characterized in that: The training of a centroid classifier based on the labeled tongue image comprises: Extracting a feature vector of the labeled tongue image through the pre-trained neural network model; Taking the mean of the feature vectors of the labeled tongue images of each tongue image category as the initial centroid vector of the centroid classifier; Calculating the similarity between the cluster center obtained by the self-supervised learning of the neural network model and the initial centroid vector; The cluster center is fused with the initial centroid vector according to the similarity and a preset weight to obtain a target centroid vector.

6. The method for obtaining a tongue image classification model according to claim 5, characterized in that: The supervised learning of the pre-trained neural network model guided by the centroid classifier includes: Calculating the distance between the target centroid vector of the centroid classifier and the feature vector of the labeled tongue image, so as to adjust the parameters of the neural network model through back propagation according to the distance; Determining a first probability distribution of the prediction of the labeled tongue image by the centroid classifier, and determining a second probability distribution of the prediction of the labeled tongue image by the neural network model; Supervised learning is performed on the neural network model by minimizing the divergence loss between the first probability distribution and the second probability distribution.

7. The method for obtaining a tongue image classification model according to claim 6, characterized in that: Determining a first probability distribution predicted by the centroid classifier for the labeled tongue image includes: Converting each of the distances into a prediction score of each of the labeled tongue images belonging to each of the tongue image categories using a probability distribution function to obtain a first probability distribution; The smoothness of the first probability distribution is controlled by a temperature parameter of the probability distribution function.

8. The method for obtaining a tongue image classification model according to claim 1, characterized in that: The testing of the trained neural network model includes: Fine-tuning the parameters of the neural network model based on the support set of the test task; The generalization performance of the neural network model on new samples is evaluated based on the query set of the test task.

9. A tongue image classification method, characterized in that: include: The tongue image to be classified is input into the tongue image classification model according to any one of claims 1 to 8, and the tongue image classification result is output.

10. An electronic device, characterized in that: The electronic device includes a processor and a memory, the memory stores a computer program, and the processor is used to execute the computer program to implement the tongue image classification model acquisition method according to any one of claims 1 to 8 and the tongue image classification method according to claim 9.

Images