Tongue picture multi-label multi-task automatic classification method and system

Abstract

The invention relates to the technical field of computer vision, in particular to a tongue picture multi-label multi-task automatic classification method and system.The method comprises the steps that tongue picture images are obtained to construct an original data set, and the tongue picture images in the original data set are labeled; performing preprocessing based on the obtained original data set, including introducing an illumination compensation factor to perform adaptive brightness normalization on the tongue picture image; the pre-processed tongue picture image is used as input to construct a classification model, multi-level feature extraction is carried out, multi-task feature decoupling is carried out by using the extracted multi-level features, multi-label classification and loss function optimization are carried out based on a feature decoupling result, and the trained classification model is optimized by using a Lion optimizer. According to the method, feature maps of different levels are fused in a cross-stage manner, multi-scale capture of tiny lesions in the tongue picture image is realized, and the detection capability of the model on features such as tongue color, tongue coating color, tongue shape and tongue coating quality is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a multi-label and multi-task automatic classification method and system for tongue images. Background Art

[0002] Tongue diagnosis in Traditional Chinese Medicine (TCM) is one of the traditional "Four Diagnoses". It comprehensively judges a person's health status by observing the shape, color, tongue coating and other characteristics of the tongue. However, traditional tongue diagnosis relies on the doctor's subjective judgment based on TCM theory and personal experience. The diagnosis results are easily affected by individual differences among doctors, resulting in low diagnostic repeatability and difficulty in achieving objectivity and standardization.

[0003] In addition, with the development of deep learning technology, convolutional neural networks based on attention mechanisms have performed well in single-label tasks such as tooth-marked tongue classification and cracked tongue recognition. For example, SE-ResNe and Competitive-SE enhance key features and suppress background noise through channel attention mechanisms, thereby improving classification accuracy and model interpretability. Multi-task learning methods jointly process tongue image segmentation and classification tasks, and use the correlation between labels to alleviate ambiguity. In addition, the Transformer model, with its global modeling capabilities, has demonstrated advantages in multi-label recognition of tongue images such as diabetes and coronary heart disease. Transfer learning alleviates the problem of insufficient data in small-sample tongue image classification by reusing pre-trained model knowledge. Regional convolutional neural networks have also achieved good results in tongue image feature detection and classification.

[0004] Although the above technologies have achieved certain results in the stage, they still have core defects. Most methods are only applicable to single syndrome classification and cannot handle the common clinical problem of multiple syndromes. The correlation between tongue features is easily ignored in single-label independent classification. In addition, the sample size of some syndromes in tongue diagnosis cases is small, and directly training multi-label models can easily lead to insufficient minority class recognition capabilities. Finally, when the multi-label task is decomposed into independent single-label binary classification tasks, the intrinsic correlation between tongue features is not fully utilized, which affects the overall performance of the model. At this stage, a multi-label and multi-task automatic classification method and system for tongue images is needed. Summary of the Invention

[0005] In order to solve the problems of single-label classification limitations, category imbalance and loss of feature association information in the existing technology, the present invention provides a multi-label and multi-task automatic classification method and system for tongue images. By combining a feature sharing network with Swin Transformer, improving the loss function and optimization algorithm, it systematically solves the problems of multi-label classification, category imbalance and loss of feature association.

[0006] In a first aspect, the present invention provides a multi-label, multi-task automatic classification method for tongue images, which adopts the following technical solutions:

[0007] A multi-label and multi-task automatic classification method for tongue images, comprising:

[0008] Obtain tongue images to construct an original data set, and annotate the tongue images in the original data set;

[0009] Preprocessing is performed based on the acquired raw data set, including introducing an illumination compensation factor to perform adaptive brightness normalization on the tongue image;

[0010] The preprocessed tongue images are used as input to build a classification model and perform multi-level feature extraction, including introducing a feature pyramid network based on the ResNeXt50-CBAM network for multi-level feature extraction.

[0011] Utilize the extracted multi-level features to perform multi-task feature decoupling, including task-aware window division of the extracted features and the introduction of a cross-task interaction mechanism;

[0012] Perform multi-label classification and loss function optimization based on feature decoupling results, including feature classification using a multi-layer fully connected network;

[0013] The Lion optimizer is used to optimize the trained classification model, and the optimized classification model is used to output the final tongue image classification results.

[0014] Furthermore, the introduction of the illumination compensation factor to perform adaptive brightness normalization on the tongue image includes obtaining the actual color temperature value of the image based on the original tongue image data set, calculating the illumination compensation factor using the actual color temperature value, and introducing the illumination compensation factor to perform adaptive brightness normalization processing on the tongue image. The adaptive brightness normalization processing formula is:

[0015]

[0016] Among them, X represents the original pixel value, X max and X min They represent the maximum and minimum values ​​of the original data respectively, k represents the illumination compensation factor, and b and a represent different normalized target ranges respectively.

[0017] Furthermore, the method introduces a feature pyramid network based on the ResNeXt50-CBAM network for multi-level feature extraction, including introducing a feature pyramid network for multi-level feature extraction to form a three-level feature pyramid including low-level texture, mid-level structure and high-level semantics. The ResNeXt50-CBAM network has a built-in feature sharing layer, which pads the input image with zeros to adapt to the model size, and completes the first stage of low-level feature extraction through the feature sharing layer. The low-level feature extraction includes convolution, batch normalization, ReLU activation and maximum pooling operations.

[0018] Furthermore, the ResNeXt50-CBAM network also includes multiple Res CBAM Blocks for multi-stage deep feature extraction. The Res CBAM Block calculates channel attention and spatial attention through the CBAM attention mechanism to enhance key features. Then, the key features are reduced in dimension using a 1×1 convolution layer, and grouped by a 3×3 group convolution. The grouped features are increased in dimension using a 1×1 convolution layer, and the grouped features are residually connected with the original input. Finally, a nonlinear transformation is performed through the ReLu activation function. The key feature calculation formula is:

[0019]

[0020] Among them, F represents the input feature map, M c Expressed as channel attention weight, M s Expressed as spatial attention weight, Represented as element-wise multiplication.

[0021] Furthermore, the method introduces a feature pyramid network based on the ResNeXt50-CBAM network for multi-level feature extraction, and also includes cross-stage fusing feature maps of multiple stages in the ResNeXt50-CBAM network through the feature pyramid network, realizing multi-scale feature fusion through upsampling and lateral connection, outputting bottom-level feature maps and high-level feature maps, and introducing a learnable grouping number parameter in the 3×3 grouped convolution of the ResCBAM Block, wherein the learnable grouping number parameter is dynamically adjusted according to the entropy value of the feature map.

[0022] Furthermore, the multi-task feature decoupling is performed using the extracted multi-level features, including task-aware window division of the extracted features. In the windowed multi-head self-attention W-MSA stage, multi-label one-hot encoding is introduced as task embedding, and the task embedding is fused with the feature map. In the task-aware window division, the dimension of the task embedding is consistent with the number of feature map channels. The multi-label one-hot encoding is mapped to a task embedding vector through linear transformation. The calculation formula of the task embedding vector is:

[0023] Task-mbedding=Linear(y one-hot ),

[0024] Among them, y one-hot It is represented as a multi-label one-hot encoding vector, and Linear is represented as a linear transformation layer.

[0025] Furthermore, the multi-task feature decoupling using the extracted multi-level features also includes introducing a cross-task interaction mechanism. In the sliding window multi-head self-attention SW-MSA stage, the attention weight is calculated by the task gating unit to dynamically suppress the interference of irrelevant task features. The calculation formula of the task gating unit is:

[0026] G=σ(W g Concat(F msa ,Task E mbedding))⊙F msa ,

[0027] Among them, F msa Represented as multi-head self-attention output feature, W g It is represented as a gated weight matrix, σ represents the Sigmoid activation function, Concat represents the concatenation operation, and ⊙ represents the element-by-element multiplication. The gating weights are used to suppress features that are not related to the current task.

[0028] Furthermore, the feature classification using a multi-layer fully connected network includes constructing a fully connected layer consisting of a shared layer and a task-specific layer. The shared layer is a feature transformation layer shared by all label groups and is used to extract common semantic features across tasks. The task-specific layer is split into independent sub-layers according to the label group, and each sub-layer corresponds to a classification task of a group of related labels. The calculation formula of the fully connected layer is:

[0029] F fc3 =Sigmoid(FC3(ReLU(FC2(ReLU(FC1(F decoupled )))))))),

[0030] Among them, F decoupled It represents the task-specific features after feature decoupling, FC1 and FC2 represent shared layers with different dimensions, FC3 represents the task-specific layer, ReLU represents the prior activation function, and Sigmoid represents the Sigmoid function.

[0031] Furthermore, the multi-label classification and loss function optimization based on the feature decoupling result also includes optimizing the classification result using a combined loss function. The loss function is the sum of the cross entropy loss and the focus loss. The label density factor is introduced into the focus loss according to the inverse document frequency of the label. The focus loss calculation formula is:

[0032]

[0033] Where N is the total number of samples in the training batch, C is the total number of labels, i is the sample index, c is the label index, and y is the ic Represented as the true value of the cth label in the i-th sample, It is expressed as the predicted probability of the model for the cth label in the i-th sample, γ is the focusing factor, and δ c Expressed as a label density factor.

[0034] In the second aspect, a tongue image multi-label multi-task automatic classification system includes:

[0035] The data acquisition module is configured to: acquire tongue images to construct an original data set, and annotate the tongue images in the original data set;

[0036] The preprocessing module is configured to: perform preprocessing based on the acquired original data set, including introducing an illumination compensation factor to perform adaptive brightness normalization on the tongue image;

[0037] The model module is configured to: use the preprocessed tongue image as input to build a classification model and perform multi-level feature extraction, including introducing a feature pyramid network based on the ResNeXt50-CBAM network for multi-level feature extraction;

[0038] The feature module is configured to: utilize the extracted multi-level features to perform multi-task feature decoupling, including task-aware window division of the extracted features and the introduction of a cross-task interaction mechanism;

[0039] The training module is configured to perform multi-label classification and loss function optimization based on the feature decoupling results, including feature classification using a multi-layer fully connected network;

[0040] The output module is configured to optimize the trained classification model using the Lion optimizer and output the final tongue image classification result using the optimized classification model.

[0041] In summary, the present invention has the following beneficial technical effects:

[0042] 1. The present invention introduces an illumination compensation mechanism to perform adaptive brightness normalization processing based on the color temperature differences in different shooting environments, effectively eliminating the color cast problem caused by uneven illumination, improving the robustness of the model to complex lighting conditions, and ensuring the consistency and reliability of the pre-processed image.

[0043] 2. The present invention introduces a feature pyramid network (FPN) based on the ResNeXt50-CBAM network, and fuses feature maps of different levels across stages to form a multi-level feature pyramid that includes low-level textures (such as tongue coating particles), mid-level structures (such as tooth mark contours), and high-level semantics (such as syndrome type associations). It can achieve multi-scale capture of subtle lesions in tongue images and significantly improve the model's ability to detect features such as tongue color, coating color, tongue shape, and coating quality.

[0044] 3. The present invention dynamically enhances the feature responses of key areas such as tooth marks and cracks and suppresses irrelevant background interference such as lips through the channel attention and spatial attention mechanisms in the Res CBAM Block. At the same time, a learnable grouping number parameter is introduced into the group convolution to adaptively adjust the grouping number according to the feature complexity, thereby improving the flexibility and efficiency of feature extraction and achieving refined processing of different tongue image complexities.

[0045] 4. The present invention introduces task label embedding in the W-MSA stage of the Swin Transformer Block, fuses multi-label one-hot encoding with feature maps, guides attention to focus on specific task-related areas, improves the extraction accuracy of target features, and reduces cross-task feature interference.

[0046] 5. The present invention uses the task gating unit in the SW-MSA stage to dynamically filter features irrelevant to the current task based on attention weight, thereby achieving effective decoupling of multi-task features and improving the model's ability to independently classify different syndromes.

[0047] 6. The fully connected layer of the present invention adopts a "shared layer + task-specific layer" structure. The shared layer extracts common semantics across labels, and the task-specific layer is split into independent sub-layers according to label groups to avoid semantic conflicts across label groups and improve the accuracy and confidence of classification decisions.

[0048] 7. The present invention combines cross-entropy loss and focal loss, and introduces a label density factor to dynamically adjust the loss weights of easy-to-classify samples and difficult-to-classify samples, focusing on enhancing the learning of low-frequency co-occurrence labels and minority class samples, significantly improving the impact of category imbalance on training, and improving the model's recognition ability for rare syndromes.

[0049] 8. This invention adopts Lion optimizer to replace the traditional Adam optimizer, and reduces computational overhead and improves training efficiency based on the gradient sign update method. Through cross-layer learning rate adaptive adjustment and periodic weight averaging (PWA) strategy, it enhances the generalization stability of the model in small sample scenarios, reduces the risk of overfitting, shortens training time, and improves model convergence speed and final performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a schematic diagram of the overall process of a multi-label and multi-task automatic classification method for tongue images according to an embodiment of the present invention.

[0051] Figure 2 This is a schematic diagram of the calculation flow of each module of a multi-label and multi-task automatic classification method for tongue images according to an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The present invention will be further described in detail below with reference to the accompanying drawings.

[0053] Example 1

[0054] Reference Figure 1 The present embodiment provides a multi-label multi-task automatic classification method for tongue images, including:

[0055] Obtain tongue images to construct an original data set, and annotate the tongue images in the original data set;

[0056] Preprocessing is performed based on the acquired raw data set, including introducing an illumination compensation factor to perform adaptive brightness normalization on the tongue image;

[0057] The preprocessed tongue images are used as input to build a classification model and perform multi-level feature extraction, including introducing a feature pyramid network based on the ResNeXt50-CBAM network for multi-level feature extraction.

[0058] Utilize the extracted multi-level features to perform multi-task feature decoupling, including task-aware window division of the extracted features and the introduction of a cross-task interaction mechanism;

[0059] Perform multi-label classification and loss function optimization based on feature decoupling results, including feature classification using a multi-layer fully connected network;

[0060] The Lion optimizer is used to optimize the trained classification model, and the optimized classification model is used to output the final tongue image classification results.

[0061] Specifically, a multi-label multi-task automatic classification method for tongue images includes the following steps:

[0062] like Figure 1 As shown, S1, obtain tongue images to construct an original data set, and annotate the tongue images in the original data set;

[0063] The original dataset is composed of tongue images obtained from professional hospital equipment (such as a tongue microscope with a resolution ≥300DPI) and non-professional equipment (mobile phones / cameras that support autofocus macro mode). To address the common problem of uneven lighting in clinical tongue image collection, the shooting environment parameters (such as color temperature and brightness value) are recorded synchronously during the data collection process to provide metadata support for subsequent preprocessing.

[0064] Finally, a multi-label annotation standard was established, and each tongue image sample was associated with 4 tongue image labels (such as "crimson tongue + yellow fur + cracked tongue + thick fur" and "pale red tongue + white fur + thin tongue + thin fur"), covering clinical multi-syndrome scenarios and solving the problem of disconnection between traditional single-label annotation and actual diagnostic needs.

[0065] like Figure 2As shown, S2, preprocessing is performed based on the acquired original data set, including introducing an illumination compensation factor to perform adaptive brightness normalization on the tongue image;

[0066] After the data acquisition phase (S1) is completed, for each tongue image in the original dataset, the illumination parameters are obtained and the compensation factor is calculated by the following method: the actual color temperature value T at the time of shooting is extracted using an acquisition device (such as the spectral sensor of a professional tongue imager or the metadata interface of a smartphone camera). actual , in Kelvin (K). If the device does not directly record the color temperature, it is estimated through the image white balance algorithm. The grayscale mean area in the tongue area is selected, assuming it is neutral gray, and the light source color temperature is reversed. Then the standard color temperature T is defined. standard =6500K (simulated daylight environment), calculate the compensation factor:

[0067]

[0068] Among them, T actual Expressed as actual color temperature, T standard Expressed as a standard color temperature. When the actual color temperature is lower than the standard color temperature (such as in a warm light environment), k>1, and the overall image brightness is increased. When the actual color temperature is higher than the standard color temperature (such as in a cold light environment), when k<1, the image brightness is reduced to correct the color cast.

[0069] Normalize the RGB channels of the original image separately to eliminate the illumination differences between different channels. For the pixel value X of each channel, calculate its extreme value within the channel and complete the normalization using the following formula:

[0070]

[0071] Among them, X represents the original pixel value, X max and X min Represent the maximum and minimum values ​​of the original data respectively, k represents the illumination compensation factor, b and a represent different normalized target ranges, if the output is an 8-bit grayscale image, take a = 0, b = 255, if the output is a floating-point feature map, take a = 0, b = 1, if X max =X min , that is, if the image is completely black or completely white, normalization is skipped and the original pixel values ​​are directly retained to avoid the abnormal situation of division by zero.

[0072] The processing effect is evaluated by calculating the image brightness variance. The brightness variance of the normalized image is required to be effectively reduced compared to the original image. For locally overexposed or underexposed areas, histogram equalization is used as an auxiliary processing to further improve the overall brightness uniformity and provide stable input for subsequent feature extraction.

[0073] S3, the pretreated tongue image is taken as input to construct a classification model, and multi-level feature extraction is performed, including introducing a feature pyramid network based on a ResNeXt50-CBAM network for multi-level feature extraction;

[0074] The classification model constructed in this embodiment takes ResNeXt50-CBAM as a basic backbone network, embeds a feature pyramid network (FPN) to realize multi-level feature extraction, forms a three-level feature processing link of “low-level texture capture, middle-level structure analysis, and high-level semantic abstraction”, and first adapts the model input dimension through zero padding operation for the pretreated tongue image (uniform size of 224x224 pixels). The edge pixel value is 0 through zero padding to ensure the spatial dimension consistency of the subsequent convolution operation. First, low-level feature extraction (Stage 1) is performed, including performing convolution operation on the pretreated tongue image, using a 7x7 convolution kernel (stride 2, padding 3) to preliminarily extract features from the input image, outputting a 64-channel feature map, capturing basic visual elements such as tongue edge and tongue texture, then performing batch normalization and activation, executing batch normalization (BN) on the convolution output to speed up convergence, then introducing nonlinearity through the ReLU activation function to enhance feature expression capability, and finally performing maximum pooling operation, using a 3x3 maximum pooling layer (stride 2) to reduce the feature map resolution to 112x112 pixels, reducing the amount of calculation while retaining the main feature outline, completing the first-stage low-level feature extraction.

[0075] Then, deep-level feature extraction is performed. The ResNeXt50-CBAM network includes four Res CBAM Blocks, which perform multi-stage deep-level feature extraction using one Res CBAM Block, corresponding to stages two to five (Stage 2-5). Each Res CBAM Block contains channel attention, spatial attention, grouped convolution, and residual connection. Taking Stage 2 as an example:

[0076] The input feature map F is subjected to global average pooling and global maximum pooling through the CBAM attention mechanism to obtain F avg and F max , and a channel weight matrix is generated through a two-layer fully connected network, with the formula being:

[0077] M c (F)=σ(FC2(ReLU(FC1(F avg ))+FC2(ReLU(FC1(F max )))),

[0078] wherein FC1 represents the first fully connected layer, FC2 represents the second fully connected layer, ReLU represents the activation function, F avg is the global average pooling feature, and Fmax is represented as global max pooling feature, σ is represented as Sigmoid activation function, M c (F) is a channel attention matrix, the dimension is C, each element corresponds to the weight value of a channel, the larger the value, the more important the channel in feature expression, the channel weight is multiplied by F element by element, and the key features such as tooth marks and cracks are enhanced.

[0079] Then the channel attention output feature map is subjected to mean and maximum value pooling, and the spatial feature descriptor is obtained by splicing. A spatial weight matrix is generated by 7x7 convolution, and the formula is:

[0080] M s (F') = σ(Conv7x7([F' avg ; F' max ])),

[0081] Wherein, F' represents the output feature map of the channel attention mechanism, F' is represented as the channel dimension of F', and F' is represented as the spatial dimension of F'. avg F' is represented as the channel dimension of F', and F' is represented as the spatial dimension of F'. max F' is represented as the channel dimension of F', and F' is represented as the spatial dimension of F'.

[0082] Then, 1x1 convolution is used for dimension reduction operation, the channel number is reduced from 256 to 4, and then 3x3 convolution (32 groups) is used. Each group independently processes the features of 4 channels, and then 1x1 convolution is used for dimension expansion, the channel number is expanded from 4 to 256, the grouped convolution output is added to the original input (residual connection), the gradient vanishing of deep network is avoided, and finally the nonlinear transformation is completed through ReLU activation.

[0083] Finally, the feature pyramid network is used to extract the feature maps of Stage 1 (112×112 pixels), Stage 2 (56×56 pixels), Stage 3 (28×28 pixels), Stage 4 (14×14 pixels), and Stage 5 (7×7 pixels) in the ResNeXt50-CBAM network, which correspond to low-level to high-level features respectively. The high-level low-resolution feature map (such as Stage 5) is upsampled to the resolution of the corresponding Stage through bilinear interpolation, and is added element-by-element with the low-level feature map of the Stage to fuse multi-scale information. For example, after the Stage 5 feature map is upsampled to 14×14 pixels, it is fused with the Stage 4 feature map to generate a feature map that contains both semantic information and structural details. The feature pyramid outputs the low-level texture layer (Stage 1-2 fusion) to retain details such as tongue coating particles and edge jaggedness, and the middle-level structure layer (Stage 3-4 fusion) highlights structural features such as tooth mark contours and crack directions, and the high-level semantic layer (Stage 5. Independent) Abstraction of the syndrome-related features. In the 3×3 group convolution of the Res CBAM Block, a learnable parameter g is introduced. The initial value of the learnable parameter is 32 and is dynamically adjusted according to the entropy value E of the feature map:

[0084] E=-∑ i,j p i,j log p i,j ,

[0085] Among them, p i,j is the normalized probability distribution of pixel values, reflecting the complexity of features, and the threshold E is set by the validation set. thr , if E>E thr , the number of groups g is increased to 64 to capture complex textures. If E≤E thr , the number of groups is reduced to 16, reducing the computational redundancy of simple features.

[0086] S4. Decoupling multi-task features using the extracted multi-level features, including task-aware window division of the extracted features and the introduction of a cross-task interaction mechanism;

[0087] Multi-label unique hot encoding is the encoding representation of the original labeled data. In the S1 data annotation stage, each tongue image sample is annotated with multiple related labels. The multi-label unique hot encoding y one-hot It is a binary vector of length C, where C is the total number of labels. Each element corresponds to the presence or absence of a label, 1 for presence and 0 for absence. For example, if a sample belongs to both "tooth-marked tongue" and "Qi deficiency syndrome", the corresponding position is set to 1, and the rest are 0. The one-hot encoding is mapped to the task embedding vector through a linear layer. The formula is:

[0088] Task_mbedding=Linear(yone-hot )=W t ·y one-hot +b t ,

[0089] Among them, W t is the weight matrix, b t is a bias vector. Through this transformation, the discrete label information is converted into a continuous feature vector with the same dimension as the number of feature map channels. The task embedding vector is concatenated with the feature map in the channel dimension to expand the task embedding vector to the same spatial size as the feature map, so that each spatial position carries the task label information. The fused feature map is divided into multiple non-overlapping windows through windowed multi-head self-attention (W-MSA), and multi-head self-attention calculation is performed in each window. The formula is:

[0090]

[0091] Among them, Q represents the query vector, K represents the key vector generated by linear projection, V represents the value vector generated by linear projection, and Softmax represents the Softmax activation function.

[0092] After completing W-MSA, a cross-task interaction mechanism (SW-MSA stage) is performed, and the window position is moved through the sliding window (Shifted Windows) mechanism to make adjacent windows overlap. The formula is:

[0093] SW-MSA(F w-msa )=WindowShift(F w-msa ,shift_size=2),

[0094] Among them, F w-msa It represents the feature map output by W-MSA, where local feature interaction is completed within non-overlapping windows. WindowShift represents the sliding window mechanism, and shift_size represents a step size of 2.

[0095] Overlapping windows allow feature information of different tasks to flow between windows, promoting global semantic interaction and solving the local limitations of traditional self-attention mechanisms.

[0096] The features output by SW-MSA are concatenated with the task embedding vector and input into the gating unit:

[0097] G=σ(W g Concat(F msa ,Task E mbedding))⊙F msa ,

[0098] Among them, F msaRepresented as multi-head self-attention output feature, W g It is represented as a gated weight matrix, σ represents the Sigmoid activation function, Concat represents the concatenation operation, and ⊙ represents the element-by-element multiplication. The gating weights are used to suppress features that are not related to the current task.

[0099] Finally, the gating weights are multiplied element-by-element with the feature map to dynamically suppress irrelevant task features:

[0100] F decoupled =G⊙F msa ,

[0101] When processing the "tongue color classification" task, the gating unit will suppress the feature areas related to "tooth-marked tongue" and retain color features such as red tongue and white tongue coating, thereby achieving effective filtering of cross-task interference.

[0102] S5. Perform multi-label classification and loss function optimization based on feature decoupling results, including using a multi-layer fully connected network for feature classification;

[0103] Input feature F decoupled It is a task-specific feature after task perception window division and cross-task interaction suppression. After step S4, the feature has eliminated the interference of irrelevant tasks and focused on the key area of ​​the current task. Then, the first shared layer is used to project the 256-dimensional feature into a 1024-dimensional high-dimensional space through the weight matrix and bias. The formula is:

[0104] FC1(F decoupled )=W1·F decoupled +b1,

[0105] Among them, W1 represents the weight matrix and b1 represents the bias. This operation expands the expression range of the features, allowing the model to capture more complex feature combinations. ReLU activation is then performed to filter negative features, enhance the nonlinear expression ability of the features, and avoid the model falling into the limitations of linear classification.

[0106] After further learning the deep features of the features after dimensionality increase, the weight matrix and bias of the second shared layer are used to compress the features after dimensionality increase. The formula is:

[0107] FC2(x)=W2·x+b2,

[0108] Among them, W2 represents the weight matrix of the second shared layer, b2 represents the bias of the second shared layer, and x represents the output of FC1, which is used to refine the common semantics across tasks. For example, features such as "tooth-marked tongue" and "fat tongue" can be abstracted into the high-level semantics of "tongue abnormality" to reduce redundant information and improve computational efficiency. The ReLU activation function is applied again to further strengthen the nonlinear correlation between features.

[0109] Finally, the task-specific layer is used to perform label group-specific classification. According to the theory of tongue diagnosis in traditional Chinese medicine, the labels are divided into groups such as "tongue quality" (such as color and shape) and "tongue coating" (such as thickness and color). Each group of labels corresponds to a specific diagnostic dimension, and each label group corresponds to an independent fully connected sublayer. The task-specific layer is split into independent sublayers according to the label group. Each sublayer corresponds to a classification task of a group of related labels. The 512-dimensional features output by the shared layer are linearly transformed. The formula is:

[0110] FC3 g (x1)=W 3,g x1+b 3,g ,

[0111] Among them, W 3,g represents the weight matrix of the task-specific layer, b 3,g Represents the bias of the task-specific layer, x1 represents the output of FC2, and realizes the mapping from general semantics to specific labels. The output value is compressed to the [0,1] interval through the Sigmoid activation function. In summary, the overall formula of the fully connected layer is:

[0112] F fc3 =Sigmoid(FC3(ReLU(FC2(ReLU(FC1(F decoupled )))))))),

[0113] Among them, F decoupled It represents the task-specific features after feature decoupling, FC1 and FC2 represent shared layers with different dimensions, FC3 represents the task-specific layer, ReLU represents the prior activation function, and Sigmoid represents the Sigmoid function.

[0114] As the basic loss for multi-label classification, cross-entropy loss forces the model to predict the probability of each label close to the true label. This embodiment adopts a combined loss function to comprehensively consider the classification accuracy and the imbalance of category samples through the synergy of cross-entropy loss and focal loss. Cross-entropy loss is used to measure the difference between the category prediction probability distribution output by the model and the true label, ensuring that the overall prediction result of the model is close to the true label. Focal loss dynamically reduces the impact of easy-to-classify samples on the total loss by introducing a focusing factor γ, and increases the focus on difficult-to-classify samples, thereby effectively improving the learning effect of the model on minority categories. During training, the optimal value of the focusing factor γ is determined by grid search.

[0115] The cross entropy loss function formula is:

[0116]

[0117] Among them, y i is the true category (one-hot encoding), For the model predicted class probability, then through the inverse document frequency (IDF) label density factor, the count of low-frequency labels c Small, so the value of IDF c = log(N / (count c +1)) is large, resulting in an increase in the label density factor, where count c Indicates the number of times the label c appears in the training set, the loss term for low-frequency labels is multiplied by a higher weight through the label density factor, and the training resource allocation for different labels is balanced, and the focal loss calculation formula is:

[0118]

[0119] Where N represents the total number of samples in the training batch, C represents the total number of labels, i represents the sample index, c represents the label index, y ic Represents the true value of the cth label in the ith sample, Represents the predicted probability of the model for the cth label in the ith sample, and γ represents the focusing factor, and δ c Represents the label density factor.

[0120] Therefore, the combined loss function formula used in the embodiment is: L = L CE + L FL , through L FL Therefore, L CE compensates for the neglect of difficult samples and minority classes, from feature extraction of ResNeXt50-CBAM to feature decoupling of Swin Transformer, to classification of the full connection layer, the entire model is trained end-to-end through back propagation, for example, when the loss function detects that the "tooth mark tongue" classification is wrong, the gradient will be transmitted back to the convolution kernel parameters of ResNeXt50-CBAM layer by layer, adjusting the response strength of the tongue body fat feature.

[0121] S6, using Lion optimizer to optimize the trained classification model, and using the optimized classification model to output the final tongue image classification result.

[0122] Finally, the Lion optimizer is used to optimize the training process, which is a gradient sign-based optimization algorithm that directly adjusts the model parameters using the sign (not the amplitude) of the gradient, and the formula is:

[0123]

[0124] Where θ t is the current parameter, and α is the learning rate, is the gradient of the loss function, sign is the function that maps the gradient value to {-1, 0, 1}, reduces the oscillation of parameter updates, improves convergence stability, and jointly optimizes the multi-layer fully connected network (including shared layers and task-specific layers) and the front-end feature extraction network (ResNeXt50-CBAM, Swin Transformer) trained in the S5 stage.

[0125] Compared to the traditional Adam optimizer, the Lion optimizer uses a gradient-sign-based update method, which reduces computational overhead while ensuring accurate update directions. Furthermore, the Lion optimizer is more stable during weight adjustment, effectively alleviating oscillation issues during training and improving the model's convergence speed and ultimate generalization capabilities. This makes it particularly suitable for multi-label classification of tongue images, such as those in this embodiment, which involve task interference and complex features. Through this training strategy, the algorithm model can fully learn the explicit and implicit features in tongue images, accurately distinguishing different tongue image categories, and providing doctors with a more reliable and objective basis for diagnosis.

[0126] Example 2

[0127] This embodiment differs from Example 1 in that it provides a tongue image multi-label multi-task automatic classification system, including:

[0128] The data acquisition module is configured to: acquire tongue images to construct an original data set, and annotate the tongue images in the original data set;

[0129] The preprocessing module is configured to: perform preprocessing based on the acquired original data set, including introducing an illumination compensation factor to perform adaptive brightness normalization on the tongue image;

[0130] The model module is configured to: use the preprocessed tongue image as input to build a classification model and perform multi-level feature extraction, including introducing a feature pyramid network based on the ResNeXt50-CBAM network for multi-level feature extraction;

[0131] The feature module is configured to: utilize the extracted multi-level features to perform multi-task feature decoupling, including task-aware window division of the extracted features and the introduction of a cross-task interaction mechanism;

[0132] The training module is configured to perform multi-label classification and loss function optimization based on the feature decoupling results, including feature classification using a multi-layer fully connected network;

[0133] The output module is configured to optimize the trained classification model using the Lion optimizer and output the final tongue image classification result using the optimized classification model.

[0134] The above are preferred embodiments of the present application, and do not limit the protection scope of the present application, so: any equivalent changes made according to the structure, shape, principle of the present application should be covered within the protection scope of the present application.

Claims

1. A multi-label multi-task automatic classification method for tongue images, characterized in that: include: Obtain tongue images to construct an original data set, and annotate the tongue images in the original data set; Preprocessing is performed based on the acquired raw data set, including introducing an illumination compensation factor to perform adaptive brightness normalization on the tongue image; The preprocessed tongue images are used as input to build a classification model and perform multi-level feature extraction, including introducing a feature pyramid network based on the ResNeXt50-CBAM network for multi-level feature extraction. Utilize the extracted multi-level features to perform multi-task feature decoupling, including task-aware window division of the extracted features and the introduction of a cross-task interaction mechanism; Perform multi-label classification and loss function optimization based on feature decoupling results, including feature classification using a multi-layer fully connected network; The Lion optimizer is used to optimize the trained classification model, and the optimized classification model is used to output the final tongue image classification results.

2. The multi-label multi-task automatic classification method for tongue images according to claim 1 is characterized in that: The introduction of the illumination compensation factor to perform adaptive brightness normalization on the tongue image includes obtaining the actual color temperature value of the image based on the original tongue image data set, calculating the illumination compensation factor using the actual color temperature value, and introducing the illumination compensation factor to perform adaptive brightness normalization processing on the tongue image. The adaptive brightness normalization processing formula is: Among them, X represents the original pixel value, X max and X min They represent the maximum and minimum values ​​of the original data respectively, k represents the illumination compensation factor, and b and a represent different normalized target ranges respectively.

3. The multi-label multi-task automatic classification method for tongue images according to claim 1 is characterized in that: The method introduces a feature pyramid network based on the ResNeXt50-CBAM network for multi-level feature extraction, including introducing a feature pyramid network for multi-level feature extraction to form a three-level feature pyramid including low-level texture, mid-level structure and high-level semantics. The ResNeXt50-CBAM network has a built-in feature sharing layer, which pads the input image with zeros to adapt to the model size, and completes the first stage of low-level feature extraction through the feature sharing layer. The low-level feature extraction includes convolution, batch normalization, ReLU activation and maximum pooling operations.

4. The multi-label multi-task automatic classification method for tongue images according to claim 1, characterized in that: The ResNeXt50-CBAM network also includes multiple Res CBAM Blocks for multi-stage deep feature extraction. The ResCBAM Block calculates channel attention and spatial attention through the CBAM attention mechanism to enhance key features. Then, the key features are reduced in dimension using a 1×1 convolution layer, and grouped by a 3×3 group convolution. The grouped features are then increased in dimension using a 1×1 convolution layer. The grouped features are residually connected with the original input, and finally a nonlinear transformation is performed using the ReLu activation function. The key feature calculation formula is: Among them, F represents the input feature map, M c Expressed as channel attention weight, M s Expressed as spatial attention weight, Represented as element-wise multiplication.

5. The multi-label multi-task automatic classification method for tongue images according to claim 1 is characterized in that: The method introduces a feature pyramid network based on the ResNeXt50-CBAM network to perform multi-level feature extraction, and also includes cross-stage fusing feature maps of multiple stages in the ResNeXt50-CBAM network through the feature pyramid network, realizing multi-scale feature fusion through upsampling and lateral connection, outputting low-level feature maps and high-level feature maps, and introducing a learnable grouping number parameter in the 3×3 grouped convolution of the Res CBAM Block, wherein the learnable grouping number parameter is dynamically adjusted according to the entropy value of the feature map.

6. The multi-label multi-task automatic classification method for tongue images according to claim 1, characterized in that: The method utilizes the extracted multi-level features to perform multi-task feature decoupling, including task-aware window division of the extracted features. In the windowed multi-head self-attention (W-MSA) stage, multi-label one-hot encoding is introduced as task embedding, and the task embedding is fused with the feature map. In the task-aware window division, the dimension of the task embedding is consistent with the number of feature map channels. The multi-label one-hot encoding is mapped to a task embedding vector through linear transformation. The calculation formula of the task embedding vector is: Task_mbedding=Linear(y one-hot ), Among them, y one-hot It is represented as a multi-label one-hot encoding vector, and Linear is represented as a linear transformation layer.

7. The multi-label multi-task automatic classification method for tongue images according to claim 6, characterized in that: The multi-task feature decoupling using the extracted multi-level features also includes introducing a cross-task interaction mechanism. In the sliding window multi-head self-attention SW-MSA stage, the attention weight is calculated by the task gating unit to dynamically suppress the interference of irrelevant task features. The calculation formula of the task gating unit is: G=σ(W g ·Concat(F msa ,Task E mbedding))⊙F msa , Among them, F msa Represented as multi-head self-attention output feature, W g It is represented as a gated weight matrix, σ represents the Sigmoid activation function, Concat represents the concatenation operation, and ⊙ represents the element-by-element multiplication. The gating weights are used to suppress features that are not related to the current task.

8. The multi-label multi-task automatic classification method for tongue images according to claim 1, characterized in that: The feature classification is performed using a multi-layer fully connected network, including constructing a fully connected layer consisting of a shared layer and a task-specific layer. The shared layer is a feature transformation layer shared by all label groups and is used to extract common semantic features across tasks. The task-specific layer is split into independent sub-layers according to the label group, and each sub-layer corresponds to a classification task of a group of related labels. The calculation formula of the fully connected layer is: F fc3 =Sigmoid(FC3(ReLU(FC2(ReLU(FC1(F decoupled )))))))), Among them, F decoupled It represents the task-specific features after feature decoupling, FC1 and FC2 represent shared layers with different dimensions, FC3 represents the task-specific layer, ReLU represents the prior activation function, and Sigmoid represents the Sigmoid function.

9. The multi-label multi-task automatic classification method for tongue images according to claim 1, characterized in that: The multi-label classification and loss function optimization based on the feature decoupling result also includes optimizing the classification result using a combined loss function, wherein the loss function is the sum of the cross entropy loss and the focus loss, and a label density factor is introduced into the focus loss according to the inverse document frequency of the label. The focus loss calculation formula is: Where N is the total number of samples in the training batch, C is the total number of labels, i is the sample index, c is the label index, and y is the ic Represented as the true value of the cth label in the i-th sample, It is expressed as the predicted probability of the model for the cth label in the i-th sample, γ is the focusing factor, and δ c Expressed as a label density factor.

10. A tongue image multi-label multi-task automatic classification system, executed according to the method of claim 1, characterized in that: include: The data acquisition module is configured to: acquire tongue images to construct an original data set, and annotate the tongue images in the original data set; The preprocessing module is configured to: perform preprocessing based on the acquired original data set, including introducing an illumination compensation factor to perform adaptive brightness normalization on the tongue image; The model module is configured to: use the preprocessed tongue image as input to build a classification model and perform multi-level feature extraction, including introducing a feature pyramid network based on the ResNeXt50-CBAM network for multi-level feature extraction; The feature module is configured to: utilize the extracted multi-level features to perform multi-task feature decoupling, including task-aware window division of the extracted features and the introduction of a cross-task interaction mechanism; The training module is configured to perform multi-label classification and loss function optimization based on the feature decoupling results, including feature classification using a multi-layer fully connected network; The output module is configured to optimize the trained classification model using the Lion optimizer and output the final tongue image classification result using the optimized classification model.

Images