Stroke patient tongue picture automatic identification system and method based on multi-task learning

Abstract

The invention belongs to the technical field of cerebral apoplexy diagnosis, and particularly relates to a cerebral apoplexy patient tongue picture automatic identification system and method based on multi-task learning, and the method comprises the steps: carrying out the multi-label classification of a tongue picture, dividing the tongue picture into a training set and a test set, and carrying out the data enhancement; constructing an LGAN model by using the tongue picture features, extracting picture features, capturing correlation between the features, and generating a classification result of each label; and optimizing model parameters by calculating a loss value between a prediction score vector and a real label, and evaluating a model performance index. According to the method, the problems that in the prior art, standardization and objectification of tongue picture recognition of a stroke patient are insufficient, diagnosis is inconsistent and accuracy is poor are solved, and by combining multi-task learning and an attention mechanism, accuracy and robustness of feature extraction are improved, dependence on doctor experience is reduced, and standardization and automation of tongue picture diagnosis are achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of stroke diagnosis, and in particular, relates to a system and method for automatically identifying tongue images of stroke patients based on multi-task learning. Background Art

[0002] Stroke is one of the leading causes of adult disability worldwide, placing a heavy burden on personal health and socioeconomics. In the rehabilitation process of stroke patients, early and precise intervention is essential to improve the rehabilitation effect. Current medical diagnostic methods, such as neuroimaging, although excellent in diagnostic accuracy, are not suitable for frequent use in the rehabilitation process of stroke patients due to their high cost and operational complexity. These limitations have prompted the clinical search for other more economical and simpler diagnostic methods. As a non-invasive, low-cost diagnostic method, TCM tongue diagnosis has become a common method used by clinicians in the assessment and diagnosis of stroke patients due to its good accessibility and convenience. However, TCM tongue diagnosis relies heavily on the doctor's experience, which brings subjectivity, especially in the diagnosis of stroke patients. For example, the tongue of a stroke patient may appear dark purple or purple-red, which is usually related to blood circulation disorders. The color change of the tongue can reflect insufficient blood supply or obstructed blood flow; the thickness and color change of the tongue coating may indicate the accumulation of damp heat in the body or dysfunction of the digestive system. Since the diagnostic results of tongue quality, tongue shape, tongue coating, etc. are highly dependent on the doctor's experience and ability, and lack standardization and objectivity, the consistency and accuracy of the diagnosis are affected.

[0003] In recent years, with the development of computer technology and deep learning, artificial intelligence has made significant progress in the field of medical image analysis. In the fields of tumor recognition and pathological image analysis, deep learning methods have been widely used, and some results have been achieved in the feature extraction and recognition of TCM tongue images. AlexNet, GoogLeNet, ResNet and other methods are used to realize the recognition of cracks, tongue color and other features, but most of the existing tongue image classification studies only realize image multi-classification methods. The multi-classification method is still a single-task method, which does not integrate the various feature connections of the tongue image. It is easy to ignore the relationship between the association and conflict between tongue color, fur color, etc., resulting in the overall effect of multiple tasks cannot be better. In addition, existing research mainly focuses on the basic processing and feature extraction of tongue diagnosis images, lacking in-depth analysis and recognition of tongue images for specific subjects of stroke patients.

[0004] For example: Method 1: Use deep convolutional neural network (DCNN) to analyze the type of tongue lesions using tongue images, and apply deep learning methods to build a tongue coating segmentation model to obtain tongue coating images in open and complex scenes; Method 2: Use YOLOv5s6, U-Net and MonbileNetV3 to perform tongue coating region segmentation and tongue coating feature classification, and develop an intelligent tongue coating diagnosis system; Method 3: A robust tongue segmentation method is proposed by fusing region-based and edge-based methods; Method 4: A weakly supervised method is used to train the tooth mark and crack detection YOLO model by using tongue images with bounding box level annotations and coarse image level annotations; Method 5: Enhance the detection accuracy of cracks by combining the hybrid parallel attention mechanism (HPAM) Method 6: The decoder of the U-Net model for tongue pattern extraction is used to improve the existing U-Net deep learning method; Method 7: The decoder of the U-Net model for tongue pattern extraction is used to improve the existing U-Net deep learning method; Method 8: The decoder of the U-Net model for tongue pattern extraction is used to improve the existing U-Net deep learning method; Method 9: The new framework TongueNe is used to identify the fissure tongue. The new framework TongueNe has a higher accuracy in identifying the fissure tongue than InceptionV3 and ResNet18; Method 10: The decoder of the U-Net model for tongue pattern extraction is used to improve the existing U-Net deep learning method; Method 11: The decoder of the U-Net model for tongue pattern extraction is used to improve the existing U-Net deep learning method; Method 12: The decoder of the U-Net model for tongue pattern extraction is used to improve the existing U-Net deep learning method; Method 13: The decoder of the U-Net model for tongue pattern extraction is used to improve the existing U-Net deep learning method; Method 14: The decoder of the U-Net model for tongue pattern extraction is used to improve the existing U-Net deep learning method; Method 15: The decoder of the U-Net model for tongue pattern extraction is used to improve the existing U-Net deep learning method; Method 16: The decoder of the U-Net model for tongue pattern extraction is used to improve the existing U-Net deep learning method; Method 17: The decoder of the U-Net model for tongue pattern extraction is used to improve the existing U-Net deep learning method; Method 18: The decoder of the U-Net model for tongue pattern extraction is used to improve the existing U-Net deep learning method; Method 19: The decoder of the U-Net model for tongue pattern extraction is used to improve the existing U-Net deep learning method; Method 20: The decoder of the U-Net model for tongue pattern extraction is used to improve the existing U-Net deep learning method; Method 21: The decoder of the U-Net model for tongue pattern extraction is used to improve the existing U-Net deep learning method;

[0005] The above methods 1-9 are single-task classification and tongue image segmentation analysis, which have the following main shortcomings in practical applications:

[0006] 1. The existing technical research mainly focuses on the basic processing and feature extraction of tongue diagnosis images, and lacks in-depth analysis and identification of specific objects of tongue images of stroke patients.

[0007] 2. Lack of integration of global features: Single-task classification often only classifies a specific feature in the tongue image, such as classifying red tongue and white tongue coating as one label, without integrating the characteristic connection between tongue color and tongue coating color, ignoring the relationship and conflict between tongue color, tongue coating color, and tongue coating quality. Tongue diagnosis not only involves a single feature, but also includes the mutual relationship between multiple features such as tongue quality, tongue coating, color, thickness, texture, etc. Single-task classification can easily ignore the relationship between these features, resulting in incomplete diagnosis results that cannot fully reflect the patient's overall health status. After the tongue image is segmented, more attention is paid to the local area in the tongue image, and specific features are analyzed by dividing the tongue into multiple blocks. However, tongue diagnosis needs to be combined with global features for comprehensive judgment, and local segmentation may lead to neglect of the overall characteristics of the tongue image, thereby reducing the accuracy and reliability of the diagnosis.

[0008] 3. Ignoring the association between features: Single-task classification treats the features in tongue images in isolation, ignoring the possible interactions and conflicts between these features. For example, there is a certain connection between the tongue color and the coating color of stroke patients (e.g., pale tongue + white coating may mean that the stroke patient is physically weak, and insufficient qi and blood lead to poor blood circulation, and red tongue + yellow greasy coating may mean that the stroke patient has heavy dampness and heat in the body, and the internal obstruction of dampness leads to the accumulation of pathological products such as blood stasis and phlegm). Classifying one of the features alone may lead to missing important diagnostic information and fail to provide accurate disease assessment. The segmentation and analysis of local features leads to the loss of overall feature information, especially when dealing with complex textures or color gradients in tongue images. This method cannot accurately capture the relationship between features, affecting the final diagnostic effect.

[0009] 4. Data sparsity and insufficient generalization ability: The acquisition of tongue image data usually has certain particularities, especially when it involves certain rare feature combinations, the amount of data may be extremely limited. Single-task classification methods may encounter data sparsity problems during training, making it difficult for the model to learn effectively, and when faced with new data, the generalization ability is poor, and overfitting or inaccurate classification may occur. The amount of local image data after segmenting the tongue image may be even sparser, especially when processing certain specific features, the problem of insufficient data is more obvious. This data sparsity will limit the learning ability of the model, making it difficult to maintain stability and reliability in practical applications.

[0010] 5. Difficulty in handling the fusion of complex features: Single-task classification is difficult to handle complex multi-feature fusion problems. For example, the combination of tongue color and fur color may provide more comprehensive health information, but single-task classification is difficult to effectively integrate these features, resulting in an incomplete understanding and diagnosis of the patient's condition. Tongue image segmentation analysis leads to the local feature analysis easily ignoring the integration of overall features, especially when it comes to the fusion of complex features. This method may be difficult to provide comprehensive diagnostic information.

[0011] 6. Limited diagnostic stability and reliability: Single-task classification is easily limited by changes in feature expression because it only focuses on a single feature, resulting in instability in diagnostic results. Under different patients or under different conditions, the changes in a single feature may be significant, and single-task classification methods may not provide consistent diagnostic results. Tongue image segmentation analysis causes local image analysis to rely on features in specific areas. This local analysis method is easily affected by noise, image quality, or inaccurate labeling, reducing the stability and reliability of diagnosis.

[0012] Therefore, a tongue analysis method for stroke patients combined with multi-task learning is needed to overcome the limitations of existing methods and provide more accurate and reliable support for tongue analysis and diagnosis of stroke patients. Summary of the invention

[0013] In view of the defects of the prior art, the present invention proposes a system and method for automatic tongue image recognition of stroke patients based on multi-task learning. For the tongue image analysis of stroke patients, combined with the multi-task learning method, it can not only integrate the complex relationship of tongue color, tongue coating color and other features, but also improve the overall effect of the model through effective data processing and feature extraction, so as to provide doctors with more accurate and objective auxiliary support in stroke rehabilitation diagnosis.

[0014] In one aspect, the present invention provides a method for automatically identifying tongue images of stroke patients based on multi-task learning, the method comprising:

[0015] Step 1: Perform multi-label classification on the features of tongue images of stroke patients, and label each feature on a certain number of samples of tongue images of stroke patients with corresponding labels;

[0016] Step 2: Divide the labeled tongue images into a training set and a test set, and enhance the tongue image data in the training set and the test set to obtain the training set and the test set with expanded sample numbers;

[0017] Step 3: Build an LGAN model based on tongue image features and multi-label classification, input the training set into the LGAN model, extract the features of each tongue image, decouple the extracted features into feature representations corresponding to each label in the LGAN model, and capture the correlation between the features in each tongue image to generate the classification results for each label;

[0018] Step 4: Input the tongue image in the sample data set into the LGAN model to obtain the predicted score vector related to the feature label. Use the cross entropy loss function to calculate the loss value between the predicted score vector and the true feature label in the sample data set, and optimize the training parameters of the LGAN model according to the loss value.

[0019] Step 5: Use the confusion matrix to calculate the basic count value of each label, calculate the index of each label in the prediction result according to the basic count value and evaluation index of the label, summarize the index of each label, and obtain the performance index of the LGAN model.

[0020] Further, step 3 includes:

[0021] Step 3.1: Construct the LGAN network using the feature extractor, label decoupling module and BRNN module;

[0022] Step 3.2: Generate an LGAN model by setting training parameters based on the LGAN network;

[0023] Step 3.3: Use the feature extractor to extract the features of each tongue image;

[0024] Step 3.4: Use the label decoupling module to decouple the extracted features into feature representations corresponding to each label in the LGAN model;

[0025] Step 3.5: Reduce the dimensionality of label-related features, use the BRNN module to capture the correlation between the features in each tongue image, and generate the classification results for each label.

[0026] Furthermore, the decoupling process of the label decoupling module in step 3.4 includes: fusing the extracted image features with the corresponding label features, then mapping the fused features, using the channel attention mechanism to adjust the importance of each channel in the feature map, and using the spatial attention mechanism to adjust the importance of different spatial positions in the feature map to generate intermediate features of the tongue image.

[0027] Further, step 4 includes:

[0028] Step 4.1: Prepare a sample dataset, each sample contains a tongue image and its corresponding feature label;

[0029] Step 4.2: Input the tongue image into the LGAN model and extract the prediction score vector associated with the feature label;

[0030] Step 4.3: Use the cross entropy loss function to calculate the loss value between the model prediction and the actual label, use the optimization algorithm to update the training parameters of the LGAN model, and perform multiple iterations by setting the batch size and number of iterations to gradually optimize the parameters of the LGAN model.

[0031] Furthermore, the cross entropy loss function in step 4.3 is determined by the following formula:

[0032]

[0033] Where: M represents the number of tongue image samples; c represents the label category; represents the actual label; Represents the score vector predicted by the LGAN model.

[0034] Furthermore, the performance evaluation index of the LGAN model in step 5 is:

[0035]

[0036] Where: B macro (h) represents the performance evaluation index of the LGAN model; M represents the number of label categories; B(TP j ,FP j ,TN j ,FN j ) represents the accuracy, precision, recall, or F1 score indicator of the j-th label.

[0037] Furthermore, the characteristics of the tongue image of the stroke patient in step 1 at least include tongue color characteristics, tongue coating color characteristics and tongue coating quality characteristics.

[0038] Furthermore, the labels of tongue color characteristics include pale tongue characteristics and no pale tongue characteristics, the labels of coating color characteristics include white coating characteristics and no white coating characteristics; the labels of coating quality characteristics include greasy tongue coating and non-greasy tongue coating.

[0039] On the other hand, the present invention also provides a system for automatically identifying tongue images of stroke patients based on multi-task learning, comprising:

[0040] Data collection module, used to collect and photograph tongue images of stroke patients;

[0041] The data preprocessing module is used to divide the labeled tongue images into a training set and a test set, and to expand the number of samples in the training set and the test set;

[0042] LGAN model, including feature extraction module, label decoupling module, BRNN module, model optimization and loss calculation module, and performance evaluation module:

[0043] The feature extraction module is used to extract the features of each tongue image;

[0044] The label decoupling module is used to decouple the extracted features into feature representations corresponding to each label in the LGAN model;

[0045] The BRNN module is used to capture the correlation between the features in each tongue image and generate the classification results for each label;

[0046] The model optimization and loss calculation module is used to generate a prediction score vector from the tongue image input to the LGAN model, calculate the loss between the prediction result and the true label, and optimize the model parameters according to the loss value;

[0047] The performance evaluation module is used to calculate the index of each label in the prediction result using the confusion matrix, summarize the index of each label, and output the overall performance of the LGAN model.

[0048] The beneficial effects of the present invention are:

[0049] First, by constructing the LGAN model and combining the multi-task learning strategy with the channel and spatial attention mechanism, the present invention can more accurately extract the important features in the tongue image and capture the correlation between the features. This method reduces the neglect of the relationship between the features, improves the accuracy and robustness of recognition, and enhances the model's adaptability to diverse tongue features. Through deep learning and multi-label classification, the automatic recognition of the tongue features of stroke patients is realized, the dependence on the doctor's experience is reduced, the subjective problem in traditional Chinese medicine tongue diagnosis is solved, the standardization and objectivity of tongue diagnosis are promoted, and the consistency and reliability of diagnosis are improved.

[0050] Second, by enhancing the data of the training set and the test set, and using the BRNN module to capture the correlation between the features in the tongue image, the model can better handle the complexity and diversity of tongue image features, improve the adaptability to different tongue image feature combinations, reduce the risk of overfitting, and thus enhance the robustness and generalization ability of the model;

[0051] Third, the present invention calculates the basic count value of each label through the confusion matrix, and comprehensively evaluates the model in combination with multiple evaluation indicators (such as accuracy, precision, recall rate, F1 score, etc.), which helps to conduct a detailed analysis of the model performance and ensure the reliability and clinical practicality of the recognition method;

[0052] Fourth, compared with high-cost diagnostic methods such as traditional neuroimaging, the present invention utilizes low-cost tongue images combined with deep learning technology to achieve a fast and automated diagnostic process, greatly improving the efficiency of tongue diagnosis of stroke patients and being suitable for large-scale clinical application. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a flow chart of the method for automatic tongue recognition of stroke patients based on multi-task learning of the present invention;

[0054] Figure 2 These are tongue images of some stroke patients collected in the method for automatic tongue recognition of stroke patients based on multi-task learning of the present invention;

[0055] Figure 3 It is a LGAN network framework diagram in the method for automatic tongue recognition of stroke patients based on multi-task learning of the present invention;

[0056] Figure 4 It is a confusion matrix diagram of tongue color features, tongue coating color features, and tongue coating quality features in the method for automatic tongue image recognition of stroke patients based on multi-task learning of the present invention. DETAILED DESCRIPTION

[0057] In order to enable those skilled in the art to better understand the technical solution of the present application, the present application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0058] In the description of this application, the terms "first", "second", and "third" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance; the term "plurality" refers to two or more, unless otherwise expressly defined. Terms such as "installed", "connected", "connected", and "fixed" should be understood in a broad sense. For example, "connected" can be a fixed connection, a detachable connection, or an integral connection; "connected" can be a direct connection or an indirect connection through an intermediate medium. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to the specific circumstances.

[0059] In the description of the present application, it should be understood that the directions or positional relationships indicated by the terms "up", "down", "left", "right", "front", "back", etc. are based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or unit referred to must have a specific direction, be constructed and operated in a specific orientation, and therefore, should not be understood as a limitation on the present application.

[0060] In the description of this specification, the description of the terms "one embodiment / method", "some embodiments / methods", "specific embodiments / methods", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment / method or example are included in at least one embodiment / method or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment / method or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments / methods or examples in a suitable manner.

[0061] As the instruction manual Figure 1 The present invention provides a method for automatically identifying tongue images of stroke patients based on multi-task learning, comprising:

[0062] Step 1: Perform multi-label classification on the features of tongue images of stroke patients, and label each feature on a certain number of sample tongue images of stroke patients with a corresponding label.

[0063] In the present invention, the characteristics of tongue images of stroke patients at least include tongue color characteristics, tongue coating color characteristics, and tongue coating quality characteristics. The tongue color characteristics are classified by labeling according to the characteristics of pale tongue and the characteristics of no pale tongue, the tongue coating color characteristics are classified by labeling according to the characteristics of white tongue coating and the characteristics of no white tongue coating, and the tongue coating quality characteristics are classified by labeling according to the characteristics of greasy tongue coating and non-greasy tongue coating.

[0064] Label classification is not limited to the labels set by the present invention. In one implementation, multi-label classification can further refine the labels according to the above features. In another implementation, in addition to identifying tongue color, coating color and coating quality features, tongue shape features (labels include fat tongue, thin tongue, cracked tongue, tooth-marked tongue, etc.), sublingual collateral features (labels include varicose sublingual collaterals, dark purple sublingual collaterals, etc.), tongue coating thickness features (labels include thick coating, thin coating, no coating, etc.) and other features can also be identified to further improve the application value and diagnostic accuracy of traditional Chinese medicine tongue diagnosis in the rehabilitation process of stroke patients.

[0065] A specific embodiment may include the following process:

[0066] We collected and screened tongue images of 165 stroke patients from a certain hospital. All images were taken by specific equipment, such as tongue imagers and high-resolution digital cameras. Three professional doctors annotated the images, including tongue color, coating color, and coating quality. Among them, there were 70 images with "pale tongue characteristics", 50 images with "white coating characteristics", and 62 images with "non-greasy tongue coating", as shown in the attached manual. Figure 2 Shown are pictures of tongue images of some stroke patients.

[0067] The present invention adopts multi-label classification to allow each tongue image to have multiple labels at the same time, which can realize the standardization of diagnosis. Tongue color, coating color and coating quality are standardized diagnostic indicators in traditional Chinese medicine tongue diagnosis, which reflect different pathological characteristics respectively. By classifying these labels, complex tongue image features can be converted into medical diagnostic information that is easy to understand and analyze, helping doctors make more accurate judgments. Although tongue color, coating color and coating quality have their own independent pathological significance, they often show certain correlations in clinical practice. Classifying these labels helps the model learn and understand each feature separately, and can also capture the potential correlation between these features through multi-task learning, thereby improving the accuracy and reliability of the overall diagnosis. In actual data sets, samples of certain feature combinations may be very rare. By classifying each feature separately, the problem of insufficient data caused by rare feature combinations can be avoided, ensuring that each feature classification has sufficient data for effective training.

[0068] Step 2: Divide the labeled tongue images into a training set and a test set, enhance the tongue image data in the training set and the test set, and obtain the training set and the test set with expanded sample quantity.

[0069] It should be noted that if the data set of tongue images of stroke patients is too small, overfitting is likely to occur in the training project, making the training results more specific to the training set, but not performing well enough on the test set, lacking sufficient generalization ability. Therefore, it is necessary to expand the number of samples of tongue images of stroke patients.

[0070] The data enhancement of the present invention at least includes adding impulse noise, changing contrast, horizontal mirroring and affine transformation. Specifically:

[0071] The addition of impulse noise is manifested as random extreme bright or dark spots in the tongue images of stroke patients, usually white and black pixels. This noise can cause the local pixel values ​​of the tongue images of stroke patients to suddenly change to the maximum or minimum value. By adding impulse noise to the tongue image, the noise interference that may occur during the acquisition of tongue images of stroke patients can be simulated. The training model can still maintain good recognition performance when dealing with noise-contaminated data, thereby improving the robustness and generalization ability of the model.

[0072] Changing the contrast can effectively affect features such as tongue color and coating color by sharpening, smoothing and denoising, and adjusting grayscale, thereby increasing data diversity and improving the generalization ability of the model. For example, by sharpening the image, the edges and details in the image can be enhanced, the clarity of the image can be improved, and the color and texture in the tongue image can be more obvious. The sharpened image helps the model better identify subtle differences in tongue color and coating, and is suitable for tasks that require high-resolution features such as analyzing tongue quality and cracks. Smoothing and denoising can reduce noise in the image, weaken the contrast of the image, and make the color and texture in the tongue image smoother. The smoothed image can help the model identify the overall color distribution in the tongue image without being disturbed by local noise, and is suitable for analyzing global features such as tongue coating thickness and uniformity. Grayscale adjustment is manifested by adjusting the grayscale of the image to change the overall contrast of the image, making the tongue color feature more prominent, and is suitable for identifying features that rely on color contrast such as tongue color and coating color.

[0073] Horizontal mirroring is the process of flipping an image along its vertical axis to generate a sample with similar content to the original image but in the opposite direction. By horizontally mirroring, more samples can be generated, increasing the diversity of the dataset and enabling the model to better adapt to different angles or postures that may appear in tongue images. Although the mirrored image is similar in content to the original image, it is visually different, which forces the model to learn a wider range of features, thereby improving its ability to generalize to new data.

[0074] Affine transformation is a geometric transformation that can maintain the flatness and proportional relationship of an image. It includes operations such as translation, rotation, scaling, shearing (tilting), etc., which can linearly transform the tongue image to generate a variety of tongue image perspectives and shapes. Affine transformation can change the structure and shape of an image more widely. Compared with simple mirror or rotation transformations, affine transformation can more flexibly generate diverse data samples, which helps to improve the robustness and generalization ability of the model under multiple perspectives and deformation conditions.

[0075] A specific embodiment may include the following process:

[0076] Based on the embodiment of step 1, the labeled tongue images of 165 stroke patients were divided into 115 tongue images as a training set and 50 tongue images as a test set. After the above four image data enhancement methods, the data in the training sample was expanded from 115 to the current 920, and the test sample was expanded from the original 50 to the current 400.

[0077] Step 3: Construct an LGAN model based on tongue image features and multi-label classification. Input the training set into the LGAN model to extract the features of each tongue image. Decouple the extracted features into feature representations corresponding to each label in the LGAN model, capture the correlation between the features in each tongue image, and generate the classification results for each label.

[0078] Specifically include:

[0079] Step 3.1: Construct the LGAN network using the feature extractor, label decoupling module and BRNN module (characterization label correlation module).

[0080] LGAN (Label-Guided Attention Network) is a deep learning network architecture based on the label-guided attention mechanism.

[0081] The feature extractor is used to extract feature maps from the input tongue image. The feature extractor is composed of a deep convolutional neural network (CNN), which includes multiple stacked convolution blocks. The convolution blocks include convolution layers, batch normalization layers (BN), nonlinear activation layers (ReLU), and maximum pooling layers. Through multi-layer convolution operations, the convolution layer generates feature maps of the tongue image. These features contain information such as tongue color, fur color, and fur texture of the tongue image. The BN layer is used to normalize the output of the convolution layer. By adjusting the distribution of activation values, the internal covariate shift is reduced, the training process is accelerated, and the stability of the model is improved. The BN layer can also reduce the sensitivity to initialization and help prevent overfitting. The ReLU layer is used to enable the network to better fit complex functions. The maximum pooling layer is used to downsample the extracted tongue feature map, reduce the spatial size of the feature map, and highlight important features.

[0082] The label decoupling module is used to decouple the features of tongue images from different labels and learn the specific features corresponding to each label. In this way, the LGAN network can focus on and process the image area related to each label more accurately.

[0083] The BRNN module (BRNN-Bidirectional Recurrent Neural Network) is used to capture the correlation between different tongue features in the tongue image. Since there may be complex interactions between tongue features (such as the relationship between tongue color and fur color), the BRNN module can further learn and characterize these correlations through the mechanism of recurrent neural network (RNN).

[0084] The LGAN network of the present invention uses label decoupling and BRNN modules to enable the LGAN model to more accurately handle the multi-label classification task of tongue images, so that the LGAN model can capture the complex label correlation in tongue images, which helps to improve the classification accuracy and the generalization ability of the model.

[0085] Step 3.2: Generate the LGAN model by setting training parameters based on the LGAN network.

[0086] Specifically, the training parameters of the feature extractor of the present invention include:

[0087] The feature extractor consists of 4 convolution blocks, each of which contains the following components: 2 convolution layers: Each convolution layer uses a 3x3 convolution kernel, and the number of filters is 32, 32, 64, and 128 respectively. As the network deepens, the number of filters gradually increases to capture more complex and abstract features. 1 batch normalization layer (BN): In each convolution block, the batch normalization layer follows the convolution layer to help stabilize the training process and standardize the distribution of the output. 1 ReLU activation layer: The ReLU activation function in each convolution block is a nonlinear transformation, which gives the network stronger expressive power. 1 maximum pooling layer: The maximum pooling layer at the end of each convolution block is used to downsample the feature map, and the pooling kernel size is 2x2, which is used to reduce the spatial dimension of the feature map while retaining the most important features.

[0088] The training parameters of the label decoupling module include:

[0089] Joint embedding and output feature dimensions: The dimensions of d1 and d2 are both set to 128 to ensure that the features and embedding space after label decoupling have sufficient expressive power. Settings of W, H, and C: These parameters represent the dimensions of the input features, which are 128, 16, 16, and 3, respectively. W and H are the width and height of the feature map, and C is the number of channels. Channel attention module: The reduction ratio is set to 16, which is used to control the degree of reduction in the number of channels. By reducing the number of channels, the computational complexity can be reduced while focusing on the most important channel information. The number of kernels in the convolutional layers Wo and Wi: 128 respectively. The two convolutional layers process the reduced channel information to further enhance the feature extraction capability. Spatial attention module: The implementation of the fb(·) function is a single convolutional layer with a kernel size of 7x7. The spatial attention module helps the LGAN model focus on important areas in the spatial dimension of the feature map and improve the spatial sensitivity of the feature representation.

[0090] The training parameters of the BRNN module include:

[0091] Number of hidden layer neurons: Set to 4. The hidden layer in BRNN has 4 neurons to process the temporal correlation or sequence correlation of input features. The fully connected layer V has an 8:1 ratio and converts the state vector of BRNN into a single output score. The fully connected layer V further extracts and compresses information so that the final output can be better used for classification or regression tasks.

[0092] As shown in Table 1, Table 1 is the training parameters set for the LGAN network of the present invention:

[0093] Layer(type) Outputshape InputLayer (3,256,256) CNNBlock1 (32,128,128) CNNBlock2 (32,64,64) CNNBlock3 (64,32,32) CNNBlock4 (128,16,16) DecouplingLayer (3,128,16,16) ChannelAttention (3,128,16,16) SpatialAttention (3,128,16,16) CNNBlock5 (3,128) BRNNlayer (None,3) OutputLayer (None,3)

[0094] Table 1

[0095] In step 3.2, each convolution block gradually extracts higher-level features. The label decoupling layer and attention mechanism help the LGAN model to better focus on key features. The BRNN layer further integrates the correlation between features. The final output layer generates the prediction results for each label.

[0096] Step 3.3: Use the feature extractor to extract the features of each tongue image.

[0097] The feature extractor is determined by the following formula:

[0098] f 1dcnn ((I) = f I ∈R N×H×W (1)

[0099] Where: f 1dcnn (I) represents the feature extractor; f I represents feature mapping; I represents the input tongue image; N represents the number of channels of the feature map; H represents the height of the feature map; W represents the width of the feature map.

[0100] Step 3.4: Use the label decoupling module to decouple the extracted features into feature representations corresponding to each label in the LGAN model.

[0101] For each tongue feature category, use a one-hot encoding vector [x1,x2,…,x c ] to represent its category information, where c represents the c-th feature category, and the length of the vector represents the total number of all feature categories. Then, the low-rank bilinear pooling method is used to pool the image feature f I With label feature x c Fusion, for each image feature and label feature, is implemented using a fully connected layer, and the fused label feature map is normalized using a hyperbolic tangent function to obtain the tongue image intermediate feature map. The fusion process of the tongue image intermediate feature map is expressed by the following formula:

[0102]

[0103] Where: tanh(·) represents the hyperbolic tangent function; represents a learnable parameter; d1, d2, and c represent the dimensions of the joint embedding, the dimensions of the output features, and the dimensions of the label vector, respectively.

[0104] After fusion by formula (1), the intermediate feature map of tongue image is obtained

[0105] Furthermore, the channel attention mechanism and spatial attention mechanism are used to map the intermediate features of the tongue image. Make enhancements.

[0106] Spatial attention is used to adjust the importance of different spatial positions in the feature map so that the network can better focus on the information of important areas of the tongue image. For each group of tongue image feature categories and feature channels (c, d), the feature attention coefficient a can be calculated by the following formula c,d :

[0107]

[0108] Where: σ(·) represents the sigmoid (S-shaped growth curve) function; f avg (·) represents global average pooling; f max (·) represents global maximum pooling, which can be achieved through convolutional layers; represents the learnable parameter and r represents the reduction ratio.

[0109] The channel attention mechanism is used to adjust the importance of different channels so that the network can focus more on the important channel features in the tongue depth information. For each group of vibration signal categories and positions (c, H, W), the position attention coefficient b c,H,W Calculated by the following formula:

[0110]

[0111] Where: σ(·) represents the sigmoid (S-shaped growth curve) function; f b (·) indicates a convolution operation with only one kernel.

[0112] Through the attention mechanism module, the neural network can learn the relationship between features more flexibly, thereby improving the network's modeling ability for input data and further improving the performance in tongue feature recognition.

[0113] Step 3.5: Reduce the dimensionality of label-related features, use the BRNN module to capture the correlation between the features in each tongue image, and generate the classification results for each label.

[0114] The present invention utilizes three convolutional layers and one global average pooling layer to reduce the dimensionality of label-related features.

[0115] Reducing the dimension of label-related features can be expressed as:

[0116] f c =f avg (f cnn (F c )) (5)

[0117] Where: f c represents the label-related feature vector; f avg (·) represents global average pooling; f cnn(·) indicates stacked convolution operation; F c Represents the feature map output by the convolutional layer.

[0118] The feature vector set {f1,f2,...,f c}, a single-layer BRNN network is used to learn the mutual influence of these feature vectors. Each BRNN neuron consists of two node vectors: the forward state vector s t With the reverse state vector The value of each state vector is determined by the input of the current position and the previous state, and can be calculated by the following formula:

[0119] s t =tanh(Uf t +Ws t-1 +b) (6)

[0120]

[0121] Where: tanh(·) represents the hyperbolic tangent function U; W, W'∈R h×h , b, b'∈R h represents the learnable weights; h and d2 represent the state vector dimension of the hidden layer and the feature vector dimension of the input layer, respectively.

[0122] Using BRNN can better capture the contextual information of input data. The input data of the present invention is a feature vector with mutual information of labels, so BRNN can understand the interaction of labels and learn the features between labels to obtain better performance.

[0123] Furthermore, in the output layer, the forward and backward state vectors are combined into an intermediate vector, which is used to predict the confidence score of the presence of the tongue image feature. The output score is calculated as:

[0124]

[0125] Where: σ(·) represents the sigmoid (S-shaped growth curve) function; V∈R 1×2h represents the weight parameter of the linear transformation, and b is the bias term.

[0126] Based on formula (8), the probability of each component between (0, 1) is obtained. Each component corresponds to a tongue feature category. Then, after threshold processing, the probability greater than or equal to 0.5 is set to 1, and the probability less than 0.5 is set to 0. The threshold function t(·) is defined as follows:

[0127]

[0128] For each tongue coating image feature category c, the predicted tongue coating feature label is calculated as This process is performed for all categories to obtain a binary label vector set

[0129] Combined with Table 1 in step 3.2, the specific embodiment of step 3 includes the following process:

[0130] First, the size of the InputLayer input layer is (3,256,256), indicating that the input image has 3 channels, and the resolution of each channel is 256x256 pixels; then the output shape of the first convolution block of CNN Block 1 is (32,128,128). After the first convolution block, 32 feature maps are output. These feature maps extract different features of the tongue image. After convolution and pooling operations, the size of the input tongue image is reduced from 256x256 to 128x128. Similarly, after CNN Block 2-4, the fourth convolution block further increases the number of feature maps to 128, and the spatial resolution is reduced to 16x16. Then the output shape of the fourth convolution block is processed in the Decoupling Layer label decoupling layer. The processed output shape is displayed as a multi-label feature (3,128,16,16), where 3 represents the number of 3 labels. The dimension of each label is consistent with the previous convolution block, showing the decoupled feature map. The ChannelAttention layer weights the three labels in the output shape (3,128,16,16) to emphasize the information of important labels. The output shape is the same as the input, indicating that the feature dimension has not changed. The spatial position of the feature map is weighted by the spatial attention layer to highlight the key spatial area. The output shape is the same as the channel attention layer. Then, the fifth convolutional block of CNN Block 5 compresses and integrates the previous feature maps to generate the final 128-dimensional features. Each label is processed independently and the output shape is (3,128). Then, the correlation between different features is captured through the bidirectional recurrent neural network (BRNN) layer. The output dimension is dynamically adjusted according to the sequence length (None), and the output scores of the three labels are finally generated. Finally, the final output layer of OutputLayer generates the final prediction result (None,3) for each label.

[0131] Step 4: Input the tongue image in the sample data set into the LGAN model to obtain the predicted score vector related to the feature label. Use the cross entropy loss function to calculate the loss value between the predicted score vector and the true feature label in the sample data set, and optimize the training parameters of the LGAN model according to the loss value.

[0132] Specifically include:

[0133] Step 4.1: Prepare a sample dataset, each sample contains a tongue image and its corresponding feature label.

[0134] The dataset is expressed as follows:

[0135]

[0136] Where: I i represents the i-th input tongue image; M represents the number of samples; y i represents the feature label corresponding to the i-th input tongue image (i.e., the actual label of the tongue image). Multiple feature labels corresponding to the tongue image are available express.

[0137] Step 4.2: Put the tongue image I i Input the LGAN model and extract the prediction score vector associated with the feature label

[0138] Step 4.3: Use the cross entropy loss function to calculate the loss value between the model prediction and the actual label, use the optimization algorithm to update the training parameters of the LGAN model, and perform multiple iterations by setting the batch size and number of iterations to gradually optimize the parameters of the LGAN model.

[0139] The cross entropy loss function is determined by the following formula:

[0140]

[0141] Where: M represents the number of tongue image samples; c represents the label category; represents the actual label; Represents the score vector predicted by the LGAN model.

[0142] The batch size of the cross entropy loss function of the present invention is set to 8, that is, 8 samples are used for training each time, and the number of training iterations is set to 100 times. The optimization algorithm of the Adam optimizer is used to update the training parameters of the LGAN model, and the initial learning rate of the Adam optimizer is set to 0.001, which determines the step size of each parameter update. Through step 4, the LGAN model gradually learns the features in the data, optimizes its classification ability, and makes the prediction more accurate.

[0143] Step 5: Use the confusion matrix to calculate the basic count value of each label, calculate the index of each label in the prediction result according to the basic count value and evaluation index of the label, summarize the index of each label, and obtain the performance index of the LGAN model.

[0144] Step 5 specifically includes:

[0145] Step 5.1: Construct a confusion matrix and extract the four basic count values ​​of true positive (TP), false positive (FP), false negative (FN), and true negative (TN) for each label.

[0146] The conditions for the establishment of each basic count value in the confusion matrix can be expressed as:

[0147] TP j =|{x (i) |l j ∈y (i) ∧l j ∈h(x (i) ),1≤i≤M}|

[0148]

[0149] FN j =|{x (i) |l j ∈y (i) ∧l j ⊕h(x (i) ),1≤i≤M}|

[0150] Where: TP j FP represents the number of samples in which the jth label is correctly predicted as positive (i.e., the jth label appears in both the true label and the predicted label); j TN represents the number of samples where the jth label is incorrectly predicted as positive (i.e., the jth label is not in the true label but is predicted as positive); j FN represents the number of samples where the jth label is correctly predicted as negative (i.e., the jth label is neither in the true label nor in the predicted label); j represents the number of samples where the jth label is incorrectly predicted as negative (i.e., the jth label is in the true label but is not predicted); x (i) Represents the data input of the i-th tongue image; l j represents the jth tongue image label; y (i) represents the actual label set of the i-th tongue image; h(x (i) ) represents the predicted label set of the LGAN model for the i-th tongue image; M represents the number of tongue images.

[0151] Step 5.2: According to the extracted four basic count values, the basic evaluation indicators of each label are calculated using accuracy, precision, recall and F1 score. The calculation results of accuracy, precision, recall and F1 score are macro-averaged to obtain the performance evaluation indicators of the LGAN model.

[0152] Accuracy refers to the proportion of samples correctly predicted by the model to the total samples, and is an indicator to measure the overall prediction ability. Its evaluation index is expressed by the following formula:

[0153]

[0154] Precision is used to measure the proportion of all samples predicted to be positive that are actually positive, reflecting the reliability of the model's prediction of positive results. Its evaluation index is expressed by the following formula:

[0155]

[0156] Recall refers to the proportion of samples that are correctly predicted as positive among all samples that are actually positive, reflecting the model's ability to find positive samples. Its evaluation index is expressed as follows:

[0157]

[0158] The F1 score represents the harmonic mean of precision and recall, and is used to find a balance between the two, especially in the case of class imbalance. Its evaluation index is expressed by the following formula:

[0159]

[0160] According to formulas (12)-(15), the performance evaluation index of the LGAN model is obtained:

[0161]

[0162] Where: B macro (h) represents the performance evaluation index of the LGAN model; M represents the number of label categories; B(TP j ,FP j ,TN j ,FN j ) represents the accuracy, precision, recall, or F1 score indicator of the j-th label.

[0163] The present invention adopts the macro-average method to calculate the evaluation index for each label separately, and then take the average value. In this way, the macro-accuracy, macro-precision, macro-recall and macro-F1 score are calculated, which can comprehensively evaluate the comprehensive performance of the model in multi-label or multi-category tasks.

[0164] According to the method for automatic tongue recognition of stroke patients based on multi-task learning of the present invention, the collected tongue image data set is manually divided into a training set and a test set in a ratio of 7:3. The training is performed for 100 rounds, and the test set data is used to verify the model results in each round to obtain the accuracy analysis results of the training set and the test set.

[0165] As the instruction manual Figure 4 As shown in ac, the upper left corner (TP) represents the proportion of actual positive and predicted positive; the upper right corner (FP) represents the proportion of actual negative and predicted positive (false positive); the lower left corner (FN) represents the proportion of actual positive and predicted negative (false negative); the lower right corner (TN) represents the proportion of actual negative and predicted negative.

[0166] Figure 4 a represents the confusion matrix of tongue color features, from Figure 4 aIt can be seen that the model performs well in identifying tongue color features, especially in distinguishing whether the "pale tongue" feature exists. Specifically, the recognition rate of the positive class (the "pale tongue" feature exists) is 80%, while the recognition rate of the negative class (the "pale tongue" feature does not exist) is 89%. The false positive rate and false negative rate are 20% and 11% respectively.

[0167] Figure 4 b represents the confusion matrix of moss color features, from Figure 4 b It can be seen that the recognition model for moss color features performs slightly worse, especially when distinguishing whether the "white moss" feature exists. Specifically, the recognition rate of the positive class (the "white moss" feature exists) is 65%, and the recognition rate of the negative class (the "white moss" feature does not exist) is 85%. The false positive rate is high (35%), indicating that the model often misjudges the presence of the "white moss" feature.

[0168] Figure 4 c represents the confusion matrix of moss characteristics, from Figure 4 c It can be seen that the model performs well in identifying moss features, especially when distinguishing whether there is a "greasy moss" feature. The recognition rate is significantly improved. Specifically, the recognition rate of the positive class (the presence of the "greasy moss" feature) is 74%, and the recognition rate of the negative class (the absence of the "greasy moss" feature) is as high as 93%. The false positive rate and false negative rate are 26% and 7% respectively, indicating that the model is very accurate in identifying negative classes.

[0169] It can be seen from the confusion matrix that the model performs well in the recognition of tongue features, especially in distinguishing the features of "pale tongue" and "greasy fur". For the recognition of the "white fur" feature, the accuracy of the model has decreased, which indicates that it may be necessary to further optimize the parameters of the model or increase the training of relevant samples to reduce the occurrence of false positives. Overall, the LGAN model of the present invention can effectively identify most tongue features.

[0170] In order to comprehensively evaluate the performance of the LGAN model of the present invention, the LGAN model, as well as the existing AlexNet model, VGG16 model, ResNet-18 model, ResNet-50 model and DenseNet model were trained and tested on the same dataset, and the performance difference between the LGAN model of the present invention and the existing models was compared.

[0171] It should be noted that AlexNet is the champion model of ILSVRC in 2012. It contains 5 convolutional layers and 3 fully connected layers. It uses ReLU activation function and overlapping maximum pooling, which significantly improves the performance of convolutional neural networks in image classification tasks and effectively reduces overfitting through Dropout technology. VGG16 consists of 16 layers, uses 3×3 convolution kernels, has a large number of layers, and the activation function after all convolutional layers is ReLU. Although the number of parameters is large and the computing and storage resource requirements are high, it has achieved excellent results on multiple image classification benchmark datasets. ResNet-18 and ResNet-50 are residual networks that use residual modules to solve the gradient vanishing problem in deep neural networks. They transfer information in deeper networks through shortcut connections, and can train very deep networks and perform well in multiple image classification tasks. DenseNet achieves feature reuse and efficient transmission of gradients by establishing dense connections between network layers. Each layer receives the input of all previous layers, which enables the network to use parameters more effectively and alleviate the gradient vanishing problem. It has a compact structure, a small number of parameters, but excellent performance.

[0172] The experimental results are shown in Table 2-4 below. Table 2-4 respectively shows the comparison of the results of tongue color feature, coating color feature, and coating quality feature recognition. The evaluation indicators include accuracy (ACC), precision (Precision), recall rate (Recall), and F1 value (F-measure).

[0173] Model Category ACC PREC RECALL F1-score alexnet 0.7200 0.5263 0.6667 0.5882 VGG16 0.7600 0.5890 0.7333 0.6471 densenet 0.7400 0.5417 0.8667 0.6667 resnet-18 0.7400 0.6000 0.4000 0.4800 resnet-50 0.7200 0.55555556 0.3333 0.4167 LGAN 0.8600 0.7500 0.8000 0.7742

[0174] Table 2

[0175] As can be seen from Table 2, the LGAN model performs best, with better accuracy (0.8600), precision (0.7500), recall (0.8000) and F1 score (0.7742) than other models. The VGG16 and Densenet models perform relatively well in terms of precision and recall, but are inferior to the LGAN model. ResNet-18 and ResNet-50 perform poorly in terms of recall, especially ResNet-50, which has a recall of only 0.3333, resulting in a lower F1 score (0.4167)

[0176] Model Category ACC PREC RECALL F1-score alexnet 0.7200 0.5882 0.5882 0.5882 VGG16 0.7000 0.5556 0.5882 0.5714 densenet 0.6800 0.5384 0.4118 0.4667 resnet-18 0.7400 0.6667 0.4706 0.5517 resnet-50 0.6800 0.6667 0.1176 0.2000 LGAN 0.7800 0.6875 0.6471 0.6667

[0177] Table 3

[0178] As can be seen from Table 3, LGAN still performs well, outperforming most of the comparison models in terms of precision (0.6875), recall (0.6471), and F1 score (0.6667). ResNet-18 performs well in terms of precision (0.6667), but has a low recall (0.4706). ResNet-50 has an extremely low recall (0.1176), which seriously affects its F1 score (0.2000), indicating that the model has significant deficiencies in this task.

[0179] Model Category ACC PREC RECALL F1-score alexnet 0.7400 0.9231 0.6857 0.7869 VGG19 0.7600 0.9259 0.7142 0.8065 densenet 0.7600 0.8966 0.1429 0.8125 resnet-18 0.8000 0.9310 0.7714 0.8438 resnet-50 0.7800 0.8750 0.8000 0.8358 LGAN 0.8000 0.9630 0.7428 0.8387

[0180] Table 4

[0181] As can be seen from Table 4, LGAN and ResNet-18 perform best, with similar accuracy and precision, and high recall and F1 scores (LGAN's accuracy is 0.9630 and F1 score is 0.8387). Densenet also has high accuracy (0.8966), but low recall (0.1429), indicating that the model misses a large number of positive examples with high accuracy. VGG19 and ResNet-50 perform relatively balanced, but are not as good as the LGAN model overall.

[0182] Furthermore, the accuracy, precision, recall and F1 score of each model are macro-averaged to obtain the macro-evaluation indicators of each model as shown in Table 5.

[0183]

[0184]

[0185] Table 5

[0186] As can be seen from Table 5, the LGAN model performs best, with the highest accuracy (0.8133), precision (0.8002), recall (0.7280) and F1 score (0.7599), showing high consistency and comprehensive performance, especially maintaining a good balance between precision and recall, significantly outperforming other models, and being able to better identify positive samples and avoid false predictions. Although the ResNet-18 model has a high accuracy (0.7328), its recall rate is low (0.5473), resulting in an F1 score of 0.6252, indicating that it performs well in identifying positive samples, but may miss more positive samples. The VGG16 model performs moderately, balanced but not outstanding. AlexNet's performance is close to VGG16, indicating that it can maintain a certain level, but there is a certain gap compared to more modern models. Densenet performs stably, but not optimally, and all indicators are at a medium to low level. Although the ResNet-50 model has good accuracy (0.6991), its recall rate is only 0.4170 and the F1 score is significantly reduced to 0.4842, indicating that it has obvious problems in the task, especially the inability to effectively identify enough positive samples.

[0187] The present invention also provides a stroke patient tongue image automatic recognition system based on multi-task learning, including a data acquisition module, a data preprocessing module and a LGAN model. The LGAN model includes a feature extraction module, a label decoupling module, a BRNN module, a model optimization and loss calculation module and a performance evaluation module.

[0188] The data acquisition module is used to collect and photograph the tongue images of stroke patients.

[0189] The data preprocessing module is used to divide the labeled tongue image into a training set and a test set, and to expand the number of samples in the training set and the test set.

[0190] The feature extraction module is used to extract the features of each tongue image.

[0191] The label decoupling module is used to decouple the extracted features into feature representations corresponding to each label in the LGAN model.

[0192] The BRNN module is used to capture the correlation between the features in each tongue image and generate the classification results for each label.

[0193] The model optimization and loss calculation module is used to generate a prediction score vector for the tongue image input into the LGAN model, calculate the loss between the prediction result and the true label, and optimize the model parameters according to the loss value.

[0194] The performance evaluation module is used to calculate the index of each label in the prediction result using the confusion matrix, summarize the index of each label, and output the overall performance of the LGAN model.

[0195] The stroke patient tongue automatic recognition system based on multi-task learning of the present invention simultaneously identifies multiple tongue image features through multi-label classification, thereby improving the diagnostic accuracy of the system, improving the efficiency and effect of model training through model optimization, and using confusion matrix and multiple evaluation indicators to comprehensively evaluate the model performance to ensure the reliability of the system in clinical applications. Application scenarios include medical auxiliary diagnosis and health monitoring. When applied to medical auxiliary diagnosis, it can assist doctors to quickly and accurately identify the tongue image features of stroke patients and provide support for diagnosis and treatment decisions; when applied to health monitoring, it can perform daily monitoring of the tongue images of stroke patients and provide early warnings and suggestions for changes in the condition. The system provides an important auxiliary tool for clinical diagnosis and health management.

[0196] The above is only an embodiment of the present invention, and the common sense such as the known specific structure and characteristics in the scheme is not described in detail here. For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present invention is defined by the attached claims rather than the above description, so it is intended to include all changes within the meaning and scope of the equivalent elements of the claims. Any figure mark in the claims should not be regarded as limiting the claims involved.

Claims

1. A method for automatic tongue recognition of stroke patients based on multi-task learning, characterized in that: Methods include: Step 1: Perform multi-label classification on the features of tongue images of stroke patients, and label each feature on a certain number of samples of tongue images of stroke patients with corresponding labels; Step 2: Divide the labeled tongue images into a training set and a test set, and enhance the tongue image data in the training set and the test set to obtain the training set and the test set with expanded sample numbers; Step 3: Build an LGAN model based on tongue image features and multi-label classification, input the training set into the LGAN model, extract the features of each tongue image, decouple the extracted features into feature representations corresponding to each label in the LGAN model, and capture the correlation between the features in each tongue image to generate the classification results for each label; Step 4: Input the tongue image in the sample data set into the LGAN model to obtain the predicted score vector related to the feature label. Use the cross entropy loss function to calculate the loss value between the predicted score vector and the true feature label in the sample data set, and optimize the training parameters of the LGAN model according to the loss value. Step 5: Use the confusion matrix to calculate the basic count value of each label, calculate the index of each label in the prediction result according to the basic count value and evaluation index of the label, summarize the index of each label, and obtain the performance index of the LGAN model.

2. The method for automatic tongue recognition of stroke patients based on multi-task learning according to claim 1, characterized in that: Step 3 includes: Step 3.1: Construct the LGAN network using the feature extractor, label decoupling module and BRNN module; Step 3.2: Generate an LGAN model by setting training parameters based on the LGAN network; Step 3.3: Use the feature extractor to extract the features of each tongue image; Step 3.4: Use the label decoupling module to decouple the extracted features into feature representations corresponding to each label in the LGAN model; Step 3.5: Reduce the dimensionality of label-related features, use the BRNN module to capture the correlation between the features in each tongue image, and generate the classification results for each label.

3. The method for automatic tongue recognition of stroke patients based on multi-task learning according to claim 1, characterized in that: Step 3.4 The decoupling process of the label decoupling module includes: fusing the extracted image features with the corresponding label features, then mapping the fused features, using the channel attention mechanism to adjust the importance of each channel in the feature map, and using the spatial attention mechanism to adjust the importance of different spatial positions in the feature map to generate the intermediate features of the tongue image.

4. The method for automatic tongue recognition of stroke patients based on multi-task learning according to claim 1, characterized in that: Step 4 includes: Step 4.1: Prepare a sample dataset, each sample contains a tongue image and its corresponding feature label; Step 4.2: Input the tongue image into the LGAN model and extract the prediction score vector associated with the feature label; Step 4.3: Use the cross entropy loss function to calculate the loss value between the model prediction and the actual label, use the optimization algorithm to update the training parameters of the LGAN model, and perform multiple iterations by setting the batch size and number of iterations to gradually optimize the parameters of the LGAN model.

5. The method for automatic tongue recognition of stroke patients based on multi-task learning according to claim 4 is characterized in that: The cross entropy loss function in step 4.3 is determined by the following formula: Where: M represents the number of tongue image samples; c represents the label category; represents the actual label; Represents the score vector predicted by the LGAN model.

6. The method for automatic tongue recognition of stroke patients based on multi-task learning according to claim 1, characterized in that: Performance evaluation indicators of the LGAN model in step 5: Where: B macro (h) represents the performance evaluation index of the LGAN model; M represents the number of label categories; B(TP j ,FP j ,TN j ,FN j ) represents the accuracy, precision, recall, or F1 score indicator of the j-th label.

7. The method for automatic tongue recognition of stroke patients based on multi-task learning according to claim 1, characterized in that: The characteristics of the tongue image of stroke patients in step 1 at least include tongue color characteristics, tongue coating color characteristics and tongue coating quality characteristics.

8. The method for automatic tongue recognition of stroke patients based on multi-task learning according to claim 7, characterized in that: The labels of tongue color characteristics include pale tongue characteristics and no pale tongue characteristics, the labels of coating color characteristics include white coating characteristics and no white coating characteristics; the labels of coating quality characteristics include greasy tongue coating and non-greasy tongue coating.

9. An automatic tongue recognition system for stroke patients based on multi-task learning, comprising: Data collection module, used to collect and photograph tongue images of stroke patients; The data preprocessing module is used to divide the labeled tongue images into a training set and a test set, and to expand the number of samples in the training set and the test set; LGAN model, including feature extraction module, label decoupling module, BRNN module, model optimization and loss calculation module, and performance evaluation module: The feature extraction module is used to extract the features of each tongue image; The label decoupling module is used to decouple the extracted features into feature representations corresponding to each label in the LGAN model; The BRNN module is used to capture the correlation between the features in each tongue image and generate the classification results for each label; The model optimization and loss calculation module is used to generate a prediction score vector from the tongue image input to the LGAN model, calculate the loss between the prediction result and the true label, and optimize the model parameters according to the loss value; The performance evaluation module is used to calculate the index of each label in the prediction result using the confusion matrix, summarize the index of each label, and output the overall performance of the LGAN model.

Images