A tongue diagnosis multi-modal data classification method, system and storage medium

Abstract

The first aspect of the present application provides a tongue diagnosis multi-modal data classification method, comprising the steps of: S1, acquiring diagnosis image data and diagnosis text data, and extracting data features respectively to obtain image data features and text data features; S2, fusing the image data features and the text data features to obtain multi-modal data fusion features; and S3, inputting the fusion features into a classification model to obtain diagnosis prediction or syndrome classification results. Further, a random attention mechanism, a Trans-CNN classifier and a CAM attention mechanism module are introduced. The above-mentioned model has good applicability and expansibility, and based on tongue image and related tongue diagnosis data, NAFLD diagnosis and syndrome classification can be realized, which provides an auxiliary means for doctors to diagnose the disease of patients, and provides a self-help disease prediction tool for patients, so as to help early discovery, early intervention of NAFLD and reduce the risk of complications.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data processing, and particularly relates to a tongue diagnosis multi-modal data classification method, system and storage medium. BACKGROUND

[0002] Non-alcoholic fatty liver disease (NAFLD) refers to a liver showing fatty degeneration. According to an imageology examination, if the liver fat content is greater than 5% or the histological examination can see that the lipid droplet cell content is greater than 5%, it can be judged as NAFLD. Ultrasonic examination is a first-line diagnostic technique for non-alcoholic fatty liver patients, which can effectively diagnose moderate or severe fatty degeneration, but this operable examination method is highly dependent and has limited sensitivity. When the fatty degeneration is less than 20% or the body mass index is greater than 40 kg / m 2 2, the accuracy of ultrasonic examination in detecting liver fatty degeneration will decrease. In addition, in large-scale screening, the implementation capacity and cost of ultrasonic examination significantly affect its feasibility. Biochemical indicators, physiological parameters and liver fatty degeneration index have been reported to be used for the diagnosis and prediction of NAFLD, however, there is a problem of low diagnostic accuracy and specificity.

[0003] Traditional Chinese medicine tongue diagnosis is a simple and intuitive method to obtain useful clinical information, and digital image processing technology has been widely applied in tongue diagnosis image analysis. However, the clinical data of traditional Chinese medicine tongue diagnosis includes tongue image, inquiry, pulse diagnosis and other four diagnosis data, and part of the physiological data of western medicine examination, and the intelligent model of single modal data processing cannot be applied to the multi-modal data of tongue diagnosis. SUMMARY

[0004] To solve the above problems, the first aspect of the application provides a tongue diagnosis multi-modal data classification method, comprising the following steps:

[0005] S1, acquiring diagnosis image data and diagnosis text data, and extracting data features to obtain image data features and text data features;

[0006] S2, fusing the image data features and the text data features to obtain multi-modal data fusion features;

[0007] and,

[0008] S3, inputting the fusion features into a classification model to obtain diagnosis prediction or symptom classification results.

[0009] The diagnosis model designed by the above scheme can realize multi-modal diagnosis data feature fusion and classification, which is helpful for the full use and information mining of clinical diagnosis data, and improves the accuracy of diagnosis.

[0010] Further, the image data is tongue image data, the diagnosis prediction result is to judge whether it is a sample of NAFLD disease, and the syndrome classification includes phlegm and blood stasis, internal retention of turbid dampness, internal retention of damp-heat and liver stagnation and spleen deficiency, so as to realize the diagnosis and syndrome classification of NAFLD.

[0011] Preferably, S2 comprises introducing a random attention mechanism to fuse global features and local features of image data features and / or text data features. The random attention mechanism adopts an adaptive fusion mode considering global and local features, strengthens the recognition of local and global features, improves the feature learning ability of the model for specific syndromes, thereby facilitating the improvement of the disease syndrome diagnosis and classification ability of the model; at the same time, the random attention mechanism optimizes the attention weight by back propagation iteration according to the error between the prediction value and the true label, avoiding the interaction problem between features such as self-attention mechanism, which can greatly reduce the model training time.

[0012] Preferably, a joint modal feature fusion architecture is used in S2 for multi-modal data feature fusion; further, vector product operation is used to combine image data features and text data features. The joint modal feature fusion architecture can realize effective fusion of multi-modal features, fully retain the semantic relationship between modal data, facilitate the transformation between different modalities, and also facilitate the correlation analysis between features of different modalities and the downstream classification or regression task.

[0013] Preferably, the classification model in S3 is a Trans-CNN classifier, which includes a feature vector encoding layer Transformer module and a classifier; further, the front end of the Transformer module includes a feature block tiling stretching linear mapping module, which is used to convert the fusion features into one-dimensional sequence features.

[0014] Preferably, S2 further comprises introducing a CAM attention mechanism for visual explanation of features, which is conducive to enhancing the understanding of the model output results and the judgment of the syndrome type by the physician.

[0015] The second aspect of the present application proposes a tongue diagnosis multi-modal data classification system, comprising:

[0016] A feature extraction module configured to obtain diagnostic image data and diagnostic text data, and extract data features respectively to obtain image data features and text data features;

[0017] A feature fusion module configured to fuse image data features and text data features to obtain multi-modal data fusion features;

[0018] and,

[0019] A feature classification module configured to input the fusion features into a classification model to obtain a diagnosis prediction or a syndrome classification result.

[0020] The third aspect of the present application provides a computer readable storage medium for tongue diagnosis multi-modal data classification, which stores one or more computer programs, and the one or more computer programs are executed by a computer processor to implement the method of any one of the above.

[0021] The present application proposes a model with good applicability and expansibility around the early auxiliary diagnosis and treatment needs of NAFLD disease, based on tongue image and related tongue diagnosis data, which can realize NAFLD diagnosis and syndrome classification, provide an auxiliary means for doctors to diagnose the disease of patients, and provide a self-help tool for patients to predict diseases, so as to help early detection, early intervention of NAFLD and reduce the risk of complications. BRIEF DESCRIPTION OF DRAWINGS

[0022] The accompanying drawings help further understand the present application. For ease of description, only parts related to the present application are shown in the drawings.

[0023] Figure 1 It is a tongue diagnosis multi-modal data classification process schematic diagram in an embodiment;

[0024] Figure 2 It is a logical framework schematic diagram of TongueNFLT-MultiMD model in another embodiment;

[0025] Figure 3 It is a feature processing flow schematic diagram of random attention mechanism in another embodiment;

[0026] Figure 4 It is a feature fusion architecture schematic diagram of joint modal in another embodiment;

[0027] Figure 5 It is a logical structure diagram of Trans-CNN Classifier in another embodiment;

[0028] Figure 6 It is a tongue diagnosis multi-modal data classification system structure schematic diagram in another embodiment;

[0029] Figure 7 It is a computer system structure schematic diagram of an electronic device suitable for implementing the embodiments of the present application in another embodiment. DETAILED DESCRIPTION

[0030] The present application will be further described in detail below in conjunction with the drawings and embodiments. The specific embodiments described herein are only used to explain the related application, and not to limit the application. The embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0031] Figure 1Fig. 1 is a schematic diagram of a tongue diagnosis multi-modal data classification process according to an embodiment, which specifically includes the following steps:

[0032] S1, obtaining diagnostic image data and diagnostic text data, and extracting data features to obtain image data features and text data features;

[0033] S2, fusing the image data features and the text data features to obtain multi-modal data fusion features;

[0034] and,

[0035] S3, inputting the fusion features into a classification model to obtain a diagnosis prediction or a symptom classification result.

[0036] In specific embodiments, the image data and the text data of tongue diagnosis are analyzed to obtain a diagnosis and a symptom classification result of NAFLD, wherein the NAFLD symptoms include phlegm and blood stasis, internal retention of dampness, internal retention of dampness and heat, and liver stagnation and spleen deficiency.

[0037] In specific embodiments, a prediction model TongueNFLT-MultiMD based on tongue image, relevant four diagnostic data and partial biochemical index fusion features is proposed, Figure 2 Fig. 2 is a logic framework of the TongueNFLT-MultiMD model according to an embodiment, which specifically includes the following steps:

[0038] Obtaining tongue diagnosis image data and diagnostic text data, and extracting data features to obtain image data features and text data features. For tongue image Tongue Images, classical image feature extraction algorithms such as Faster-CNN and Mask-CNN can be used for image data feature extraction; for text data Indexes Text, such as four diagnostic data and detection index data, common text processing methods such as Word2vec and FastText can be used for feature extraction;

[0039] Fusing the image data features and the text data features to obtain multi-modal data fusion features CombinedFeatures;

[0040] Inputting the fusion features into a classification model to obtain a diagnosis prediction or a symptom classification result. Depending on the nature of the task, different deep learning classification or regression models can be selected to perform feature classification.

[0041] It can be seen that the TongueNFLT-MultiMD model has good expansibility and can also provide a reference for the research of other types of disease diagnosis methods.

[0042] In a preferred embodiment, a stochastic attention mechanism is introduced during the data feature extraction process. Typically, attention mechanisms or self-attention mechanisms focus on the data itself, thus relying heavily on the data's own features and affecting the model's generalization ability. The stochastic attention mechanism, however, iteratively optimizes the attention weights through backpropagation based on the error between the predicted value and the true label, avoiding the feature interaction problems encountered in self-attention mechanisms and significantly reducing model training time. Figure 3 This is a schematic diagram of the feature processing flow of the random attention mechanism in this embodiment. For example... Figure 3 As shown, in the stochastic attention mechanism module, two learnable matrices are randomly initialized. (Where n is the dimension of the input feature matrix), a random matrix R is generated by multiplying R1 and R2. Then, the attention weights of matrix R are calculated using the Softmax function, resulting in a matrix Ri where the sum of all attention weights is 1. weight Matrix R weight Multiplying the input feature matrix A by its corresponding elements yields the feature matrix A. R-att Feature A before entering the stochastic attention mechanism module has good global characteristics, and feature A after processing by the stochastic attention mechanism module... R-att Local features have been added, which is beneficial for subsequent feature analysis operations.

[0043] In NAFLD syndrome classification and diagnosis, some tasks are determined by global features, while others may be influenced by minority class features. Therefore, optimizing the ratio of global features to local features is crucial for the model's prediction accuracy. To automatically optimize this ratio, this embodiment employs an adaptive feature fusion method to complete the feature fusion task for each single modality before fusing features from different modalities. First, a learnable adaptive scaling parameter λ is set, and then feature A and feature A' are fused using this parameter λ. R-att Obtain the new feature matrix The formula is expressed as follows:

[0044]

[0045] in, Defined as feature A and feature A R-att The stacking operation of two sets of feature matrices, where f(x) is a piecewise function of range independent variable, and the function value after iterative update of the scaling parameter λ is in the interval (0,1), the function f(x) is defined as follows:

[0046]

[0047] In the preferred embodiment, a joint modal feature fusion architecture is used for multi-modal data feature fusion. The multi-modal fusion architectures commonly used in the prior art generally include collaborative architecture, joint architecture and codec architecture. In the present embodiment, a joint modal feature fusion architecture is used, and its logical architecture is shown in Figure 4 As shown in the figure, each single-modal data is mapped to a shared semantic subspace through a feature semantic expression function after being encoded separately.

[0048] In the joint modal feature fusion architecture, a direct addition method can be used to combine each single-modal feature vector, thereby completing the fusion task of multi-modal data in the shared semantic subspace, and the formula expression is as follows

[0049]

[0050] Where g is a semantic space mapping function, W i is a weight coefficient, X i represents each single-modal feature, and G is the multi-modal feature fusion result.

[0051] The direct addition method is simple in principle and high in model operation efficiency, but it is easy to cause semantic information loss. Therefore, in the preferred embodiment, vector product operation is used to combine image data features and text data features,

[0052]

[0053] Where represents the outer product, i.e. the vector product of two vectors.

[0054] In the preferred embodiment, for the classification of fused features, refer to Figure 2 , a Trans-CNN classifier Trans-CNNClassifier is designed to complete the syndrome classification or disease syndrome diagnosis task. Figure 5 is the logical structure diagram of Trans-CNN Classifier in the present embodiment. Refer to Figure 5 In the structure of Trans-CNN Classifier, Transformer is used to further extract high-level features from the multi-modal fusion features. A feature block flat stretching linear mapping module is designed at the front end of Transformer, which is used to complete the conversion of the fusion feature matrix to one-dimensional sequence features; at the end of Transformer, a classifier based on convolutional network is designed, which is used to complete the non-alcoholic fatty liver disease syndrome diagnosis binary classification and four classifications of syndromes.

[0055] Specifically, the standard Transformer encoder at the bottom layer can only accept one-dimensional word vectors, so the multi-channel two-dimensional fusion feature map The extrusion is a flat two-dimensional feature block where (H, W) is the size of the original fused feature map, C is the number of channels, (P, P) is the size of each stretched flat two-dimensional feature block, and the number of stretched two-dimensional feature blocks N, i.e. the effective sequence length of input to the Transformer, is calculated as

[0056]

[0057] The fixed length D of the hidden vector is used in all layers of the Transformer, so the feature block is further mapped to a one-dimensional vector with length D using a trainable linear mapping method shown in the following formula,

[0058]

[0059] where the linear mapping output of all feature blocks is used as the embedding vector of the feature block, E token is the feature block information, E posi is the feature position information, and E posi ∈R (N+1)×D .

[0060] The feature vector encoding layer of the Trans-CNN Classifier classifier can directly refer to a standard fourteen-layer Transformer module, and a multi-classifier or binary classifier composed of a one-dimensional convolutional network, a maximum pooling layer and a nonlinear classification function is connected downstream of the Transformer encoding layer. After the feature block sequence is encoded by the Transformer encoding layer, the output hidden vector sequence T = {T cls , T1, T2, …, T n-1 , T n} is obtained, where T cls is the aggregation vector of the input feature block sequence, which is used as input data of the downstream classifier together with other hidden vector sequences T' = {T1, T2, …, T n-1 , T n}. For the hidden vector sequence T', the N-gram feature of the sequence is extracted using a one-dimensional convolutional neural network, and the i-th N-gram feature c i in T' is calculated as

[0061]

[0062] where f is an activation function, K is a convolution kernel sequence, m K is the number of convolution kernels, h T is the dimension of the Transformer hidden layer vector, and k is defined as the one-dimensional convolution window size, which is equal to the number of N-grams in the N-gram feature.

[0063] After performing all window sliding convolution operations on T′, an N-gram feature sequence C = {c1, c2, ..., c} can be generated. n-k+3}, where n is the size of the latent vector sequence T′. C is then subjected to global max pooling to generate the representation vector T of the input feature block. s and T s With aggregate vector T cls The components are spliced ​​together and then linearly transformed using a Fully Connected Network (FCN).

[0064] Finally, depending on the classification task requirements, the Softmax or Sigmoid function is used to output the category labels. The Sigmoid and Softmax functions are used respectively when performing NAFLD disease diagnosis binary classification and syndrome four-category classification tasks.

[0065] In the preferred embodiment, see Figure 2 Since deep learning model operations are black-box operations with poor interpretability, this embodiment introduces a CAM attention mechanism in the image feature extraction part for the visualization and interpretation of features. During model iterative optimization, the weights of corresponding positions in the tongue image are continuously updated, and finally, a heatmap is output based on the weight values ​​of corresponding positions in the image feature matrix, thereby obtaining the relationship between tongue image regions and NAFLD. Experimental results show that the tongue image features of NAFLD patients are mainly reflected in the lateral edges, while the model's capture of tongue image features in healthy individuals is almost irregular, exhibiting a random state. Therefore, the model proposed in this embodiment intuitively illustrates the correlation between liver diseases and the edge reflection areas of the tongue image.

[0066] In a specific embodiment, experiments were conducted on the TongueNFLT-MultiMD model, and the results showed that its NAFLD diagnostic accuracy reached 86.45%, and its four-category syndrome classification accuracy was also higher than other common classification models.

[0067] Figure 6 This is a schematic diagram of the structure of a tongue diagnosis multimodal data classification system 600 in one embodiment, which includes:

[0068] The feature extraction module 601 is configured to acquire diagnostic image data and diagnostic text data, and extract data features respectively to obtain image data features and text data features;

[0069] The feature fusion module 602 is configured to fuse image data features and text data features to obtain multimodal data fusion features;

[0070] as well as,

[0071] The feature classification module 603 is configured to input fused features into a classification model to obtain diagnostic predictions or symptom classification results.

[0072] Figure 7 A schematic diagram of the structure of a computer system 700 suitable for implementing an electronic device according to embodiments of the present application is shown. Figure 7 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0073] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 702 or programs loaded from storage section 708 into random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the system 700. The CPU 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0074] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a liquid crystal display (LCD) and speakers, etc.; a storage section 708 including a hard disk, etc.; and a communication section 707 including a network interface card such as a LAN card and a modem, etc. The communication section 707 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0075] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the computer program is executed by central processing unit (CPU) 701, it performs the functions defined in the methods of this application.

[0076] This application proposes a TongueNFLT-MultiMD model with good applicability and scalability, which can be used for the diagnosis and syndrome classification of NAFLD. Experimental results show that the TongueNFLT-MultiMD model has significant advantages over other advanced models. The construction method of the TongueNFLT-MultiMD model can also provide a reference for the research of diagnostic methods for other types of diseases.

[0077] Although the contents of this application have been specifically shown and described in conjunction with preferred embodiments, those skilled in the art should understand that any changes in form and detail made to this application without departing from the spirit and scope of this application as defined by the appended claims and without inventive effort are within the scope of protection of this application.

Claims

1. A method for classifying multimodal data from tongue diagnosis, characterized in that, The following steps are performed in the prediction model TougueNFLT-MultiMD: S1, acquire diagnostic image data and diagnostic text data, extract data features respectively, and obtain image data features and text data features; S2, fuse the image data features and the text data features to obtain multimodal data fusion features; and, S3, input the fused features into the classification model to obtain diagnostic prediction or symptom classification results; The prediction model TougueNFLT-MultiMD includes a feature extraction module, a feature fusion module, and a feature classification module connected in sequence. The feature extraction module is used to extract image data features and text data features. It extracts image data features by embedding a CAM attention mechanism in a Faster-CNN model or a Mask-CNN model, and extracts text data features using Word2vec or FastText. The feature fusion module employs an adaptive feature fusion method to perform single-modal feature fusion on text data features and image data features respectively. Then, through a joint modality feature fusion architecture, each single-modality data feature is encoded separately and mapped to a shared semantic subspace through a feature semantic expression function. Finally, the image data features and text data features are combined using vector product operations to perform multimodal feature fusion. The feature classification module adopts a Trans-CNN structure to classify the multimodal data fusion features.

2. The tongue diagnosis multimodal data classification method according to claim 1, characterized in that, The image data is tongue image data, and the diagnostic prediction result is to determine whether it is a NAFLD symptom sample. The syndrome classification includes phlegm and blood stasis, dampness and turbidity retention, damp heat retention, and liver stagnation and spleen deficiency.

3. The tongue diagnosis multimodal data classification method according to claim 1, characterized in that, S2 includes introducing a random attention mechanism to fuse global and local features of image data features and / or text data features.

4. The tongue diagnosis multimodal data classification method according to claim 1, characterized in that, S2 employs a joint modality feature fusion architecture for multimodal data feature fusion.

5. The tongue diagnosis multimodal data classification method according to claim 4, characterized in that, The image data features and the text data features are combined using vector product operations.

6. The tongue diagnosis multimodal data classification method according to claim 1, characterized in that, The classification model described in S3 is a Trans CNN classifier, which includes a Transformer module for feature vector encoding and a classifier.

7. The tongue diagnosis multimodal data classification method according to claim 6, characterized in that, The front end of the Transformer module includes a feature block tiling stretching linear mapping module, which is used to convert the fused features into one-dimensional sequence features.

8. The tongue diagnosis multimodal data classification method according to claim 2, characterized in that, The S1 image feature extraction process introduces a CAM attention mechanism for the visualization and interpretation of features.

9. A tongue diagnosis multimodal data classification system, characterized in that, The prediction model TougueNFLT-MultiMD includes a feature extraction module, a feature fusion module, and a feature classification module connected in sequence. The feature extraction module is configured to acquire diagnostic image data and diagnostic text data, and extract data features respectively to obtain image data features and text data features. Specifically, image data features are extracted by embedding a CAM attention mechanism in a Faster-CNN model or a Mask-CNN model, and text data features are extracted using Word2vec or FastText. The feature fusion module is configured to fuse the image data features and the text data features to obtain multimodal data fusion features. Specifically, the feature fusion module uses an adaptive feature fusion method to perform single-modal feature fusion on the text data features and image data features respectively. Then, it encodes each single-modal data feature separately through a joint modality feature fusion architecture and maps it to a shared semantic subspace through a feature semantic expression function. Finally, it combines the image data features and text data features using vector product operations to perform multimodal feature fusion. The feature classification module is configured to use a Trans-CNN structure to classify the fused features into the classification model to obtain diagnostic predictions or symptom classification results.

10. A computer-readable storage medium for classifying multimodal data from tongue diagnosis, having stored thereon one or more computer programs, characterized in that, When the one or more computer programs are executed by a computer processor, they perform the method according to any one of claims 1 to 8.

Images