A tongue image analysis method based on a hybrid model architecture

Abstract

The present invention discloses a tongue image analysis method based on a hybrid model architecture, which relates to the technical field of medical image analysis and includes three steps: tongue image segmentation, tongue image detection, and disease judgment. In the tongue image segmentation stage, by acquiring image and mask features, the edge feature information of the mask is extracted using the Sobel operator, thereby improving the accuracy of the model and its sensitivity to image edge details. In the tongue image detection and disease judgment stages, a multi-scale feature extraction mechanism is adopted, and multi-level feature extraction of tongue image is performed with the help of a convolutional neural network, which can capture features of different scales from local to global, ensuring the model's accurate recognition of tongue surface features. In addition, by introducing a sparse self-attention mechanism, the computational burden of the self-attention mechanism on high-resolution images is effectively alleviated, and the ability to capture global tongue image information is significantly improved while ensuring performance, so that the model can focus more accurately on important features related to the disease.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image analysis, and in particular to a tongue image analysis method based on a hybrid model architecture. Background Art

[0002] In healthcare, tongue analysis is an important clinical diagnostic tool, particularly in Traditional Chinese Medicine (TCM) diagnosis. By observing features such as the tongue coating and tongue texture, tongue analysis can help doctors assess and judge a patient's health status, thus possessing significant clinical value. However, traditional tongue analysis methods rely primarily on the doctor's experience and manually extracted image features, a reliance that often presents limitations when faced with complex tongue images and the patient's individual health status. Specifically, manually extracted image features are often overly subjective and one-sided, making it difficult to fully and deeply explore the complex correlation between tongue features and the patient's health status. Furthermore, faced with heterogeneous and diverse tongue image data, traditional methods are significantly limited in accuracy, stability, and processing power, failing to meet the demands of modern healthcare for efficient and precise diagnosis.

[0003] Therefore, leveraging advanced technologies to more comprehensively analyze tongue patterns and enhance diagnostic accuracy and objectivity has become a key research area in the medical field. This provides room for the development of tongue analysis systems based on deep learning and intelligent algorithms. These systems can overcome the limitations of traditional methods and fully explore the deep connections between tongue characteristics and health status, thereby providing more effective support for personalized diagnosis and treatment, as well as health assessments.

[0004] With the rapid development of artificial intelligence (AI) technology, deep learning has shown great potential in the field of medical image analysis. In recent years, tongue analysis methods based on deep learning have gradually attracted widespread attention. In particular, convolutional neural networks (CNNs) have achieved significant breakthroughs in medical image analysis. Many researchers have improved the analytical capabilities of medical images through innovative network structures. For example, in 2019, Zhuoling Li et al. published a paper titled "CLU-CNNs: Object Detection for Medical Images" in the journal "Neurocomputing", in which a CLU-CNN network was constructed to detect lesions in medical images, greatly improving the accuracy of image detection; in 2020, Kang Zhou et al. published a paper titled "Automated Prostate Cancer Diagnosis Based on Gleason Grading Using Convolutional Neural Network" in the journal "Computer Vision and Pattern Recognition", in which a PBIR Network was constructed, which significantly improved the accuracy of target detection by reconstructing abnormalities in medical images; in 2024, Xu Qiao et al. published a paper titled "Intelligent tongue diagnosis model for gastrointestinal diseases based on tongue images" in the journal "Biomedical Signal Processing and Control", in which a new information fusion detection method was introduced, which integrated hand-crafted and automatically encoded features, as well as disease detection using squeeze and excitation (SE) and time-slot attention mechanisms.

[0005] While existing methods have achieved remarkable results in specific tasks, most models relying on CNN architectures still have limitations. These models are insufficient in capturing the global information in tongue images, and traditional neural network structures are limited when dealing with the complex and heterogeneous characteristics of tongue images. To address these issues, more innovative hybrid model architectures and deep learning methods are needed to comprehensively capture the detailed features and global patterns of tongue images, thereby advancing the application of tongue analysis in medical imaging and providing stronger support for personalized medicine and precise diagnosis. Summary of the Invention

[0006] In order to solve the above technical problems, the present invention provides a tongue image analysis method based on a hybrid model architecture, comprising the following steps:

[0007] S1. Build a hybrid model architecture and obtain patient tongue images to form a tongue image dataset; input the patient tongue image images in the tongue image dataset into the improved backbone network for feature extraction;

[0008] S2. Input the extracted features into the encoder for encoding to obtain the encoded feature vector;

[0009] S3. Extract features from the pre-annotated mask through a convolutional layer, and combine the extracted mask image features with the encoded feature vector obtained in the previous step to obtain a combined feature vector;

[0010] S4. Input the combined feature vector into the decoder for decoding to obtain the final segmented tongue image. The decoder includes a transformer architecture decoder and a pixel decoder.

[0011] S5, scaling the segmented tongue image to 224*224 pixels while maintaining the original three color channels;

[0012] S6, inputting the scaled tongue image into the block coding layer for encoding to obtain an encoded feature vector;

[0013] S7, concatenating the pre-labeled category with the encoded feature vector obtained in the previous step to obtain a feature vector with category information;

[0014] S8, encode the position information and add it to the feature vector with category information, then perform regularization through the DropPath layer to keep the vector dimension unchanged;

[0015] S9, inputting the feature vector obtained in the previous step into the encoder for encoding to obtain an encoded feature vector;

[0016] S10, input the encoded feature vector into the decoder for decoding to obtain a decoded feature vector, the decoder is the same as the transformer architecture decoder in step S4;

[0017] S11, input the decoded feature vector into the normalization layer and extract the classification information to obtain the decoded classification vector;

[0018] S12. Input the decoded classification vector into the detection layer to obtain the rating of metabolism-related fatty liver disease corresponding to the tongue image, which includes mild fatty liver, moderate fatty liver and severe fatty liver.

[0019] The technical solution further defined in the present invention is:

[0020] Furthermore, in the improved backbone network of step S1, the patient's tongue image is respectively convolved with convolution kernels with kernel sizes of 1 / 32*scales, 1 / 16*scales, 1 / 8*scales, and 1 / 4*scales, and step sizes of 2, 1, 1 / 2, and 1 / 4, to obtain a four-layer pyramid structure image feature matrix from top to bottom, where scales represents the image pixel size.

[0021] As described above, in the tongue image analysis method based on the hybrid model architecture, step S2 specifically includes the following sub-steps:

[0022] S2.1. Divide the image feature matrix into blocks and embed the position code, which is expressed as follows:

[0023] Emb i =Linear(Patch i )

[0024] z l-1 =Emb+PE

[0025] Among them, Emb i Represents the linear layer calculation results of each small block of the image, Patch i represents the i-th non-overlapping matrix block, z l-1 It represents the result of linear layer calculation with position encoding added, and PE represents position encoding;

[0026] S2.2. The non-overlapping feature matrix with position encoding is expressed as follows through the multi-head self-attention mechanism:

[0027] Q=z l-1 W Q

[0028] K=z l-1 W K

[0029] V=z l-1 W V

[0030]

[0031] MHA(z l-1 )=concat(head1, head2,..., head n )W O

[0032] z′ l =LayerNorm(z l-1 +MHA(z l-1 ))

[0033] Among them, Q, K, and V represent the query matrix Query, the key matrix Key, and the value matrix Value, respectively. Q 、W K 、W V Represents the weight matrices of query matrix Query, key matrix Key and value matrix Value respectively, l represents the current number of self-attention layers, d k Represents the dimension of K vector, W O Represents the linear transformation weight matrix of multi-head output, z′ l represents the normalized output of layer l;

[0034] S2.3. Pass the calculation result obtained in the previous step through the feedforward neural network layer to obtain the final encoded feature vector, which is expressed as follows:

[0035] FFN(z′ l ) = ReLU(z′) l W1+b1)W2+b2

[0036] z l =LayerNorm(z′ l +FFN(z′ l ))

[0037] Among them, FFN(z′ l ) represents z′ l The output of the feedforward neural network, W1 represents the weight matrix of the first linear layer, b1 represents the bias term of the first linear layer, W2 represents the weight matrix of the second linear layer, b2 represents the bias term of the second linear layer, z l Represents the output result of the l-th layer residual connection and normalization.

[0038] As described above, in the tongue image analysis method based on the hybrid model architecture, in step S3, the Sobel operator and the convolution layer are combined to perform feature extraction on the mask, which specifically includes the following sub-steps:

[0039] S3.1. Obtain the mask image features through the convolution layer, which is expressed as follows:

[0040] Z l =Conv(P, W 11 )+b 11

[0041] I(x, y) = ReLU(BatchNorm(Z l ))

[0042] Among them, P represents the pre-labeled mask image, W 11 represents the convolution kernel weight, b 11 represents the bias term, Z lrepresents the initial feature map;

[0043] S3.2. Obtain mask edge features using the Sobel operator:

[0044]

[0045] Where I(x, y) represents the input mask; edge information in the horizontal and vertical directions is obtained by calculation;

[0046] S3.3, calculate G by the following formula x With G y Merged into edge strength G, which is used to express the overall strength of the mask at the edge:

[0047]

[0048] S3.4. Concatenate the mask image features and the mask edge features, as follows:

[0049]

[0050] Where I(x, y) represents the input mask image feature, G(x, y) represents the overall strength of the mask at the edge, and X represents the mask feature;

[0051] S3.5, mask feature X and image encoding feature z l Perform splicing to obtain a new feature vector, which is expressed as follows:

[0052] F low-level =concat(X, z l )

[0053] Among them, F low-level Indicates new comprehensive features.

[0054] As described above, in the tongue image analysis method based on the hybrid model architecture, step S4 specifically includes the following sub-steps:

[0055] S4.1. Decode the combined feature vector through the transformer architecture decoder and obtain the mask classification result through the mask loss function. The self-attention mechanism is the same as the multi-head self-attention mechanism in step S2.2 except for the output layer. In this step, the output layer of the self-attention mechanism uses the softmax function to obtain the probability distribution, which is expressed as follows:

[0056] y t_mask =softmax(z L W out +b out )

[0057] Among them, zL represents the output of the last layer decoder, W out represents the weight of the output layer, b out Represents the bias term of the output layer;

[0058] S4.2. Decode the combined feature vector through the pixel decoder, and express it as follows:

[0059]

[0060] F fused =F low-level +F upsampled

[0061] F conv =ReLU(Conv(F fused , W)+b)

[0062] y t_pixel =softmax(Conv 1*1 (F conv , W out )+b out )

[0063] Among them, F low-level (x i ,y i ) represents the value of the neighboring pixels in the comprehensive feature, w i，j represents the interpolation weight, F low-level represents the combined feature vector output in step S3, W represents the convolution kernel, b represents the bias term, Conv 1*1 Represents the convolution calculation of 1*1 convolution kernel, W out represents the output layer convolution kernel, b out Represents the output layer bias term;

[0064] S4.3. Cross-multiply the mask prediction result with the pixel prediction result to obtain the segmentation result, which is expressed as follows:

[0065] y seg =y t_mask ⊙y t_pixel

[0066] Among them, y t_mask Represents the mask prediction result, y t_pixel Represents the pixel prediction result.

[0067] As described above, in a tongue image analysis method based on a hybrid model architecture, in step S6, the scaled tongue image is input into the block coding layer for encoding to obtain the encoded feature vector, which is expressed as follows:

[0068] Z l=Conv(X, W1)+b1

[0069] I(x, y) = ReLU(BatchNorm(Z l ))

[0070] Among them, W1 represents the convolution kernel weight, b1 represents the bias term, and Z l Indicates the initial image.

[0071] In the tongue image analysis method based on the hybrid model architecture described above, in steps S7 and S8, the category feature and the position feature are fused into the feature vector to obtain a feature vector with complete information, which is expressed as follows:

[0072] z concat =Concat(z encoded ,y label )

[0073] Among them, z encoded Represents the encoded feature vector, y label Indicates the corresponding label;

[0074] The position information is encoded and added to the feature vector with category information, and then regularized through the DropPath layer to keep the vector dimension unchanged, as shown below:

[0075] z combined =z concat +z pos

[0076] z dropath =Dropath(z combined )

[0077] Among them, z pos represents the position vector, z combined Indicates the fusion of the encoded feature vector and the position vector, z dropath Represents the output of the Dropout layer.

[0078] As described above, in the tongue image analysis method based on the hybrid model architecture, step S9 specifically includes the following sub-steps:

[0079] S9.1. Divide the image feature matrix into blocks and embed the position code, which is expressed as follows:

[0080] Emb i =Linear(Patch i )

[0081] z l-1 =Emb+PE

[0082] Among them, Embi The linear layer calculation results for each small patch of the image, Patch i represents the i-th non-overlapping matrix block, and PE represents the position encoding;

[0083] S9.2. The non-overlapping feature matrix with positional encoding is expressed as follows through the sparse multi-head self-attention mechanism:

[0084] Q=z l-1 W Q

[0085] K=z l-1 W K

[0086] V=z l-1 W V

[0087]

[0088] MHA(z l-1 )=concat(head1, head2,..., head n )W O

[0089] z′ l =LayerNorm(z l-1 +MHA(z l-1 ))

[0090] Among them, Q, K, and V represent the query matrix Query, the key matrix Key, and the value matrix Value, respectively. Q 、W K 、W V Represents the weight matrices of query matrix Query, key matrix Key, and value matrix Value respectively, l represents the current number of self-attention layers, d k Represents the dimension of K vector, W O Represents the linear transformation weight matrix of multi-head output, z l-1 represents the result of the linear layer calculation with position encoding added, z′ l Represents the normalization layer output;

[0091] S9.3, z′ l The input decoder decodes the feature vector through the sparse attention layer and the feedforward layer, which is expressed as follows:

[0092] Q′=z′ l W Q′

[0093] K′=z′ l W K′

[0094] V′=z′ l W V′

[0095]

[0096] Among them, Q′, K′, and V′ represent the query matrix Query′, the key matrix Key′, and the value matrix Value′, respectively. Q′ 、W K′ 、W V′ Represents the weight matrices of query matrix Query′, key matrix Key′, and value matrix Value′, respectively. l represents the current number of self-attention layers, and d k′ represents the dimension of the K′ vector, and M is the sparse mask matrix;

[0097] S9.4. Pass the result of step S9.2 through the feedforward neural network layer to obtain the final encoded feature vector, which is expressed as follows:

[0098] FFN(z′ l ) = ReLU(z′) l W1+b1)W2+b2

[0099] z l =LayerNorm(z′ l +FFN(z′ l ))

[0100] Among them, z′ l represents the normalized layer output in step S9.2, W1 represents the weight matrix of the first linear layer, b1 represents the bias term of the first linear layer, W2 represents the weight matrix of the second linear layer, b2 represents the bias term of the second linear layer, z l Represents the normalized output.

[0101] As described above, in a tongue image analysis method based on a hybrid model architecture, in step S11, the decoded feature vector z l After inputting the normalization layer, the classification information is extracted to obtain the decoded classification vector, which specifically includes the following steps:

[0102] S11.1. Input the decoded feature vector into the normalization layer, which is expressed as follows:

[0103]

[0104] Where μ represents the eigenvector z l The mean of σ represents the eigenvector z l The standard deviation of ∈ indicates a small value to prevent division by zero, and the value is 10 -6 ;

[0105] S11.2. Input the normalized result into the activation function layer, expressed as follows:

[0106]

[0107] Among them, z represents the input vector, e z represents the exponential function of z, e -z represents the exponential function of -z;

[0108] S11.3. Extract classification information from the decoded classification vector through a linear layer, as shown below:

[0109] z class =z norm W class +b class

[0110] Among them, W class represents the weight matrix of the linear layer, b class Represents the bias vector.

[0111] In the tongue image analysis method based on the hybrid model architecture described above, in step S12, the decoded classification vector is input into the detection layer to obtain the rating of metabolic-related fatty liver disease corresponding to the tongue image, which is expressed as follows:

[0112]

[0113] Among them, z i Represents the i-th element in the input vector z.

[0114] The beneficial effects of the present invention are:

[0115] (1) The present invention can capture the global information of tongue images in a more detailed manner, and improve the understanding of the macroscopic and microscopic characteristics of tongue images through precise analysis, thereby improving the comprehensiveness and accuracy of tongue image analysis;

[0116] (2) The present invention has stronger performance capabilities when facing complex image features and can effectively process complex textures, morphological changes and heterogeneous features, making its application in disease diagnosis more reliable and accurate;

[0117] (3) In the present invention, excellent representation and learning capabilities are demonstrated for large-scale data sets. Through the optimization of the deep learning architecture, comprehensive learning and processing of complex tongue image information in a big data environment are achieved, and the adaptability and generalization ability of the model are improved. Overall, this analysis method has significant advantages in the ability to capture details and global information, and can better serve tongue image analysis and disease screening, providing more scientific and accurate support for clinical diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0118] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0119] Figure 2 A schematic diagram of a process flow of an improved backbone network in an embodiment of the present invention;

[0120] Figure 3 Schematic diagram of the process of mask feature extraction in an embodiment of the present invention;

[0121] Figure 4 Schematic diagram of the process of the decoder in the tongue image segmentation stage according to an embodiment of the present invention;

[0122] Figure 5 Schematic diagram of the process of feature vector encoding in an embodiment of the present invention;

[0123] Figure 6 2 is a flow chart of the judgment layer in the tongue image judgment stage in an embodiment of the present invention. DETAILED DESCRIPTION

[0124] This embodiment provides a tongue image analysis method based on a hybrid model architecture, which is used to analyze the patient's tongue image to determine whether the patient suffers from metabolic-related fatty liver disease. The method of this embodiment is divided into three main stages, namely the tongue image segmentation stage, the tongue image detection stage, and the disease judgment stage. Figure 1 As shown, the specific steps include:

[0125] (1) Tongue segmentation stage:

[0126] S1. Build a hybrid model architecture and obtain patient tongue images to form a tongue image dataset, which is divided into a training set and a test set in a ratio of 8:2; input the patient tongue image images in the tongue image dataset into the improved backbone network for feature extraction.

[0127] like Figure 2 As shown in the figure, in the improved backbone network, the patient's tongue image is convolved with convolution kernels of kernel sizes of 1 / 32*scales, 1 / 16*scales, 1 / 8*scales, and 1 / 4*scales, and step sizes of 2, 1, 1 / 2, and 1 / 4, respectively, to obtain a four-layer pyramid structure image feature matrix from top to bottom, where scales represents the image pixel size.

[0128] S2. Input the extracted features into the encoder for encoding to obtain the encoded feature vector, which specifically includes the following steps:

[0129] S2.1. Divide the image feature matrix into blocks and embed the position code, which is expressed as follows:

[0130] Emb i =Linear(Patchi )

[0131] z l-1 =Emb+PE

[0132] Among them, Emb i Represents the linear layer calculation results of each small block of the image, Patch i represents the i-th non-overlapping matrix block, z l-1 It represents the result of linear layer calculation with position encoding added, and PE represents position encoding.

[0133] S2.2. The non-overlapping feature matrix with position encoding is expressed as follows through the multi-head self-attention mechanism:

[0134] Q=z l-1 W Q

[0135] K=z l-1 W K

[0136] V=z l-1 W V

[0137]

[0138] MHA(z l-1 )=concat(head1, head2,..., head n )W O

[0139] z′ l =LayerNorm(z l-1 +MHA(z l-1 ))

[0140] Among them, Q, K, and V represent the query matrix Query, the key matrix Key, and the value matrix Value, respectively. Q 、W K 、W V Represents the weight matrices of query matrix Query, key matrix Key and value matrix Value respectively, l represents the current number of self-attention layers, d k Represents the dimension of K vector, W O Represents the linear transformation weight matrix of multi-head output, z′ l represents the normalized output of layer l.

[0141] S2.3. Pass the calculation result obtained in the previous step through the feedforward neural network layer to obtain the final encoded feature vector, which is expressed as follows:

[0142] FFN(z′ l) = ReLU(z′) l W1+b1)W2+b2

[0143] z l =LayerNorm(z′ l +FFN(z′ l ))

[0144] Among them, FFN(z′ l ) represents z′ l The output of the feedforward neural network, W1 represents the weight matrix of the first linear layer, b1 represents the bias term of the first linear layer, W2 represents the weight matrix of the second linear layer, b2 represents the bias term of the second linear layer, z l Represents the output result of the l-th layer residual connection and normalization.

[0145] S3, such as Figure 3 As shown in the figure, the convolution layer is used to extract features from the pre-annotated mask, and the extracted mask image features are combined with the encoded feature vector obtained in the previous step to obtain a combined feature vector; combining the Sobel operator with the convolution layer, feature extraction is performed on the mask, which specifically includes the following steps:

[0146] S3.1. Obtain the mask image features through the convolution layer, which is expressed as follows:

[0147] Z l =Conv(P, W 11 )+b 11

[0148] I(x, y) = ReLU(BatchNorm(Z l ))

[0149] Among them, P represents the pre-labeled mask image, W 11 represents the convolution kernel weight, b 11 represents the bias term, Z l Represents the initial feature map.

[0150] S3.2. Obtain mask edge features using the Sobel operator:

[0151]

[0152] Wherein, I(x, y) represents the input mask; edge information in the horizontal and vertical directions is obtained by calculation.

[0153] S3.3, calculate G by the following formula x With G y Merged into edge strength G, which is used to express the overall strength of the mask at the edge:

[0154]

[0155] S3.4. Concatenate the mask image features and the mask edge features, as follows:

[0156]

[0157] Among them, I(x, y) represents the input mask image feature, G(x, y) represents the overall intensity of the mask at the edge, and X represents the mask feature.

[0158] S3.5, mask feature X and image encoding feature z l Perform splicing to obtain a new feature vector, which is expressed as follows:

[0159] F low-level =concat(X, z l )

[0160] Among them, F low-level Indicates new comprehensive features.

[0161] S4. Input the combined feature vector into the decoder for decoding to obtain the final segmented tongue image. The decoder includes a transformer architecture decoder and a pixel decoder.

[0162] The merged features are passed through the transformer architecture decoder to determine the category of the segmented pixel group. At the same time, the decoding result of the transformer architecture decoder is cross-multiplied with the pixel decoder, and the mask probability is determined to finally obtain the segmentation result, such as Figure 4 As shown, it specifically includes the following steps:

[0163] S4.1. Decode the combined feature vector through the transformer architecture decoder and obtain the mask classification result through the mask loss function. The self-attention mechanism is the same as the multi-head self-attention mechanism in step S2.2 except for the output layer. In this step, the output layer of the self-attention mechanism uses the softmax function to obtain the probability distribution, which is expressed as follows:

[0164] y t_mask =softmax(z L W out +b out )

[0165] Among them, z L represents the output of the last layer decoder, W out represents the weight of the output layer, b out Represents the bias term of the output layer.

[0166] S4.2. Decode the combined feature vector through the pixel decoder, and express it as follows:

[0167]

[0168] F fused =F low-level +F upsampled

[0169] F conv =ReLU(Conv(F fused , W)+b)

[0170] y t_pixel =softmax(Conv 1*1 (F conv , W out )+b out )

[0171] Among them, F low-level (x i ,y i ) represents the value of the neighboring pixels in the comprehensive feature, w i，j represents the interpolation weight, F low-level represents the combined feature vector output in step S3, W represents the convolution kernel, b represents the bias term, Conv 1*1 Represents the convolution calculation of 1*1 convolution kernel, W out represents the output layer convolution kernel, b out Represents the output layer bias term.

[0172] S4.3. Cross-multiply the mask prediction result with the pixel prediction result to obtain the segmentation result, which is expressed as follows:

[0173] y seg =y t_mask ⊙y t_pixel

[0174] Among them, y t_mask Represents the mask prediction result, y t_pixel Represents the pixel prediction result.

[0175] (2) Tongue examination stage, such as Figure 5 As shown:

[0176] S5. Scale the segmented tongue image to 224*224 pixels while maintaining the original three color channels.

[0177] S6. Input the scaled tongue image into the block coding layer for encoding to obtain the encoded feature vector, which is expressed as follows:

[0178] Z l=Conv(X, W1)+b1

[0179] I(x, y) = ReLU(BatchNorm(Z l ))

[0180] Among them, W1 represents the convolution kernel weight, b1 represents the bias term, and Z l Indicates the initial image.

[0181] S7. Connect the pre-labeled category with the encoded feature vector obtained in the previous step to obtain a feature vector with category information, which is expressed as follows:

[0182] z concat =Concat(z encoded ,y label )

[0183] Among them, z encoded Represents the encoded feature vector, y label Indicates the corresponding label.

[0184] S8. Encode the position information and add it to the feature vector with category information. Then, regularize it through the DropPath layer to keep the vector dimension unchanged. It can be expressed as follows:

[0185] z combined =z concat +z pos

[0186] z dropath =Dropath(z combined )

[0187] Among them, z pos represents the position vector, z combined Indicates the fusion of the encoded feature vector and the position vector, z dropath Represents the output of the Dropout layer.

[0188] S9, inputting the feature vector obtained in the previous step into the encoder for encoding to obtain the encoded feature vector, which specifically includes the following sub-steps:

[0189] S9.1. Divide the image feature matrix into blocks and embed the position code, which is expressed as follows:

[0190] Emb i =Linear(Patch i )

[0191] z l-1 =Emb+PE

[0192] Among them, Emb iThe linear layer calculation results for each small patch of the image, Patch i represents the i-th non-overlapping matrix block, and PE represents the position encoding.

[0193] S9.2. The non-overlapping feature matrix with positional encoding is expressed as follows through the sparse multi-head self-attention mechanism:

[0194] Q=z l-1 W Q

[0195] K=z l-1 W K

[0196] V=z l-1 W V

[0197]

[0198] MHA(z l-1 )=concat(head1, head2,..., head n )W O

[0199] z′ l =LayerNorm(z l-1 +MHA(z l-1 ))

[0200] Among them, Q, K, and V represent the query matrix Query, the key matrix Key, and the value matrix Value, respectively. Q 、W K 、W V Represents the weight matrices of query matrix Query, key matrix Key, and value matrix Value respectively, l represents the current number of self-attention layers, d k Represents the dimension of K vector, W O Represents the linear transformation weight matrix of multi-head output, z l-1 represents the result of the linear layer calculation with position encoding added, z′ l Represents the normalization layer output.

[0201] S9.3, z′ l The input decoder decodes the feature vector through the sparse attention layer and the feedforward layer, which is expressed as follows:

[0202] Q′=z′ l W Q′

[0203] K′=z′ l W K′

[0204] V′=z′ l W V′

[0205]

[0206] Among them, Q′, K′, and V′ represent the query matrix Query′, the key matrix Key′, and the value matrix Value′, respectively. Q′ 、W K′ 、W V′ Represents the weight matrices of query matrix Query′, key matrix Key′, and value matrix Value′, respectively. l represents the current number of self-attention layers, and d k′ represents the dimension of the K′ vector, and M is the sparse mask matrix.

[0207] The sparse mask matrix M is composed of a diagonal matrix and is expressed as follows:

[0208]

[0209] S9.4. Pass the result of step S9.2 through the feedforward neural network layer to obtain the final encoded feature vector, which is expressed as follows:

[0210] FFN(z′ l ) = ReLU(z′) l W1+b1)W2+b2

[0211] z l =LayerNorm(z′ l +FFN(z′ l ))

[0212] Among them, z′ l represents the normalized layer output in step S9.2, W1 represents the weight matrix of the first linear layer, b1 represents the bias term of the first linear layer, W2 represents the weight matrix of the second linear layer, b2 represents the bias term of the second linear layer, z l Represents the normalized output.

[0213] S10. Input the encoded feature vector into the decoder for decoding to obtain a decoded feature vector. The decoder is the same as the transformer architecture decoder in step S4.

[0214] (3) Tongue image judgment stage:

[0215] S11, the decoded feature vector z l After inputting the normalization layer, the classification information is extracted to obtain the decoded classification vector, such as Figure 6 As shown, it specifically includes the following steps:

[0216] S11.1. Input the decoded feature vector into the normalization layer, which is expressed as follows:

[0217]

[0218] Where μ represents the eigenvector z l The mean of σ represents the eigenvector z l The standard deviation of ∈ indicates a small value to prevent division by zero, and the value is 10 -6 .

[0219] S11.2. Input the normalized result into the activation function layer, expressed as follows:

[0220]

[0221] Among them, z represents the input vector, e z represents the exponential function of z, e -z Represents the exponential function of -z.

[0222] S11.3. Extract classification information from the decoded classification vector through a linear layer, as shown below:

[0223] z class =z norm W class +b class

[0224] Among them, W class represents the weight matrix of the linear layer, b class Represents the bias vector.

[0225] S12. Input the decoded classification vector into the detection layer to obtain the rating of metabolic-related fatty liver disease corresponding to the tongue image, which is expressed as follows:

[0226]

[0227] Among them, z i Represents the i-th element in the input vector z, and the ratings include mild fatty liver, moderate fatty liver, and severe fatty liver.

[0228] In this embodiment, the data from the physical examination center of Changhai Hospital is used as a data set to compare this embodiment with U-Net, DeepLabV3+, and ResNet-50. The results are shown in Table 1 below. As can be seen from Table 1, this embodiment performs well in the tongue segmentation task. Key indicators such as the Dice coefficient and IoU are better than other comparison models, fully demonstrating its powerful ability in processing fine-grained medical image tasks; this shows that this embodiment can more accurately capture and segment the complex details in the tongue image, thereby improving the overall segmentation effect.

[0229] At the same time, DeepLabV3+ also performed well in segmentation performance, with results close to those of this embodiment, demonstrating its significant advantages in capturing tongue edges and local details. However, ResNet-50 performed relatively poorly in segmentation tasks, which may be due to its main optimization for classification tasks and its limitations in fine-grained image segmentation. Therefore, this embodiment, by integrating multi-scale feature extraction and deep learning technology, is particularly suitable for high-precision medical image tasks such as tongue segmentation, showing stronger segmentation effects and detail capture capabilities, providing a solid foundation for subsequent disease analysis and diagnosis.

[0230] Table 1

[0231]

[0232] As shown in Table 2 below, this embodiment performs excellently in the disease judgment task. Its key indicators such as accuracy, F1 score, and AUC significantly surpass those of other comparison models, fully demonstrating its outstanding effect in combining tongue image features with disease judgment. In comparison, ResNet-50 performs relatively well in the classification task, but its overall performance is still slightly inferior to that of this embodiment. This shows that although ResNet-50 has good classification capabilities, it still has shortcomings when it comes to the specific analysis of the association between tongue images and diseases.

[0233] U-Net and DeepLabV3+ performed well in segmentation tasks, especially in capturing image boundaries and details. However, since they are not optimized for disease classification, their performance in classification tasks is relatively poor. These results show that by integrating multi-scale feature extraction and deep learning technology, this embodiment is more suitable for disease judgment tasks and can more comprehensively utilize tongue image information for accurate classification, providing stronger support and guarantee for clinical diagnosis.

[0234] Table 2

[0235]

[0236] To address the complexity challenges in the field of tongue image analysis, this embodiment proposes a tongue image analysis method based on a hybrid model architecture. This method combines the advantages of convolutional neural networks (CNNs) in local feature extraction with the sensitivity of visual transformers (ViTs) in capturing global information, forming a more efficient tongue image processing architecture.

[0237] Specifically, CNN uses layer-by-layer convolution operations to deeply explore detailed features in tongue images, such as color, texture, and cracks, enabling the model to identify subtle changes and local differences. At the same time, ViT's global attention mechanism can effectively capture the macrostructure and overall pattern in tongue images, making up for the shortcomings of traditional models in processing global information. Through this deep learning strategy that combines local and global aspects, the hybrid model can automatically learn complex disease-related features and improve the ability to recognize and analyze personalized characteristics of different patients. The method of this embodiment not only significantly improves the accuracy and reliability of tongue image analysis, but also provides more accurate support for clinical diagnosis, and is expected to play an important role in applications such as medical imaging analysis, health assessment, and early disease screening.

[0238] In addition to the above embodiments, the present invention may also have other implementations. Any technical solution formed by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A tongue image analysis method based on a hybrid model architecture, characterized by: The following steps are involved: S1. Build a hybrid model architecture and obtain patient tongue images to form a tongue image dataset; input the patient tongue image images in the tongue image dataset into the improved backbone network for feature extraction; S2. Input the extracted features into the encoder for encoding to obtain the encoded feature vector; S3. Extract features from the pre-annotated mask through a convolutional layer, and combine the extracted mask image features with the encoded feature vector obtained in the previous step to obtain a combined feature vector; S4. Input the combined feature vector into the decoder for decoding to obtain the final segmented tongue image. The decoder includes a transformer architecture decoder and a pixel decoder. S5, scaling the segmented tongue image to 224*224 pixels while maintaining the original three color channels; S6, inputting the scaled tongue image into the block coding layer for encoding to obtain an encoded feature vector; S7, concatenating the pre-labeled category with the encoded feature vector obtained in the previous step to obtain a feature vector with category information; S8, encode the position information and add it to the feature vector with category information, then perform regularization through the DropPath layer to keep the vector dimension unchanged; S9, inputting the feature vector obtained in the previous step into the encoder for encoding to obtain an encoded feature vector; S10, input the encoded feature vector into the decoder for decoding to obtain a decoded feature vector, the decoder is the same as the transformer architecture decoder in step S4; S11, input the decoded feature vector into the normalization layer and extract the classification information to obtain the decoded classification vector; S12. Input the decoded classification vector into the detection layer to obtain the rating of metabolism-related fatty liver disease corresponding to the tongue image, which includes mild fatty liver, moderate fatty liver and severe fatty liver.

2. The tongue image analysis method based on a hybrid model architecture according to claim 1, characterized in that: In the improved backbone network of step S1, the patient's tongue image is respectively convolved with convolution kernels with kernel sizes of 1 / 32*scales, 1 / 16*scales, 1 / 8*scales, and 1 / 4*scales, and step sizes of 2, 1, 1 / 2, and 1 / 4, to obtain a four-layer pyramid structure image feature matrix from top to bottom, where scales represents the image pixel size.

3. The tongue image analysis method based on a hybrid model architecture according to claim 2, characterized in that: The step S2 specifically includes the following sub-steps: S2.

1. Divide the image feature matrix into blocks and embed the position code, which is expressed as follows: Emb i =Linear(Patch i ) z l-1 =Pack+PE Among them, Emb i Represents the linear layer calculation results of each small block of the image, Patch i represents the i-th non-overlapping matrix block, z l-1 It represents the result of linear layer calculation with position encoding added, and PE represents position encoding; S2.

2. The non-overlapping feature matrix with position encoding is expressed as follows through the multi-head self-attention mechanism: Q=z l-1 W Q K=z l-1 IN K V=z l-1 W V MHA(z l-1 )=concat(head1,head2,...,head n )W O With' l =LayerNorm(z l-1 +MHA(z l-1 )) Among them, Q, K, and V represent the query matrix Query, the key matrix Key, and the value matrix Value, respectively. Q 、W K 、W V Represents the weight matrices of query matrix Query, key matrix Key and value matrix Value respectively, l represents the current number of self-attention layers, d k Represents the dimension of K vector, W O Represents the linear transformation weight matrix of multi-head output, z′ l represents the normalized output of layer l; S2.

3. Pass the calculation result obtained in the previous step through the feedforward neural network layer to obtain the final encoded feature vector, which is expressed as follows: <h2 style=";text-align:left;direction:ltr">FFN(z′<h2 style=";text-align:left;direction:ltr"> l <h2 style=";text-align:left;direction:ltr"> )=ReLU(z′<h2 style=";text-align:left;direction:ltr"> l <h2 style=";text-align:left;direction:ltr"> W1+b1)W2+b2 z l =LayerNorm(z′ l +FFN(z′ l )) Among them, FFN(z′ l ) represents z′ l The output of the feedforward neural network, W1 represents the weight matrix of the first linear layer, b1 represents the bias term of the first linear layer, W2 represents the weight matrix of the second linear layer, b2 represents the bias term of the second linear layer, z l Represents the output result of the l-th layer residual connection and normalization.

4. The tongue image analysis method based on a hybrid model architecture according to claim 3, characterized in that: In step S3, the Sobel operator and the convolution layer are combined to perform feature extraction on the mask, which specifically includes the following sub-steps: S3.

1. Obtain the mask image features through the convolution layer, which is expressed as follows: Z l =Conv(P,W 11 )+b 11 I(x,y)=ReLU(BatchNorm(Z l )) Among them, P represents the pre-labeled mask image, W 11 represents the convolution kernel weight, b 11 represents the bias term, Z l represents the initial feature map; S3.

2. Obtain mask edge features using the Sobel operator: Where I(x, y) represents the input mask; edge information in the horizontal and vertical directions is obtained by calculation; S3.3, calculate G by the following formula x With G y Merged into edge strength G, which is used to express the overall strength of the mask at the edge: S3.

4. Concatenate the mask image features and the mask edge features, as follows: Where I(x, y) represents the input mask image feature, G(x, y) represents the overall strength of the mask at the edge, and X represents the mask feature; S3.5, mask feature X and image encoding feature z l Perform splicing to obtain a new feature vector, which is expressed as follows: F low-level =concat(X,z l ) Among them, F low-level Indicates new comprehensive features.

5. The tongue image analysis method based on a hybrid model architecture according to claim 4, characterized in that: The step S4 specifically includes the following sub-steps: S4.

1. Decode the combined feature vector through the transformer architecture decoder and obtain the mask classification result through the mask loss function. The self-attention mechanism is the same as the multi-head self-attention mechanism in step S2.2 except for the output layer. In this step, the output layer of the self-attention mechanism uses the softmax function to obtain the probability distribution, which is expressed as follows: y t_mask =softmax(z L W out +b out ) Among them, z L represents the output of the last layer decoder, W out represents the weight of the output layer, b out Represents the bias term of the output layer; S4.

2. Decode the combined feature vector through the pixel decoder, and express it as follows: F fused =F low-level +F upsampled F conv =ReLU(Conv(F fused ,W)+b) y t_pixel =softmax(Conv 1*1 (F conv ,W out )+b out ) Among them, F low-level (x i ,y i ) represents the value of the neighboring pixels in the comprehensive feature, w i,j represents the interpolation weight, F low-level represents the combined feature vector output in step S3, W represents the convolution kernel, b represents the bias term, Conv 1*1 Represents the convolution calculation of 1*1 convolution kernel, W out represents the output layer convolution kernel, b out Represents the output layer bias term; S4.

3. Cross-multiply the mask prediction result with the pixel prediction result to obtain the segmentation result, which is expressed as follows: and seg =and t_mask ⊙and t_pixel Among them, y t_mask Represents the mask prediction result, y t_pixel Represents the pixel prediction result.

6. The tongue image analysis method based on a hybrid model architecture according to claim 5, characterized in that: In step S6, the scaled tongue image is input into the block coding layer for encoding to obtain the encoded feature vector, which is expressed as follows: Z l =Conv(X,W1)+b1 I(x,y)=ReLU(BatchNorm(Z l )) Among them, W1 represents the convolution kernel weight, b1 represents the bias term, and Z l Indicates the initial image.

7. The tongue image analysis method based on a hybrid model architecture according to claim 6, characterized in that: In steps S7 and S8, the category features and the position features are fused into the feature vector to obtain a feature vector with complete information, which is expressed as follows: from concat =Concat(from encoded ,y label ) Among them, z encoded Represents the encoded feature vector, y label Indicates the corresponding label; The position information is encoded and added to the feature vector with category information, and then regularized through the DropPath layer to keep the vector dimension unchanged, as shown below: With combined =Z concat +Z pos With dropath =Dropath(z combined ) Among them, z pos represents the position vector, z combined Indicates the fusion of the encoded feature vector and the position vector, z dropath Represents the output of the Dropout layer.

8. The tongue image analysis method based on a hybrid model architecture according to claim 7, characterized in that: The step S9 specifically includes the following sub-steps: S9.

1. Divide the image feature matrix into blocks and embed the position code, which is expressed as follows: Emb i =Linear(Patch i ) z l-1 =Pack+PE Among them, Emb i The linear layer calculation results for each small patch of the image, Patch i represents the i-th non-overlapping matrix block, and PE represents the position encoding; S9.

2. The non-overlapping feature matrix with positional encoding is expressed as follows through the sparse multi-head self-attention mechanism: Q=z l-1 W Q K=z l-1 IN K V=z l-1 W V MHA(z l-1 )=concat(head1,head2,...,head n )W O With' l =LayerNorm(z l-1 +MHA(z l-1 )) Among them, Q, K, and V represent the query matrix Query, the key matrix Key, and the value matrix Value, respectively. Q 、W K 、W V Represents the weight matrices of query matrix Query, key matrix Key, and value matrix Value respectively, l represents the current number of self-attention layers, d k Represents the dimension of K vector, W O Represents the linear transformation weight matrix of multi-head output, z l-1 represents the result of the linear layer calculation with position encoding added, z′ l Represents the normalization layer output; S9.3, z′ l The input decoder decodes the feature vector through the sparse attention layer and the feedforward layer, which is expressed as follows: Q′=z′ l W Q′ K′=z′ l IN K′ V′=z′ l W V′ Among them, Q′, K′, and V′ represent the query matrix Query′, the key matrix Key′, and the value matrix Value′, respectively. Q′ 、W K′ 、W V′ Represents the weight matrices of query matrix Query′, key matrix Key′, and value matrix Value′, respectively. l represents the current number of self-attention layers, and d k′ represents the dimension of the K′ vector, and M is the sparse mask matrix; S9.

4. Pass the result of step S9.2 through the feedforward neural network layer to obtain the final encoded feature vector, which is expressed as follows: <h2 style=";text-align:left;direction:ltr">FFN(z′<h2 style=";text-align:left;direction:ltr"> l <h2 style=";text-align:left;direction:ltr"> )=ReLU(z′<h2 style=";text-align:left;direction:ltr"> l <h2 style=";text-align:left;direction:ltr"> W1+b1)W2+b2 z l =LayerNorm(z′ l +FFN(z′ l )) Among them, z′ l represents the normalized layer output in step S9.2, W1 represents the weight matrix of the first linear layer, b1 represents the bias term of the first linear layer, W2 represents the weight matrix of the second linear layer, b2 represents the bias term of the second linear layer, z l Represents the normalized output.

9. The tongue image analysis method based on a hybrid model architecture according to claim 8, characterized in that: In step S11, the decoded feature vector z l After inputting the normalization layer, the classification information is extracted to obtain the decoded classification vector, which specifically includes the following steps: S11.

1. Input the decoded feature vector into the normalization layer, which is expressed as follows: Where μ represents the eigenvector z l The mean of σ represents the eigenvector z l The standard deviation of ∈ indicates a small value to prevent division by zero, and the value is 10 -6 ; S11.

2. Input the normalized result into the activation function layer, expressed as follows: Among them, z represents the input vector, e z represents the exponential function of z, e -z represents the exponential function of -z; S11.

3. Extract classification information from the decoded classification vector through a linear layer, as shown below: With class =z nnorm IN class +b class Among them, W class represents the weight matrix of the linear layer, b class Represents the bias vector.

10. The tongue image analysis method based on a hybrid model architecture according to claim 9, characterized in that: In step S12, the decoded classification vector is input into the detection layer to obtain the rating of metabolic-related fatty liver disease corresponding to the tongue image, which is expressed as follows: Among them, z i Represents the i-th element in the input vector z.

Images