Traditional Chinese medicine tongue coating image analysis method based on deep learning

Abstract

The invention discloses a traditional Chinese medicine tongue coating image analysis method based on deep learning, and the method employs an innovative region-texture dual-channel fusion network which comprises a parallel region space attention channel and a multi-scale texture enhancement channel. The ZSAC simulates the idea of'partitioning and fixing viscera 'in traditional Chinese medicine, and focuses the model on a key diagnosis area by using a space attention mechanism; the MTEC adopts multi-expansion-rate cavity convolution to specially capture fine texture features such as thickness, greasy rot and the like of the coated tongue. And after the features output by the two channels are fused, synchronous and accurate analysis of multiple attributes such as the coating color and the coating quality is realized through a multi-task learning framework. According to the method, traditional Chinese medicine priori knowledge and a deep learning model are deeply fused, and the accuracy, robustness and interpretability of analysis are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence and computer vision technology, and specifically relates to a method for medical image processing using deep learning, especially an automated and high-precision analysis method for tongue coating images in traditional Chinese medicine tongue diagnosis. Background Technology

[0002] Tongue diagnosis is an important component of the four diagnostic methods in Traditional Chinese Medicine: observation, auscultation and olfaction, inquiry, and palpation. By observing the shape, color, moisture, and dryness of the tongue (including the tongue body and coating), one can understand the physiological functions and pathological changes of the body's internal organs, qi, blood, and body fluids. Among these, the "texture" (thickness, moisture, greasiness, etc.) and "color" (white, yellow, gray, black, etc.) of the tongue coating are key bases for syndrome differentiation and treatment.

[0003] Traditional tongue diagnosis relies on the visual observation and personal experience of Chinese medicine practitioners, which has the following problems: Highly subjective: Different doctors may have different judgments on the same tongue appearance, lacking a unified, quantifiable, and objective standard.

[0004] The color of the tongue coating is greatly affected by the environment: environmental factors such as lighting conditions and color temperature can seriously affect a doctor's judgment of the color of the tongue coating.

[0005] High dependence on experience: Improving tongue diagnosis skills requires long-term clinical practice and accumulation, resulting in high training costs and a scarcity of excellent TCM doctors.

[0006] With the development of computer technology, some studies on the objectification of TCM tongue diagnosis based on image processing have emerged. Early methods mainly employed traditional image processing techniques, such as color space conversion (e.g., from RGB to HSI), edge detection, and texture feature extraction (e.g., Gray-Level Co-occurrence Matrix (GLCM), Local Binary Pattern (LBP), etc.), combined with machine learning classifiers such as Support Vector Machines (SVM) and decision trees for analysis. These methods improved objectivity to some extent, but they have the following drawbacks: Complex feature design: Hand-crafted features are difficult to fully capture the complex texture, color and spatial distribution information of the tongue coating.

[0007] Poor robustness: It is very sensitive to changes in lighting, shooting angle, and interference from non-tongue areas within the oral cavity.

[0008] Process segmentation: It usually adopts a multi-stage process of "image segmentation - feature extraction - classification". Errors in the previous stage will accumulate and affect the accuracy of the final result.

[0009] In recent years, deep learning, especially convolutional neural networks (CNNs), has achieved great success in the field of image recognition. Some research has begun to attempt to apply existing general-purpose models such as ResNet, VGGNet, and U-Net to tongue image analysis. However, these existing techniques still have shortcomings: The general-purpose model is disconnected from traditional Chinese medicine theory: The general-purpose CNN model treats tongue images as ordinary pictures and cannot effectively combine the prior knowledge of "regional identification of internal organs" in traditional Chinese medicine tongue diagnosis, that is, different regions of the tongue, such as the tip, middle, and root, correspond to different organ diseases. The model lacks the ability to pay attention to the characteristics of these specific regions.

[0010] Coupling of texture and color features: Existing models typically extract all features such as color, shape, and texture simultaneously in a single backbone network, making it difficult to specifically and precisely capture features of tongue coating such as "greasy coating" and "rotten coating," which are mainly characterized by fine texture. For example, "thin white coating" and "thin white greasy coating" are similar in color, but the "greasy" texture is a unique texture representation that can easily be confused by general models.

[0011] The black box problem: The model's decision-making process is not transparent, and the basis for its judgment cannot be explained, making it difficult for TCM practitioners to trust and accept it. Summary of the Invention

[0012] The purpose of this section is to outline some aspects of embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0013] The purpose of this invention is to overcome the aforementioned shortcomings of existing technologies and provide a novel, deep learning-based method for analyzing tongue coating images in Traditional Chinese Medicine. This method aims to: This addresses the disconnect between general deep learning models and the TCM regional diagnosis and treatment theory, enabling the model to focus on specific tongue areas that have diagnostic significance.

[0014] To address the issue of mutual interference between tongue coating color features and fine texture features during the extraction process, a dedicated channel is used to decouple and extract the two features separately.

[0015] To construct an end-to-end, high-precision, robust, and interpretable tongue coating image analysis model, thereby achieving automated, standardized, and precise analysis of tongue coating color and texture.

[0016] To address the aforementioned technical problems, this invention provides the following technical solution: a deep learning-based method for analyzing tongue coating images in Traditional Chinese Medicine, comprising the following steps: a. acquiring the tongue coating image to be analyzed and preprocessing it; b. inputting the preprocessed tongue coating image into a region-texture dual-channel fusion network model for feature extraction, wherein the fusion network model includes a shared backbone network and a region spatial attention channel and a multi-scale texture enhancement channel connected in parallel thereafter; c. fusing the region spatial features output by the region spatial attention channel with the texture features output by the multi-scale texture enhancement channel to obtain fused features; d. classifying or analyzing at least one physiological or pathological attribute of the tongue coating based on the fused features and outputting the results.

[0017] As a preferred embodiment of the deep learning-based TCM tongue coating image analysis method described in this invention, step b, specifically implements the regional spatial attention channel as follows: based on the TCM tongue diagnosis zoning theory, at least two diagnostic region masks are preset in the feature map space of the tongue coating image; for each diagnostic region, an attention sub-network is used to learn the spatial attention weight map within that region; the spatial attention weight map is used to weight the basic feature map output from the shared backbone network to enhance the feature expression of regions with diagnostic significance; and all attention-weighted regional features are aggregated to form the regional spatial features.

[0018] As a preferred embodiment of the deep learning-based TCM tongue coating image analysis method described in this invention, step b, specifically implements the multi-scale texture enhancement channel by: employing a set of parallel dilated convolutional layers with at least two different dilation rates; inputting the basic feature maps output from the shared backbone network into the parallel dilated convolutional layers respectively to capture tongue coating texture information at different scales; and splicing and fusing the output feature maps of the parallel dilated convolutional layers to form the texture features.

[0019] As a preferred embodiment of the deep learning-based TCM tongue coating image analysis method described in this invention, the feature fusion method in step c is to splice the regional spatial features and the texture features in the channel dimension.

[0020] As a preferred embodiment of the deep learning-based TCM tongue coating image analysis method described in this invention, the physiological or pathological attributes of the tongue coating in step d include at least one of coating color, coating thickness, and coating texture; the classification or analysis adopts a multi-task learning architecture, and simultaneously predicts multiple attributes.

[0021] As a preferred embodiment of the deep learning-based TCM tongue coating image analysis method described in this invention, the method further includes visualizing the spatial attention weight map and overlaying it onto the original tongue coating image to indicate the key areas for model decision-making.

[0022] As a preferred embodiment of the deep learning-based TCM tongue image analysis method described in this invention, the preprocessing in step a includes color correction, tongue segmentation, and size normalization.

[0023] Compared with the prior art, the present invention has the following significant advantages: 1. Integration of Traditional Chinese Medicine Theory: The "Regional Spatial Attention Channel (ZSAC)" integrates the prior knowledge of "regional division to determine organs" in Traditional Chinese Medicine into the deep learning model, enabling the model to adaptively focus on key diagnostic regions, significantly improving the accuracy of identification for specific pathological features, and solving the problem of the disconnect between general models and Traditional Chinese Medicine theory.

[0024] 2. Feature Decoupling and Specific Enhancement: The "Multi-Scale Texture Enhancement Channel (MTEC)" is specifically designed to extract the fine texture features of the tongue coating, decoupling them from the color and spatial distribution features that ZSAC primarily focuses on. This specific enhancement design makes the model's ability to recognize subtle texture changes such as "greasy," "rotten," and "dry" far exceed that of traditional CNN models.

[0025] 3. Higher analysis accuracy and robustness: ZT-FusionNet integrates spatial location importance and texture detail information through dual-channel fusion, forming a more comprehensive and discriminative feature representation. The end-to-end design avoids error accumulation in segmented methods, and combined with standardized preprocessing workflows, it significantly improves overall analysis accuracy and robustness to environmental changes.

[0026] 4. Enhanced interpretability: By visualizing the attention map generated by ZSAC, the image regions on which the model bases its decisions can be clearly shown. This "what you see is what you get" visualization greatly enhances the trust of TCM practitioners in the model's results, facilitating clinical application and promotion.

[0027] 5. Powerful multi-task processing capability: The model architecture supports multi-task output, and can complete a comprehensive analysis of multiple dimensions such as tongue coating color, thickness, and characteristics in one go, outputting a comprehensive tongue coating assessment report, which is far more efficient than single-task models. Detailed Implementation

[0028] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0029] The present invention aims to: This addresses the disconnect between general deep learning models and the TCM regional diagnosis and treatment theory, enabling the model to focus on specific tongue areas that have diagnostic significance.

[0030] To address the issue of mutual interference between tongue coating color features and fine texture features during the extraction process, a dedicated channel is used to decouple and extract the two features separately.

[0031] To construct an end-to-end, high-precision, robust, and interpretable tongue coating image analysis model, thereby achieving automated, standardized, and precise analysis of tongue coating color and texture.

[0032] To achieve the above objectives, this invention discloses a method for analyzing tongue coating images in Traditional Chinese Medicine based on a Region-Textural Dual-channel Fusion Network (ZT-FusionNet). This method includes the following steps: S1: Image Acquisition and Preprocessing S1.1 Image Acquisition: Using standardized image acquisition equipment, images of the patient's tongue are acquired under constant color temperature (e.g., 5500K), fixed light intensity, and shooting distance.

[0033] S1.2 Image Correction: Perform color correction on the acquired raw image to eliminate color differences caused by ambient light.

[0034] S1.3 Tongue Segmentation and Normalization: Using a pre-trained segmentation model (such as U-Net) or traditional image processing methods, the tongue region is accurately segmented, and background interference such as lips and teeth is removed. The segmented tongue image is then rotated and normalized, for example, by uniformly scaling it to 256x256 pixels.

[0035] S2: Feature extraction based on ZT-FusionNet The preprocessed tongue coating image is input into the ZT-FusionNet model. The core structure of this model includes a shared shallow feature extraction backbone, and two parallel proprietary feature extraction channels: the Zonal Spatial Attention Channel (ZSAC) and the Multi-scale Texture Enhancement Channel (MTEC).

[0036] S2.1 Shared Backbone: A lightweight CNN (such as the first few layers of MobileNetV3 or a simplified ResNet) is used as the shared backbone to perform preliminary shallow feature extraction on the input tongue coating image, resulting in a basic feature map F. base ∈R H×W×C This step aims to extract low-level information from the image, such as edges, corners, and basic color blocks.

[0037] S2.2 Regional Spatial Attention Channel (ZSAC): This channel aims to simulate the diagnostic logic of "regional division to determine the internal organs" in traditional Chinese medicine, enabling the model to learn to focus on specific regions of the tongue.

[0038] Define the region mask: Based on Traditional Chinese Medicine theory, the normalized tongue surface space is predefined into multiple diagnostic regions, such as: tongue tip region (heart and lungs), tongue middle region (spleen and stomach), tongue root region (kidneys), and tongue side region (liver and gallbladder). Create a binary spatial mask M for each region k. k ∈R H×W .

[0039] Region feature extraction: extracting the basic feature map F base With each region mask M k Element-wise multiplication is performed to obtain the unique feature map F for each region. k =F base ⊙M k .

[0040] Region attention generation: For each region feature map F... k The input is fed into a small attention subnetwork (e.g., two 1×1 1×1 convolutional layers followed by a sigmoid activation function) to generate the attention weight map A for that region. k ∈R H×W This weight map indicates the key areas that the model should focus on within this region.

[0041] A k =σ(W k2 *ReLU(W k1 *F k +bk1 )+b k2 ) Where W and b are the weights and biases of the convolutional layer, σ is the sigmoid function, and ∗ is the convolution operation.

[0042] Attention feature weighting: Weighting the attention weight map A k Application back to basic feature map F base The feature map F obtained after regional attention weighting is obtained. k =F base ⊙A k .

[0043] Feature aggregation: Aggregate the weighted feature maps of all regions (e.g., by addition or concatenation) to obtain the final regional spatial features F. zonal =∑ k F k ′.

[0044] S2.3 Multiscale Texture Enhancement Channel (MTEC): This channel is specifically designed to capture the fine textural features of the tongue coating, such as thickness, moisture, dryness, greasiness, etc.

[0045] Atrous Convolution Pyramid: The core technique is to use a set of parallel atrous convolutions with different dilation rates. This involves applying a base feature map F... base Multiple parallel 3×3 convolutional layers are fed into the feed, each with a different dilation rate d, for example, d={1,2,4,8}. Dilated convolutions can exponentially expand the receptive field without increasing computational cost or parameters, thereby capturing texture information at different scales. texture (d) =Conv 3×3,dilation=d (F base ) Texture feature fusion: F is the feature map output from all convolutions with different dilation rates. texture (d) Concatenate along the channel dimension to form an enhanced feature map that contains multi-scale texture information. texture =Concat([F texture (1) ,F texture (2) ,F texture (4) ,F texture (8) ]) Feature dimensionality reduction: A 1×1 convolution is used to process the concatenated feature map to integrate information and reduce the number of channels, resulting in the final texture feature F. texture′ .

[0046] S3: Feature Fusion and Classification / Analysis S3.1 Dual-channel feature fusion: The features F output from the regional spatial attention channel are fused together. zonal and the feature F of the multi-scale texture enhancement channel output texture The features are stitched together along the channel dimension to form a powerful fusion feature map that simultaneously contains information on "where it matters" and "how the texture is". fusion =Concat([F zonal ,F texture ′]) S3.2 Classification and Analysis: Merging Feature Map F fusion The input is then fed into the subsequent Classification Header. This Classification Header typically consists of a Global Average Pooling layer and several Fully Connected Layers.

[0047] Multi-task learning: The output layer can be designed for multi-task output, simultaneously predicting multiple tongue coating attributes. For example: Moss color classification task: Output categories include "white moss", "yellow moss", "gray-black moss", etc.

[0048] Moss thickness classification task: Output categories include "thin moss", "thick moss", "peeling moss", etc.

[0049] Moss quality-trait classification task: Output categories include "moist moss", "dry moss", "greasy moss", "rotten moss", etc.

[0050] Loss function: During training, a joint loss function for multiple tasks is used. For example, the cross-entropy loss function is used for each classification task, and they are weighted and summed.

[0051] S4: Results Output and Visualization S4.1 Structured Report Generation: Integrates the model's analysis results (such as moss color: yellow moss, moss texture: thick, greasy) into a structured text report.

[0052] S4.2 Attention Visualization: Visualizing the attention map A generated from the regional spatial attention channels. k A heatmap visualization is created and overlaid on the original tongue image. This visually demonstrates the tongue surface area that the model focuses on when making judgments, providing a basis for decision-making for TCM practitioners and enhancing the interpretability of the model.

[0053] The technical solution of the present invention will be further described in detail below with reference to specific embodiments.

[0054] The entire process includes: Step 1: Image acquisition and preprocessing.

[0055] Inside a specially designed darkroom with an internal diffuse coating, a ring-shaped LED light source (color temperature 5500K±100K) and a high-definition camera (e.g., a Sony IMX586 sensor) are positioned. The patient is asked to extend their tongue naturally, filling 70%-80% of the viewfinder. A raw image with a resolution of 4000x3000 is captured.

[0056] Use a color correction chart (such as X-Rite ColorChecker) to calibrate the camera and apply a correction matrix to each captured image to ensure color consistency.

[0057] A pre-trained U-Net model is used for pixel-level segmentation of the tongue. The minimum bounding rectangle of the segmented tongue is calculated using a binary mask, and the angle between its major axis and the horizontal line is calculated. An affine transformation is then used to rotate the tongue to a horizontal position. Finally, the tongue image is cropped and scaled to a uniform size of 256x256 pixels.

[0058] Step 2: Feature extraction from the ZT-FusionNet model.

[0059] Input a 256x256x3 image of the tongue coating into the ZT-FusionNet model.

[0060] Shared backbone: The first two stages of ResNet-18 are used as the shared backbone. After the input image passes through conv1, bn1, relu, maxpool, and layer1, a basic feature map F with a size of 64x64x64 is output. base .

[0061] Regional Spatial Attention Channel (ZSAC): For a 64x64 spatial dimension, four region masks M are predefined. k (k∈{tip, middle, root, edge}). For example, the tongue tip mask M. 尖 The top 1 / 4 of the image is 1, and the rest is 0.

[0062] F base With each M k Element-wise multiplication yields F k Each F k A 64x64x1 attention map A is generated through an attention subnetwork (1×1 Conv(64->16) -> ReLU -> 1×1 Conv(16->1) -> Sigmoid).k .

[0063] Four attention maps A k Compared with the original basic feature map F respectively base Multiply to get F k ′.

[0064] Finally, four Fs k By adding element by element, we obtain the final spatial feature F of the 64x64x64 region. zonal .

[0065] Multi-scale Texture Enhancement Channel (MTEC): This channel enhances the underlying feature map F... base (64x64x64) Four branches are fed in parallel: Branch 1: 3×3 Conv, dilation=1, padding=1, output F texture (1) (64x64x64).

[0066] Branch 2: 3×3 Conv, dilation=2, padding=2, output F texture(2) (64x64x64).

[0067] Branch 3: 3×3 Conv, dilation=4, padding=4, output F texture (4) (64x64x64).

[0068] Branch 4: 3×3 Conv, dilation=8, padding=8, output F texture (8) (64x64x64).

[0069] These four output feature maps are concatenated along the channel dimension to obtain a 64x64x256 feature map. Then, a 1×1 Conv(256->64) is used for dimensionality reduction to obtain the final 64x64x64 texture feature F. texture ′.

[0070] Step 3: Feature fusion and multi-task classification.

[0071] F zonal (64x64x64) and F texture The 64x64x64 graph is concatenated along the channel dimension to obtain the fused feature map F. fusion (64x64x128).

[0072] F fusion After passing through a global average pooling layer, it becomes a 1x1x128 feature vector.

[0073] The feature vector is fed into three independent classification heads (each classification head consists of one or two fully connected layers): Moss classification head: FC -> Softmax, outputs the probability of 3 categories (white, yellow, gray-black).

[0074] Thickness / thinness classification head: FC -> Softmax, outputs the probability of 3 categories (thin, thick, peeling).

[0075] Trait classification head: FC -> Sigmoid, outputs the probability of 4 binary labels (moist, dry, greasy, rotten). Since traits may coexist, Sigmoid is used for multi-label classification.

[0076] During model training, a dataset of 50,000 tongue coating images jointly annotated by three senior TCM doctors was used. The Adam optimizer was employed with an initial learning rate of 1e-4, a batch size of 32, and a training duration of 100 epochs. The total loss function was L. total =λ1L color +λ2L thickness +λ3L property , where λ i The task weight can be set to 1.0, 1.0, or 1.5 based on experience.

[0077] Step 4: Output the results.

[0078] For a new tongue coating image, after model inference, the output is similar to: "Coating color: yellow (95%); Coating texture - thickness: thick (98%); Coating texture - characteristics: greasy (92%), moist (78%)". Simultaneously, the four attention maps A generated in the ZSAC channel are... k A weighted summation yields a total attention map, which is then overlaid on the original image after pseudo-color rendering. The visualization shows that the model mainly focuses on the thick, yellowish areas in the middle and root regions of the tongue.

[0079] The system implementing this invention can be a high-performance workstation, including: a central processing unit (CPU), a graphics processing unit (GPU, such as an NVIDIA RTX 4090), memory (RAM, such as 64GB), a hard disk (SSD, used to store datasets and models), an image acquisition interface (such as USB 3.0, for connecting a camera), and a monitor. On the software side, a deep learning framework (such as PyTorch or TensorFlow) runs on top of the operating system, and the program described in this invention is deployed.

[0080] To gain a deeper understanding of the technical solution of this invention, the following implementation examples are provided: Suppose we have a preprocessed image of tongue coating, and we input it into the ZT-FusionNet model.

[0081] S1: Preprocessing complete Input image: After preprocessing, a tongue coating image with a size of 4x4 pixels and 3 color channels (RGB) is obtained.

[0082] Input tensor I (size: 4x4x3): For simplicity, consider only one channel, or assume that the shared trunk has already been processed.

[0083] S2.1: Shared Backbone Output Input: The preprocessed image.

[0084] Process: The image is processed through a simplified shared backbone network (such as several layers of convolution and pooling).

[0085] Output: Basic feature map F base (Size: 4x4x2) Assume the output base feature map has two channels, represented by two 4x4 matrices: F base (Channel 1): [[1, 1, 2, 2], [1, 1, 2, 2], [3, 3, 4, 4], [3, 3, 4, 4]] F base (Channel 2): [[0, 0, 1, 1], [0, 0, 1, 1], [2, 2, 3, 3], [2, 2, 3, 3]] There is a 2-dimensional feature vector at each pixel location. For example, the feature vector at position (0, 2) is [2, 1].

[0086] S2.2: Zone Spatial Attention Channel (ZSAC) Define the region mask (size: 4x4). According to traditional Chinese medicine theory, the 4x4 space is divided into two regions: the "tip of the tongue" and the "root of the tongue".

[0087] M 尖 (Tongue tip mask, top two lines): [[1, 1, 1, 1], [1, 1, 1, 1], [0, 0, 0, 0], [0, 0, 0, 0]] M 根 (Tongue root mask, bottom two lines): [[0, 0, 0, 0], [0, 0, 0, 0], [1, 1, 1, 1], [1, 1, 1, 1]] Regional feature extraction (taking the tip of the tongue as an example) F 尖 =F base ⊙M 尖 (element-by-element multiplication) F 尖 (Channel 1): [[1, 1, 2, 2], [1, 1, 2, 2], [0, 0, 0, 0], [0, 0, 0, 0]] F 尖 (Channel 2): [[0, 0, 1, 1], [0, 0, 1, 1], [0, 0, 0, 0], [0, 0, 0, 0]] Region attention generation (taking a single pixel in the tip of the tongue region as an example) Calculate the attention value at position (0, 2). The feature vector at this position is v=[2, 1].

[0088] The attention subnetwork consists of two 1×1 convolutional layers, which is equivalent to two fully connected layers acting on the feature vector of each pixel.

[0089] First layer (1×1Conv): Weight W k1 (Size 2x1): [[0.5], [0.8]] bias b k1 [0.1] = (2*0.5 + 1*0.8) + 0.1 = 1.0 + 0.8 + 0.1 = 1.9 Activation function (ReLU): Calculate: a1 = ReLU(z1) = ReLU(1.9) = 1.9 Second layer (1×1Conv): Weight W k2 (Size 1x1): [[0.6]] bias b k2 [-0.4] Calculate: z2 = a1⋅W k2 +b k2 =1.9∗0.6−0.4=1.14−0.4=0.74 Activation function (Sigmoid): σ(x) = 1 / (1 + e−x) Calculate: A 尖 (0,2)=σ(0.74)=1 / (1+e−0.74)≈1 / (1+0.477)≈0.677 For F 尖 Repeat this process for all non-zero pixels, and assume we obtain the complete attention map A. 尖 (Size 4x4): [[0.55, 0.55, 0.68, 0.68], [0.55, 0.55, 0.68, 0.68], [0.00, 0.00, 0.00, 0.00], / / 0 outside the region [0.00, 0.00, 0.00, 0.00]] Similarly, calculate A 根 .

[0090] Attention feature weighting F 尖 =F base ⊙A 尖 F 尖 (Channel 1): [[1*0.55, 1*0.55, 2*0.68, 2*0.68], [1*0.55, 1*0.55, 2*0.68, 2*0.68], [0, 0, 0, 0], [0, 0, 0, 0]] = [[0.55, 0.55, 1.36, 1.36], [0.55, 0.55, 1.36, 1.36], [0.00, 0.00, 0.00, 0.00], [0.00, 0.00, 0.00, 0.00]] The calculation method for channel 2 is the same.

[0091] Feature aggregation Assuming F has been calculated 根 ′.

[0092] F zonal =F 尖 ′+F 根 (Add element by element).

[0093] Assumptions (for simplicity, the results are given directly): F 根 (Channel 1): [[0.00, 0.00, 0.00, 0.00], [0.00, 0.00, 0.00, 0.00], [2.55, 2.55, 3.60, 3.60], [2.55, 2.55, 3.60, 3.60]] Then F zonal (Channel 1): [[0.55, 0.55, 1.36, 1.36], [0.55, 0.55, 1.36, 1.36], [2.55, 2.55, 3.60, 3.60], [2.55, 2.55, 3.60, 3.60]] F zonal The calculation method for (Channel 2) is the same. The final result is the complete F. zonal (Dimensions: 4x4x2).

[0094] S2.3: Multi-Scale Texture Enhancement Channel (MTEC) Dilated convolution (using dilation=2 as an example) Input: F base Channel 1 (4x4).

[0095] Convolution kernel K (size 3x3): [[1, 0, 1], [0, 1, 0], [1, 0, 1]] Dilation rate: d=2 Padding: To maintain the size, padding=2 is required.

[0096] Calculate the output feature map F texture (2)The value at position (1, 1).

[0097] Since d=2, the elements of the convolution kernel will be spaced d-1=1 pixels apart.

[0098] The receptive field positions acting on the input feature map are: (1,1) surrounding (-1,-1), (-1,0), (-1,1), (0,-1), (0,0), (0,1), (1,-1), (1,0), (1,1). After multiplying by the expansion rate d=2, the actual sampling position offset relative to the center (1,1) is: (-2,-2), (-2,0), (-2,2), (0,-2), (0,0), (0,2), (2,-2), (2,0), (2,2). This means that to calculate the output (1,1), we need to look at the values ​​at the input positions (1-2, 1-2)=(-1,-1), (1-2, 1+0)=(-1,1), and so on. These positions are valid due to padding.

[0099] Centered on (1,1) of the padded input, the positions and values ​​of kernel K-samples are as follows: Input Padded: [[p, p, p, p, p, p, p, p], [p, p, 1, 1, 2, 2, p, p], <- corresponds to input (0,_) [p, p, 1, 1, 2, 2, p, p], <- corresponds to input (1,_) [p, p, 3, 3, 4, 4, p, p], <- corresponds to input (2,_) [p, p, 3, 3, 4, 4, p, p], <- corresponds to input (3,_) ... K[0,0] = 1 multiplied by Input[1−2,1−2] = Input[−1,−1] = padding value (assumed to be 0). K[0,2] = 1 multiplied by Input[1−2,1+2] = Input[−1,3] = padding value (assumed to be 0). ... K[1,1]=1 multiplied by Input[1+0,1+0]=Input[1,1]=1 ... This is a complex calculation, but the principle is that the core is "expanded".

[0100] To simplify the demonstration, let's assume that after two convolutions with different dilation rates, we get: F texture (1) (dilation=1, dimensions 4x4x2) and F texture (2) (dilation=2, dimensions 4x4x2).

[0101] Texture feature fusion and dimensionality reduction Concatenate: Combine F texture (1） and F texture (2) Concatenate along the channel dimension. F texture_cat =Concat([F texture (1) ,F texture (2) The output size is 4x4x4.

[0102] Dimensionality reduction (1×1 Conv): Use 1×1 convolution to reduce the number of channels from 4 back to 2. Assume the final result after dimensionality reduction is... Ftexture (Dimensions: 4x4x2)

[0103] S3: Feature Fusion and Classification Dual-channel feature fusion Input: F zonal (4x4x2) and F texture ′(4x4x2).

[0104] Process: Concatenation along the channel dimension. F fusion =Concat([F zonal ,F texture ′]) Output: Fused feature map F fusion (Size: 4x4x4) Classification / Analysis Global Average Pooling (GAP): For F fusion For each channel, the average value at all spatial locations is calculated.

[0105] Assume F fusion The four channels are C1, C2, C3, and C4.

[0106] vec[0] = Average(C1) vec[1] = Average(C2) vec[2] = Average(C3) vec[3] = Average(C4) Assume the calculated average vector is v gap = [1.8, 2.5, 0.9, 1.2] (Size: 1x4).

[0107] Category Header (taking "Moss Color" as an example): Input: v gap = [1.8, 2.5, 0.9, 1.2] Task: Classify into 3 categories (white moss, yellow moss, gray-black moss) Fully connected layer weights W fc (Size 4x3): [[ 0.2, -0.1, 0.5], / / to white, yellow, gray-black [ 0.8, 0.6, -0.3], [-0.5, 0.4, 0.7], [ 0.1, 0.9, -0.2]] bias b fc (Size 1x3): [0.1, 0.05, -0.1] Calculate Logits: L = v gap ⋅W fc +b fc L[0](white) = (1.8*0.2 + 2.5*0.8 - 0.9*0.5 + 1.2*0.1) + 0.1 = (0.36 + 2.0 - 0.45 + 0.12) + 0.1 = 2.03 + 0.1 = 2.13 L[1](yellow)=(1.8∗(−0.1)+2.5∗0.6+0.9∗0.4+1.2∗0.9)+0.05=(−0.18+1.5+0.36+1.08)+0.05=2.76+0.05=2.81 L[2](grayish-black) = (1.8*0.5−2.5*0.3+0.9*0.7−1.2*0.2)−0.1 = (0.9−0.75+0.63−0.24)−0.1 = 0.54−0.1 = 0.44 Output Logits: L = [2.13, 2.81, 0.44] Softmax activation: P(i)=e L[i] / ∑j eL[j] e 2.13 ≈8.41 e 2.81 ≈16.61 e 0.44 ≈1.55 Sum = 8.41 + 16.61 + 1.55 = 26.57 P(white moss) = 8.41 / 26.57 ≈ 0.317 P(yellow moss) = 16.61 / 26.57 ≈ 0.625 P(grayish-black moss) = 1.55 / 26.57 ≈ 0.058 S4: Output Results Structured Report: Moss color prediction: Yellow moss (probability: 62.5%) Visualization: The attention map A generated in the middle of ZSAC 尖 and A 根 By weighted summation and rendering with a heatmap, overlaid on the original 4x4 image, it can be observed that the model has different levels of attention in rows 1 and 2, and rows 3 and 4.

[0108] This simplified data example fully demonstrates the entire computational chain from the input feature map to the final classification probability, clearly illustrating the mathematical operations of the two core modules: regional spatial attention and multi-scale texture enhancement.

[0109] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A deep learning-based method for analyzing tongue coating images in Traditional Chinese Medicine, characterized in that, Includes the following steps: a. Obtain the image of the tongue coating to be analyzed and preprocess it; b. Input the preprocessed tongue coating image into a region-texture dual-channel fusion network model for feature extraction. The fusion network model includes a shared backbone network and a region spatial attention channel and a multi-scale texture enhancement channel connected in parallel thereafter. c. Fuse the regional spatial features output by the regional spatial attention channel with the texture features output by the multi-scale texture enhancement channel to obtain fused features; d. Based on the fusion features, classify or analyze at least one physiological or pathological attribute of the tongue coating and output the results.

2. The deep learning-based TCM tongue coating image analysis method according to claim 1, characterized in that, In step b, the specific implementation of the regional spatial attention channel includes: Based on the theory of tongue diagnosis in traditional Chinese medicine, at least two diagnostic region masks are preset in the feature map space of the tongue coating image; For each diagnostic region, a spatial attention weight map within that region is learned using an attention subnetwork; The spatial attention weight map is used to weight the base feature map output from the shared backbone network to enhance the feature representation of diagnostically significant regions; All attention-weighted regional features are aggregated to form the regional spatial features.

3. The deep learning-based TCM tongue coating image analysis method according to claim 2, characterized in that, In step b, the specific implementation of the multi-scale texture enhancement channel includes: A set of parallel dilated convolutional layers with at least two different dilation rates is used; The basic feature maps output from the shared backbone network are respectively input into the parallel dilated convolutional layer to capture tongue texture information at different scales; The output feature maps of the parallel dilated convolutional layers are spliced ​​and fused to form the texture features.

4. The deep learning-based TCM tongue coating image analysis method according to claim 3, characterized in that: The feature fusion method in step c is to splice the regional spatial features and the texture features in the channel dimension.

5. The deep learning-based TCM tongue coating image analysis method according to claim 4, characterized in that: The physiological or pathological attributes of the tongue coating in step d include at least one of the following: coating color, coating thickness, and coating texture; the classification or analysis adopts a multi-task learning architecture to predict multiple attributes simultaneously.

6. The deep learning-based TCM tongue coating image analysis method according to claim 5, characterized in that: It also includes visualizing the spatial attention weight map and overlaying it onto the original tongue image to indicate the key areas for model decision-making.

7. The deep learning-based TCM tongue coating image analysis method according to claim 6, characterized in that: The preprocessing in step a includes color correction, tongue segmentation, and size normalization.

Images