Tongue body segmentation and tongue coating dryness moistening identification method based on multi-modal image fusion

By combining multimodal feature fusion of color and polarized images with gated convolution, the problems of strong subjectivity in traditional Chinese medicine tongue diagnosis and inaccuracy of existing recognition methods are solved, and high-precision recognition of tongue coating moisture and dryness is achieved.

CN119007236APending Publication Date: 2024-11-22BEIJING UNIV OF TECH
View PDF 0 Cites 5 Cited by

Patent Information

Application Number
CN202411214623.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-01
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

Traditional Chinese medicine tongue diagnosis relies on the doctor's subjective observation, which is easily affected by external factors, affecting the accuracy and reliability of the diagnosis. Existing methods for identifying the moisture and dryness of the tongue coating do not yield ideal predictive results.

Method used

By combining color and polarized images of the tongue, a two-stage fusion network based on dual-stream Transformer is used for tongue segmentation. Multimodal and multi-scale feature fusion and gated convolution are used to extract the bright spot features of tongue coating moisture, and a tongue coating moisture recognition algorithm is constructed.

Benefits of technology

It significantly improves the accuracy of tongue coating moisture and dryness identification, avoids the process of manually extracting moisture-bright areas and calculating the moisture coefficient, and enhances the objectivity and accuracy of diagnosis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119007236A_ABST
    Figure CN119007236A_ABST
Patent Text Reader

Abstract

The invention discloses a tongue body segmentation and tongue coating dryness moistening identification method based on multi-modal image fusion, and belongs to the crossing field of traditional Chinese medicine tongue diagnosis and image processing. At present, a tongue coating dryness-moistening identification technology mainly depends on a traditional dichotomy light reflection model or machine learning. The methods often cannot adaptively capture the characteristics of a tongue moisture bright spot area and an adjacent area, so that the accuracy of dryness moistening identification is influenced. According to the method, the polarization degree image is creatively fused, the moisture bright spot area of the tongue coating is extracted through gating convolution, and the network adaptively pays attention to the moisture bright spot area in the process of extracting the visible light image features of the tongue body through a step-by-step dot multiplication mode. Compared with manual calculation of the dryness-moistening coefficient, the accuracy of tongue coating dryness-moistening identification is remarkably improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the intersection field of traditional Chinese medicine tongue diagnosis and image processing, and particularly relates to tongue segmentation and tongue coating moisture and dryness identification analysis, which is used to predict the moisture and dryness category of the tongue coating. Background Art

[0002] Tongue diagnosis in traditional Chinese medicine is a method of diagnosing diseases by observing changes in tongue shape, tongue color, tongue coating, and tongue quality. In tongue diagnosis, the degree of dryness and moistness of the tongue is one of the important indicators for judging the deficiency or excess of body fluid in the human body, and is of great significance for diagnosing cold and heat syndromes. However, traditional tongue diagnosis mainly relies on the subjective observation and experience of doctors, so the diagnostic results are easily affected by factors such as external lighting conditions and the patient's tongue extension posture, which limits the reliability and popularity of its clinical application. The advancement of information technology has provided a new opportunity for the standardization and automation of tongue diagnosis in traditional Chinese medicine. In order to improve the accuracy and efficiency of diagnosis, it is necessary to build a tongue coating dryness and moistness recognition system that integrates integrity, accuracy, and automation to achieve objective tongue diagnosis and promote the widespread application of tongue diagnosis.

[0003] At present, most tongue coating moistness and dryness identification methods mainly rely on the color image of the tongue for analysis, and most algorithms use image-based reflection models to identify the tongue coating moisture bright spot area. However, such methods often lead to unsatisfactory prediction results. Given that polarization information can intuitively reflect the moisture bright spot information on the tongue surface, the present invention is based on the idea of ​​multimodal information fusion, combining the polarization image and color image of the tongue to achieve tongue segmentation and tongue coating moistness and dryness identification based on multimodal image fusion. Summary of the invention

[0004] The polarization degree map of the tongue can intuitively reflect the bright spots of tongue coating moisture and distinguish the white coating area, while the polarization angle map can intuitively reflect the edge information of the tongue. Based on this, the present invention innovatively combines the color and polarization images of the tongue to invent a tongue coating moist and dry identification method. First, in order to retain more edge detail information in tongue segmentation, the present invention proposes a two-stage fusion network based on intra-layer fusion and cross-layer enhancement based on the dual-stream Transformer. By designing a cross-modal multi-scale feature fusion block and a step-by-step joint enhancement module, the tongue's visible light-polarization angle (RGB-A) image features of the same level and the complementary information between multi-level features are fully mined and fused, achieving a more accurate tongue segmentation. In order to predict the tongue coating moist and dry category, the present invention inputs the tongue RGB image features obtained in the tongue segmentation stage into a multi-layer neural network to extract deep features. At the same time, the tongue polarization degree map as an auxiliary modality is sent to the gated convolution to generate an attention map of the tongue coating moisture bright spot area, and integrates it step by step into the RGB tongue image feature extraction process. Finally, a fully connected layer is used to realize the prediction of the tongue coating moist and dry category. Compared with the existing algorithms, the present invention avoids the process of manually extracting the moisture bright spot area and calculating the moistness coefficient, and significantly improves the accuracy of moistness identification.

[0005] The present invention provides a tongue segmentation and tongue coating moistness and dryness recognition algorithm based on multimodal image fusion, comprising the following steps:

[0006] 1. Obtaining visible light-polarization multimodal tongue images

[0007] In order to efficiently fuse the multimodal tongue images later, the present invention needs to obtain aligned visible light-polarization multimodal image pairs. According to the existing experimental conditions (the polarization sensor and the color sensor are not integrated), the present invention controls the servo by computer to quickly rotate the reflector 90 degrees to ensure that the tongue image is first presented on the color sensor and then on the polarization sensor, thereby obtaining the color and polarization images of the tongue in sequence. If the experimental conditions have met the integration of the polarization sensor and the color sensor, the aligned visible light-polarization multimodal image pairs can be directly collected. Subsequently, the polarization angle image and the polarization degree image are calculated by the Stokes vector.

[0008] 2. Constructing the tongue segmentation network

[0009] The present invention proposes a two-stage multimodal feature fusion method based on the parallel extraction of RGB and polarization angle image features of the tongue by dual-branch Transformer. The method consists of a multimodal multi-scale fusion stage and a step-by-step enhancement stage. Different from the current multimodal segmentation network based on the interaction of single-scale features at the same level, the present invention performs multi-scale processing on each level of multimodal features through multi-scale strip convolution operations in the multimodal multi-scale fusion stage to reveal the differences of multimodal features at different scales. In addition, in feature interaction and fusion, the present invention does not adopt the conventional attention mechanism, but proposes a cross-modal multi-scale interaction mechanism to achieve the interaction between multimodal multi-scale features at each level through complementary channel splicing operations. Subsequently, multimodal feature fusion is achieved through feature compression operations. In the step-by-step enhancement stage, the present invention combines the detail feature enhancement block and the semantic feature enhancement block, and realizes the bidirectional enhancement of the fused multi-level features through dual-path filters and gating mechanisms; finally, the decoder accurately predicts the tongue area through two fully connected layers.

[0010] 3. Tongue segmentation based on multimodal image fusion

[0011] Training process: The model training adopts the pre-training method. The encoder is pre-trained using the ImageNet dataset to enable it to have good feature extraction capabilities. The visible light-polarization angle tongue image in the training set is then input to adjust the model to further improve its ability to extract features from the tongue image. During the adjustment process, the actual tongue area is taken as the true value, and it and the predicted tongue segmentation map are input into the loss function to generate errors. The network is optimized by continuously back-propagating the error gradient.

[0012] Testing process: Input the visible light-polarization angle tongue image in the test set into the trained tongue segmentation network to obtain the predicted tongue area. Calculate the intersection over union (IoU) of the tongue segmentation image and the true value. The higher the IoU, the higher the segmentation accuracy of the tongue.

[0013] 4. Constructing a tongue coating moistness and dryness recognition network

[0014] After the tongue segmentation network realizes accurate segmentation of the tongue, the present invention constructs a tongue coating moisturizing and dryness identification network and adopts an efficient feature extraction strategy. Specifically, the present invention does not re-extract the RGB tongue image features, but directly inputs the underlying features of the visible light image extracted by the tongue segmentation network into the multi-layer neural network to further extract the tongue features. In addition, the present invention combines gated convolution, which is composed of a combination of multiple convolutional layers and activation functions to effectively capture the characteristics of the moisture bright spot area in the tongue polarization map. By multiplying the polarization map features with the RGB map features step by step and element by element, the network pays more attention to the characteristics of the tongue moisture bright spot area in the visible light map. Finally, the present invention predicts the tongue coating moisturizing and dryness category through three fully connected layers and a softmax function to identify whether the tongue coating is moist, slippery or dry.

[0015] The first two convolutional layers in the gated convolution are responsible for extracting local features and downsampling, while the third convolutional layer focuses on reducing the number of channels and generating the final gating signal through the activation function.

[0016] 5. Tongue coating moistness and dryness recognition based on multimodal image fusion

[0017] Training process: The visible light-polarization angle image in the training set is input into the tongue segmentation network to obtain the tongue segmentation map. Then the underlying parameters of the tongue segmentation network are frozen, and the visible light image tongue features and polarization degree tongue image are simultaneously sent to the tongue coating moisturizing and dryness recognition network to achieve feature encoding, and finally the category of tongue coating moisturizing and dryness is output. The true value and the network prediction result are input into the loss function to generate an error, and the network is optimized by continuously back-propagating the error gradient.

[0018] Testing process: The visible light-polarization angle map in the test set is input into the tongue segmentation network to obtain the tongue area. Then the RGB tongue feature map and polarization tongue map are sent to the trained tongue coating moisturizing recognition network one by one to predict the tongue coating moisturizing category. Finally, the accuracy of tongue coating moisturizing recognition is evaluated by F1 and Micro-F1 scores.

[0019] Beneficial Effects

[0020] 1. This paper proposes a tongue segmentation and tongue coating moistness recognition method based on multimodal image fusion

[0021] As far as we know, this is the first time that multimodal image fusion has been applied to the entire process of tongue segmentation and tongue coating moistness and dryness identification. By sharing the underlying features, a multimodal tongue segmentation network and a tongue coating moistness and dryness identification network are constructed, which effectively utilizes the color image and polarization image information of the tongue, significantly improving the accuracy of tongue coating moistness and dryness identification.

[0022] 2. This paper proposes a multimodal image tongue segmentation network

[0023] The present invention proposes a progressive two-stage multimodal feature fusion strategy in combination with a dual-stream Transformer. Taking into account the differences between the same-level features of the tongue's visible light image and polarization angle image and the differences between multi-level features, the present invention performs intra-layer fusion through a multimodal multi-scale fusion module and performs inter-layer feature enhancement using a step-by-step joint enhancement module, fully mining and fusing the same-level features of the visible light-polarization angle image and the complementary information between multi-level features.

[0024] In current research, the feature fusion methods used in multimodal image segmentation networks are mostly limited to a single scale and rely on a general attention mechanism. The potential complementary relationship between multimodalities needs to be further explored. Therefore, the present invention combines multi-scale operations to promote feature interaction. Through the splicing operation of multi-channel groups between multimodalities, in-depth interaction of multimodal features at the same level is achieved, while retaining more channel and spatial information. In addition, in order to achieve information interaction between multiple levels, the present invention proposes a detail feature enhancement module and a semantic feature enhancement module. The former retains important information of shallower features step by step through a dual-branch filter to enrich detail information such as edge contours of the current level; the latter filters deeper feature information step by step through a gating mechanism, thereby optimizing the spatial information of the current level.

[0025] 3. The present invention combines polarization information to achieve tongue coating moistness and dryness identification

[0026] At present, the technology for identifying the moisture of the tongue coating mainly relies on the traditional dichotomous reflectance model or machine learning. These methods often cannot adaptively capture the characteristics of the moisture bright spot area of ​​the tongue and its adjacent areas, thus affecting the accuracy of moisture identification. The present invention innovatively fuses the polarization image, extracts the moisture bright spot area of ​​the tongue coating through gated convolution, and uses a step-by-step dot multiplication method to enable the network to adaptively focus on the moisture bright spot area in the process of extracting the features of the tongue visible light image. Compared with the manual calculation of the moisture coefficient, the accuracy of tongue coating moisture identification is significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a structural diagram of the multimodal tongue image acquisition device.

[0028] Figure 2 This is a network structure diagram of the present invention, including a tongue segmentation stage and a tongue coating recognition stage.

[0029] Figure 3 This is the first stage fusion of the tongue segmentation part of the present invention, that is, the cross-modal multi-scale feature fusion block.

[0030] Figure 4 It is a detail feature enhancement module for the second stage fusion of the tongue segmentation part of the present invention.

[0031] Figure 5 It is the semantic feature enhancement module of the second stage fusion of the tongue segmentation part of the present invention. DETAILED DESCRIPTION

[0032] The specific method of the present invention is described in detail below with reference to the accompanying drawings.

[0033] Step 1: Multimodal tongue image acquisition

[0034] According to the existing experimental conditions, in order to obtain aligned multimodal tongue images, the present invention first constructs a multimodal tongue acquisition system. Figure 1 As shown, it mainly consists of 1. color camera, 2. polarization camera, 3. height-adjustable chin rest, 4. full-spectrum standard light source, 5. transmission cable, 6. desktop computer, 7. lampshade, 8. servo, 9. reflector, etc. The light source illumination condition is a square light source, the color camera can be Hikvision MV-CS050-10UC, and the polarization camera can be Daheng MER2-503-36U3M POL polarization camera. The transmission cable uses the USB cable that comes with the camera. One end is connected to the output port of the camera, and the other end is connected to the USB serial port of the computer. In order to capture the aligned color and polarization image pairs, the color camera and polarization camera lenses are placed opposite to each other so that they are on the same axis. A reflector is placed between the two cameras and connected to the servo. The computer controls the rotation of the servo, so that the reflector can rotate in two directions, thereby realizing rapid imaging of the subject on different cameras.

[0035] The specific process of collecting a set of aligned multimodal tongue images is as follows: the computer first triggers the color camera to collect RGB tongue images, and then triggers the servo after an interval of 1 second, so that the servo rotates 90 degrees, and the reflector reverses 90 degrees. After an interval of 3 seconds, the polarization camera is triggered to collect polarized tongue images. Finally, the servo returns to its original position, and the reflector returns to its original position.

[0036] Step 2: Build a tongue segmentation network

[0037] The tongue segmentation network constructed by the present invention is as follows: Figure 2 As shown in the lower part, the specific parameters of each part are as follows:

[0038] 1) Segformer Since the introduction of self-attention can effectively capture the long-distance dependencies in the input image, the present invention migrates this feature extraction method to the tongue image feature extraction. R1, R2, R3, R4 and P1, P2, P3, P4 are all feature extraction layers. Among them, Ri extracts visible light tongue image features, and Pi extracts polarization angle tongue image features. Ri has the same structure as Pi, so the specific structure is described in detail using Ri as an example. Ri consists of basic Transformer blocks, including an overlapping block merge, N effective self-attentions, and a hybrid feedforward network. Specifically, the overlapping block merge merges image blocks in an overlapping manner through a convolution operation. Among them, the R1 layer receives a visible light image with a size of 512×612 and a number of channels of 3. The R1 layer performs overlapping block merging on the original tongue image through a 7×7 convolution kernel, a two-dimensional convolution with a step size of 4 and an edge padding of 3, and layer normalization. R2, R3, and R4 all set the image block size to 3×3, and use 3×3 convolution kernel, stride 2, and edge padding 1 to perform overlapping block merging, combined with layer normalization. The input sizes of R2, R3, and R4 are 128×153, 64×76, and 32×33, respectively. The number of input channels is 64, 128, and 320, respectively, and the number of output channels is 128, 320, and 512. The self-attention calculation includes linear embedding and self-attention mechanism. In the linear embedding process of Ri, the feature B (number of samples per round of training) × C (number of channels) × H (height) × W (width) after merging overlapping blocks is first flattened to B×C×(H*W), and then converted to B×(H*W)×C and passed through the fully connected layer to obtain Q (query), K (key), and V (value). The number of output channels of the fully connected layer is the same as the number of input channels. Next, Ri uses a multi-head self-attention mechanism to capture the correlation between different feature blocks. The number of heads of R1, R2, R3, and R4 are 1, 2, 5, and 8, respectively. The implementation process of the self-attention mechanism can be expressed as: The effective number of self-attentions N in R1, R2, R3, and R4 is 3, 4, 6, and 3, respectively. The features after the self-attention mechanism are added element-by-element to the original input and fed into the hybrid feedforward network. In the hybrid feedforward network, for the R1, R2, R3, and R4 feature extraction layers, a fully connected layer is used to expand the number of channels of the feature map to three times the number of input channels. Then, position encoding is achieved through a 3×3 convolution kernel, a step size of 1, and a depthwise convolution with an edge padding of 1. After that, the GELU activation function and the fully connected layer are used in turn to reduce the number of output channels to one-third of the number of input channels.

[0039] 2) During the feature extraction process, multi-modal features are fused step by step simultaneously. The structure of the cross-modal multi-scale fusion block (CMFF) is shown in the attached figure. Figure 3 shown.

[0040] First, in the channel dimension, the visible light image features r and polarization angle image features p obtained by the feature extraction layer are randomly divided into 3 groups. For each group of features, different sizes of depth-wise convolutions are used for processing. The group1 features are cascaded through two depth-wise convolutions to obtain r1 and p1, including convolutions with a kernel size of 1×3, a stride of 1, 1 column padded on the left and right, and a padding value of 0, and convolutions with a stride of 3×1, 1 row padded on the top and bottom, and a padding value of 0. The group2 features are also cascaded through two depth-wise convolutions to obtain r2 and p2, including convolutions with a kernel size of 1×5, a stride of 1, 2 columns padded on the left and right, and a padding value of 0, and convolutions with a stride of 5×1, 2 rows padded on the top and bottom, and a padding value of 0. The group3 features are subjected to depth-wise convolution with a kernel size of 1×7, a stride of 1, 3 columns padded on the left and right, and a padding value of 0, and a depth-wise convolution with a kernel size of 7×1, a stride of 1, 3 rows padded on the top and bottom, and a padding value of 0 to obtain r3 and p3.

[0041] Then, cross-modal multi-scale interaction is carried out. The specific structure is as follows: Figure 3 In order to achieve full interaction between different modalities and channels, three branches are used to complete multi-modal feature interaction. p1 In the branch, the features of r2, r3 and p1 are concatenated and then fused through a 1×1 convolution kernel, with the number of input channels equal to the sum of the number of channels of r2, r3 and p1, and the number of output channels equal to the number of channels of p1. p2 In the branch, the features of r1, r3 and p2 are concatenated and then fused through a 1×1 convolution kernel, with the number of input channels equal to the sum of the number of channels of r1, r3 and p2, and the number of output channels equal to the number of channels of p2. p3 The r1, r2 and p3 features are concatenated and then fused through a 1×1 convolution kernel, with the number of input channels equal to the sum of the number of r1, r2 and p3 channels, and the number of output channels equal to the number of p3 channels. The output features of the above three branches are then linearly integrated through a point-by-point convolution with a convolution kernel size of 1×1 and the number of output channels equal to the number of input channels. The feature p after interacting with r is obtained through batch normalization and sigmoid activation function. E In order to retain more original information, p E It is not directly fused, but multiplied with p, then added with r and sent to the channel compression module.

[0042] In addition, if r is taken as the subject, the feature r after interacting with p E Three branches are also required to complete the task. r1 In the branch, the features of p2, p3 and r1 are concatenated and then fused through a 1×1 convolution kernel, with the number of input channels equal to the sum of the number of channels of p2, p3 and r1, and the number of output channels equal to the number of channels of r1.r2 In the branch, the features of p1, p3, and r2 are concatenated and then fused through a 1×1 convolution kernel, with the number of input channels equal to the sum of the number of channels of p1, p3, and r2, and the number of output channels equal to the number of channels of r2. r3 The p1, p2 and r3 features are concatenated and then fused through a 1×1 convolution kernel, with the number of input channels equal to the sum of the number of p1, p2 and r3 channels, and the number of output channels equal to the number of r3 channels. The output features of the above three branches are then linearly integrated through a point-by-point convolution with a convolution kernel size of 1×1 and the number of output channels equal to the number of input channels. The feature r after interaction with p is finally obtained through batch normalization and sigmoid activation function. E In order to retain more original information, r E It is not directly fused, but multiplied with r, then added with p and sent to the channel compression module.

[0043] Finally, through Figure 3 The channel compression in the above method fuses the multimodal features after interaction to obtain the fused feature F. Channel compression consists of channel splicing and three convolutional layers. The convolution kernel of the first convolution is 1×1, the stride is 1, and the number of output channels is half of the number of input channels. The kernel size of the second depth-wise convolution is 3×3, the stride is 1, and the input features are padded with 1 layer on each side with a padding value of 0. The Relu activation function is then added to increase the nonlinear expression ability of the model. The convolution kernel of the third convolution is 1×1, the stride is 1, and the number of input and output channels is the same.

[0044] 3) In the process of feature extraction and fusion, in order to strengthen the information exchange between different levels, the fused features F at each level are enhanced by the detail enhancement module (DFE) and the semantic enhancement module (SFE). i (i=1,2,3,4) for feature enhancement.

[0045] Among them, the detail enhancement module structure is as follows Figure 4 As shown, it consists of two branches. Branch, first depth separable convolution DSConv on the previous level feature F i-1 To further extract features, the convolution kernel size is 3×3, the stride is 2, the padding size is 1, the padding value is 0, and the number of output channels is half of the number of input channels. Then batch normalization and activation function Relu are added. The convolution kernel size of the second convolution is 1×1, the stride is 1, there is no padding, the number of output channels is twice the number of input channels, and then batch normalization is added. The branch consists of an average pooling layer and two convolutional layers. The average pooling layer converts the previous level feature F into a shape of B (number of samples per training round) × C (number of channels) × H (height) × W (width). i-1The features are compressed to B×C×1. The convolution kernel size of the first convolution of the branch is 1×1, the stride is 1, there is no padding, and the number of output channels is half of the number of input channels. Then batch normalization and activation function Relu are added. The convolution kernel size of the second convolution of the branch is 1×1, the stride is 1, there is no padding, and the number of output channels is twice the number of input channels. Then batch normalization is added. Finally, and The results of the branches are added and convolved with a kernel size of 1×1, a step size of 1, and an input channel number of F. i-1 The number of channels and the number of output channels are F i The number of channels and unfilled convolution further fuse the results of the two branches. Finally, after passing the sigmoid activation function, it is combined with F i Multiply element by element to obtain the features after detail enhancement

[0046] The structure of the semantic enhancement module is as follows: Figure 5 As shown. i+1 The branch generates a gating signal through a convolution with a convolution kernel of 1×1, an output channel of 1, a step size of 1, no padding, and a sigmoid activation function. The gating signal is then compared with F i Multiply, enhance F i Considering the semantic information of F i+1 To improve the integrity of the transfer, a parallel fusion flow is also introduced. Specifically, first, F i+1 Upsampling is performed by bilinear interpolation, so that F i+1 The spatial resolution and F i Same as F i Then, the input channel is compressed to the same size as F by convolution with a 3×3 convolution kernel, a stride of 2, and a padding size of 1. i The same number of channels, batch normalization and ReLU activation function are added. Finally, the results of the two branches are added together to obtain the semantically enhanced features. Finally and Add element by element to get

[0047] 4) Obtain tongue segmentation map through MLP

[0048] MLP mainly consists of two fully connected layers. The first fully connected layer first transforms the enhanced features of each level (i=1,2,3,4) for channel splicing, and then reduce the number of input feature channels to one-fourth of the original through convolution with a convolution kernel size of 1×1. Then batch normalization and Relu activation function are added to improve the generalization ability of the model, and a dropout layer is added with a dropout ratio set to 0.1. The second layer of full connection reduces the input channel to 2 through convolution with a convolution kernel size of 1×1, and then obtains the tongue prediction map through the softmax function.

[0049] Step 3: Build a tongue coating moistness and dryness recognition network

[0050] The tongue coating moistening and dryness recognition network constructed by the present invention is as follows Figure 2 As shown in the upper part, C1, C2, C3, and C4 further extract deep features from the underlying features of the tongue visible light image, G1, G2, and G3 are responsible for extracting the moisture bright spot area features of the tongue polarization image, and FC predicts the final tongue coating moisturizing category. The detailed parameters of each layer are as follows:

[0051] 1) C1 includes two convolutional layers and one maximum pooling layer. The input size of the two convolutional layers is 128×153, the convolution kernel size is 3×3, the number of convolution kernels is 128, the stride is 1, the edge padding is 1, and the output is 128×153. Then the Relu activation function is added. Then, after passing through the maximum pooling layer with a kernel size of 2×2 and a stride of 2, the length and width of the input feature are halved.

[0052] 2) C2 includes two convolutional layers and one maximum pooling layer. The input size of the two convolutional layers is 64×76, the convolution kernel size is 3×3, the number of convolution kernels is 256, the stride is 1, the edge padding is 1, and the output is 64×76. Then the Relu activation function is added. Then, after passing through the maximum pooling layer with a kernel size of 2×2 and a stride of 2, the length and width of the input feature are halved.

[0053] 3) C3 includes two convolutional layers and one maximum pooling layer. The input size of the two convolutional layers is 32×38, the convolution kernel size is 3×3, the number of convolution kernels is 512, the stride is 1, the edge padding is 1, and the output is 32×38. Then the Relu activation function is added. Then, after passing through the maximum pooling layer with a kernel size of 2×2 and a stride of 2, the length and width of the input feature are halved.

[0054] 4) C4 includes two convolutional layers and one maximum pooling layer. The input size of the two convolutional layers is 16×19, the convolution kernel size is 3×3, the number of convolution kernels is 512, the stride is 1, the edge padding is 1, and the output is 16×19. Then the Relu activation function is added. Then, after passing through the maximum pooling layer with a kernel size of 2×2 and a stride of 2, the length and width of the input feature are halved.

[0055] 5) FC includes three fully connected layers. First, the C4 output features are flattened to obtain a B (number of samples in each round of training) × (512*8*9) tensor. Then it is sent to the linear change layer nn.Linear to obtain a B×4096 tensor. Then the activation function Relu and the dropout layer are added, and the dropout ratio is 0.5. After another layer of linear change layer, activation function Relu and dropout layer, the dropout ratio is 0.5, a B×1024 tensor is obtained. Finally, after a layer of linear change layer, a B×3 tensor is obtained, and then the final probability distribution of tongue coating moisturizing category is obtained through the softmax function.

[0056] 6) G1, G2 and G3 are all gated convolutions, all containing three layers of convolution and activation function sigmoid. The convolution kernel size of the first convolution layer of G1 is 3×3, the stride is 1, the padding size is 1, the number of input channels is 1, and the number of output channels is 8; the convolution kernel size of the second convolution layer is 3×3, the stride is 4, the padding size is 1, and the number of input and output channels is the same; the last convolution layer uses a 1×1 convolution kernel and the number of output channels is 1. The convolution kernel size of each convolution layer of G2 and G3 is the same as that of G1, except that the input of the first convolution layer of G2 comes from the output of the second convolution layer of G1, the number of output channels is 16, and the stride of the second convolution kernel of G2 is 2; the input of the first convolution layer of G3 comes from the output of the second convolution layer of G2, the number of output channels is 32, and the stride of the second convolution kernel of G3 is 2.

[0057] Step 4: Implement tongue segmentation based on tongue segmentation network

[0058] 1) Training process: The visible light-polarization angle images in the training set are randomly inverted and randomly cropped. Then they are sent to the tongue segmentation network in batches, where the encoder of the network has been pre-trained on the ImageNet dataset. After the pixel-by-pixel binary classification probability of a batch of images is obtained by the tongue segmentation network, it is input into the cross entropy loss function with the real tongue segmentation map, as shown in formula (1), Y i represents the true category (0 represents the non-tongue area, 1 represents the tongue area), y i Represents the predicted value of the network output after softmax regression, and N represents the number of pixels.

[0059]

[0060] Subsequently, the loss error gradient is back-propagated, and the AdamW optimizer is used to update the network parameters to minimize the loss function L1 and complete a round of optimization. The above process is repeated to make the network output approach the true value. During the experiment, the learning rate warm-up and Poly strategy are used to adjust the learning rate. The momentum parameter is set to 0.9, the weight decay factor is 0.01, and the batch size is set to 4. The training is completed after 500 iterations. The number of iterations in the warm-up phase is 10, and the learning rate lrs1 in this phase increases linearly. The specific formula is as follows:

[0061]

[0062] After the warm-up phase, the Poly strategy is used to adjust the learning rate lrs2. The specific formula is as follows:

[0063]

[0064] In formulas (2) and (3), the base learning rate base_lr is 3e-5, warmup_epoch is 10, cur_epoch is the current iteration number, num_epoch is 500, and power is 0.9.

[0065] 2) Testing process: Send the visible light-polarization angle images in the test set to the trained tongue segmentation network one by one to obtain the predicted tongue area. The segmentation accuracy of the tongue is evaluated by the intersection over union (IoU) of the predicted value and the true value, as shown in Formula 4. TP represents the number of pixels predicted to be 1 and actually 1. FN represents the number of pixels predicted to be 0 and actually 1. FP represents the number of pixels predicted to be 1 and actually 0. If the tongue segmentation accuracy IoU is greater than 95%, step 5 can be performed.

[0066]

[0067] Step 5: Identify the dryness or moistness of the tongue coating

[0068] 1) Training process: The visible light-polarization angle images in the training set are input into the trained tongue segmentation network one by one to obtain the tongue segmentation map. Then, the R1 parameters are frozen, the visible light image is sent to the R1 feature extraction layer, and the features output by R1 are multiplied by the tongue segmentation map to obtain the tongue area features. At the same time, the polarization map is multiplied by the segmentation map to obtain the tongue area. Both are then input into the tongue coating moisturizing and dryness recognition network in batches. The moisture bright spot area features in the tongue polarization map are extracted by gated convolution, and the deep features of the moisture bright spot area in the tongue visible light map are mined by a multi-layer neural network. Subsequently, the predicted probability distribution of the tongue coating moisturizing and dryness category is obtained through the fully connected layer and softmax. The predicted value and the actual moisturizing and dryness category are sent to the cross entropy loss function, as shown in formula (5), where c represents the category, N represents the number of pixels, and Y represents the number of pixels. icIndicates the true one-hot encoding of the sample, y ic It represents the predicted probability that the observed sample belongs to category c.

[0069]

[0070] The error gradient is back-propagated according to the loss value L2, and the AdamW optimizer is used to update the network parameters to minimize the loss function. The learning rate is 0.001, the batch size is set to 4, and the training is completed after 800 iterations.

[0071] 2) Test process: Send the visible light-polarization angle images in the test set to the trained tongue segmentation network one by one to obtain the tongue segmentation map. Then, the visible light features and polarization map after R1 are multiplied with the tongue segmentation map and sent to the tongue coating moistness and dryness recognition network, and the tongue coating moistness and dryness prediction category is obtained at the output of the network. Compare the predicted category with the true category, calculate the F1 scores of moist, slippery and dry coatings, and the overall Micro-F1, and evaluate the prediction results. The specific calculation formula is as follows:

[0072]

[0073]

[0074]

[0075]

[0076]

[0077]

[0078] In formulas (6), (7), (9), and (10), l i represents the number of samples predicted by the model to be of the i-th class and actually belonging to the i-th class, m i Indicates the number of samples predicted by the model as the i-th class, n i It represents the number of samples that actually belong to the i-th category, and k represents the number of categories.

Claims

1. A tongue segmentation and tongue coating moistness recognition method based on multimodal image fusion, characterized in that The following steps are involved: Step 1: Multimodal tongue image acquisition Step 2: Build a tongue segmentation network The specific parameters of the tongue segmentation constructed are as follows: 1) R1, R2, R3, R4 and P1, P2, P3, P4 are all feature extraction layers; among them, Ri extracts the visible light tongue image features, and Pi extracts the polarization angle tongue image features; Ri and Pi have the same structure; Ri includes an overlapping block merge, N effective self-attention and hybrid feedforward networks; among them, the R1 layer receives a visible light image with a size of 512×612 and a channel number of 3; the R1 layer performs overlapping block merge processing on the original tongue image through a 7×7 convolution kernel, a two-dimensional convolution with a step size of 4 and an edge padding of 3, and layer normalization; R2, R3, and R4 all set the image block size to 3×3, and use a 3×3 convolution kernel, a step size of 2, and a two-dimensional convolution with an edge padding of 1, combined with layer normalization, to merge overlapping blocks; the input sizes of R2, R3, and R4 are 128×153, 64×76, and 32×33, respectively, the number of input channels is 64, 128, and 320, respectively, and the number of output channels is 128, 320, and 512; The self-attention calculation includes linear embedding and self-attention mechanism. In the linear embedding process of Ri, the feature B (number of samples per round of training) × C (number of channels) × H (height) × W (width) after merging the overlapping blocks is first flattened to B × C × (H*W), and then converted to B × (H*W) × C and passed through the fully connected layer to obtain Q (query), K (key) and V (value). The number of output channels of the fully connected layer is the same as the number of input channels. Then, Ri captures the correlation between different feature blocks through a multi-head self-attention mechanism. The number of heads of R1, R2, R3, and R4 are 1, 2, 5, and 8 respectively. The implementation process of the self-attention mechanism can be expressed as: The effective number of self-attention N in R1, R2, R3, and R4 is 3, 4, 6, and 3 respectively; the features after the self-attention mechanism are added to the original input element by element and sent to the hybrid feedforward network; in the hybrid feedforward network, for the R1, R2, R3, and R4 feature extraction layers, the number of channels of the feature map is expanded to 3 times the number of input channels using a fully connected layer; then, position encoding is achieved through a 3×3 convolution kernel, a step size of 1, and a depthwise convolution with an edge padding of 1; after that, the number of output channels is reduced to one-third of the number of input channels through the GELU activation function and the fully connected layer in turn; 2) During the feature extraction process, the multi-modal feature fusion operation is performed synchronously step by step as follows; First, in the channel dimension, the visible light image feature r and polarization angle image feature p obtained by the feature extraction layer are randomly divided into 3 groups; for each group of features, different sizes of depth-wise convolution are used for processing; group1 features are subjected to two cascaded depth-wise convolutions to obtain r1 and p1, including convolutions with a kernel size of 1×3, a step size of 1, 1 column padded on the left and right, and a padding value of 0, and convolutions with a step size of 3×1, a step size of 1, 1 row padded on the top and bottom, and a padding value of 0; group2 features are also subjected to two cascaded depth-wise convolutions to obtain r2 and p2, including convolutions with a kernel size of 1×5, a step size of 1, 2 columns padded on the left and right, and a padding value of 0, and convolutions with a step size of 5×1, a step size of 1, 2 rows padded on the top and bottom, and a padding value of 0; group3 features are subjected to depth-wise convolutions with a kernel size of 1×7, a step size of 1, 3 columns padded on the left and right, and a padding value of 0, and depth-wise convolutions with a step size of 7×1, a step size of 1, 3 rows padded on the top and bottom, and a padding value of 0 to obtain r3 and p3; Subsequently, cross-modal multi-scale interaction is performed; in order to achieve full interaction between different modalities and channels, three branches are used to complete multi-modal feature interaction; among them, in L p1 In the branch, the features of r2, r3 and p1 are concatenated and then fused through a 1×1 convolution kernel, with the number of input channels equal to the sum of the number of channels of r2, r3 and p1, and the number of output channels equal to the number of channels of p1; in L p2 In the branch, the features of r1, r3 and p2 are concatenated and then fused through a 1×1 convolution kernel, with the number of input channels equal to the sum of the number of channels of r1, r3 and p2, and the number of output channels equal to the number of channels of p2; in L p3 In the branch, the features of r1, r2 and p3 are concatenated, and then fused through a convolution with a 1×1 convolution kernel, the number of input channels is the sum of the number of channels of r1, r2 and p3, and the number of output channels is equal to the number of channels of p3; then the output features of the above three branches are linearly integrated through a point-by-point convolution with a convolution kernel size of 1×1 and the number of output channels equal to the number of input channels; and the feature p after interaction with r is obtained through batch normalization and sigmoid activation function E ; In order to retain more original information, p E It is not directly fused, but multiplied with p, then added with r and sent to the channel compression module; Taking r as the subject, the feature r after interacting with p E It also requires three branches to complete; r1 In the branch, the features of p2, p3 and r1 are concatenated and then fused through a 1×1 convolution kernel, with the number of input channels equal to the sum of the number of channels of p2, p3 and r1, and the number of output channels equal to the number of channels of r1; in L r2 In the branch, the features of p1, p3 and r2 are concatenated and then fused through a 1×1 convolution kernel, with the number of input channels equal to the sum of the number of channels of p1, p3 and r2, and the number of output channels equal to the number of channels of r2; in L r3 In the branch, the features of p1, p2 and r3 are concatenated, and then fused through a convolution with a 1×1 convolution kernel, the number of input channels is the sum of the number of channels of p1, p2 and r3, and the number of output channels is equal to the number of channels of r3; then the output features of the above three branches are linearly integrated through a point-by-point convolution with a convolution kernel size of 1×1 and the number of output channels equal to the number of input channels; and finally the feature r after interaction with p is obtained through batch normalization and sigmoid activation function E ; In order to retain more original information, r E It is not directly fused, but multiplied by r, then added to p and sent to the channel compression module; Finally, the interactive multimodal features are fused through channel compression to obtain the fused feature F; channel compression consists of channel splicing and three convolution layers; the convolution kernel of the first convolution is 1×1, the step size is 1, and the number of output channels is half of the number of input channels; the kernel size of the second depth-wise convolution is 3×3, the step size is 1, and the input features are padded with 1 layer on each side, and the padding value is 0; then the Relu activation function is added to increase the nonlinear expression ability of the model; the convolution kernel of the third convolution is 1×1, the step size is 1, and the number of input and output channels is the same; 3) In the process of feature extraction and fusion, the fused features F at each level are processed by the detail enhancement module (DFE) and the semantic enhancement module (SFE). i (i=1,2,3,4) for feature enhancement; Among them, the detail enhancement module structure consists of two branches; Branch, first depth separable convolution DSConv on the previous level feature F i-1 To further extract features, the convolution kernel size is 3×3, the stride is 2, the padding size is 1, the padding value is 0, and the number of output channels is half of the number of input channels; then batch normalization and activation function ReLU are added; the convolution kernel size of the second convolution is 1×1, the stride is 1, there is no padding, the number of output channels is twice the number of input channels, and then batch normalization is added; The branch consists of an average pooling layer and two convolutional layers. The average pooling layer converts the previous level feature F into a shape of B (number of samples per training round) × C (number of channels) × H (height) × W (width). i-1 Features are compressed to B×C×1; The convolution kernel size of the first convolution of the branch is 1×1, the stride is 1, there is no padding, and the number of output channels is half of the number of input channels; then batch normalization and activation function Relu are added; The convolution kernel size of the second convolution of the branch is 1×1, the stride is 1, there is no padding, and the number of output channels is twice the number of input channels; then batch normalization is added; finally, and The results of the branches are added and convolved with a kernel size of 1×1, a step size of 1, and an input channel number of F. i-1 The number of channels and the number of output channels are F i The number of channels and unfilled convolution further fuse the results of the two branches; finally, after passing the sigmoid activation function, it is combined with F i Multiply element by element to obtain the features after detail enhancement The semantic enhancement module is as follows: i+1 The branch generates a gating signal through a convolution with a convolution kernel of 1×1, an output channel of 1, a step size of 1, no padding, and a sigmoid activation function; and the gating signal is combined with F i Multiply, enhance F i Semantic information of F i+1 To improve the integrity of the transfer, a parallel fusion flow is also introduced; specifically, first, F i+1 Upsampling is performed by bilinear interpolation, so that F i+1 The spatial resolution and F i Same as F i Channel concatenation is performed; then, the input channel is compressed to the same size as F through a convolution with a 3×3 convolution kernel, a stride of 2, and a padding size of 1 on all sides. i The same number of channels, batch normalization and ReLU activation function are added; finally, the results of the two branches are added together to obtain the semantically enhanced features Finally and Add element by element to get 4) Obtain tongue segmentation map through MLP MLP consists of two fully connected layers. The first fully connected layer first transforms the enhanced features of each level. (i=1,2,3,4) for channel splicing, and then reduce the number of input feature channels to one-fourth of the original through convolution with a convolution kernel size of 1×1; then add batch normalization and Relu activation function to improve the generalization ability of the model, and add a dropout layer with a dropout ratio of 0.1; the second layer of full connection reduces the input channel to 2 through convolution with a convolution kernel size of 1×1, and then obtains the tongue prediction map through the softmax function; Step 3: Build a tongue coating moistness and dryness recognition network The constructed tongue coating moisturizing and dryness identification network is as follows: C1, C2, C3, and C4 further extract deep features from the underlying features of the tongue visible light image, G1, G2, and G3 are responsible for extracting the moisture bright spot area features of the tongue polarization image, and FC predicts the final tongue coating moisturizing and dryness category; the detailed parameters of each layer are as follows: C1 includes two convolutional layers and one maximum pooling layer. The input size of the two convolutional layers is 128×153, the convolution kernel size is 3×3, the number of convolution kernels is 128, the stride is 1, the edge padding is 1, and the output is 128×153. Then the Relu activation function is added. Then, after passing through the maximum pooling layer with a kernel size of 2×2 and a stride of 2, the length and width of the input feature are halved. C2 includes two convolutional layers and one maximum pooling layer. The input size of the two convolutional layers is 64×76, the convolution kernel size is 3×3, the number of convolution kernels is 256, the stride is 1, the edge padding is 1, and the output is 64×76. Then the Relu activation function is added. Then, after passing through the maximum pooling layer with a kernel size of 2×2 and a stride of 2, the length and width of the input feature are halved. C3 includes two convolutional layers and one maximum pooling layer. The input size of the two convolutional layers is 32×38, the convolution kernel size is 3×3, the number of convolution kernels is 512, the stride is 1, the edge padding is 1, and the output is 32×38. Then the Relu activation function is added. Then, the maximum pooling layer with a kernel size of 2×2 and a stride of 2 is passed to halve the length and width of the input feature. C4 includes two convolutional layers and one maximum pooling layer. The input size of the two convolutional layers is 16×19, the convolution kernel size is 3×3, the number of convolution kernels is 512, the stride is 1, the edge padding is 1, and the output is 16×19. Then the Relu activation function is added. Then, after passing through the maximum pooling layer with a kernel size of 2×2 and a stride of 2, the length and width of the input feature are halved. FC includes three fully connected layers. First, the C4 output features are flattened to obtain a B (number of samples per round of training) × (512*8*9) tensor; then it is sent to the linear change layer nn.Linear to obtain a B×4096 tensor; then the activation function Relu and the dropout layer are added, and the dropout ratio is 0.5; after another linear change layer, activation function Relu and dropout layer, the dropout ratio is 0.5, a B×1024 tensor is obtained; finally, after a linear change layer, a B×3 tensor is obtained, and then the final tongue coating moisturizing category probability distribution is obtained through the softmax function; G1, G2 and G3 are all gated convolutions, all containing three layers of convolution and activation function sigmoid; the convolution kernel size of the first layer of convolution of G1 is 3×3, the stride is 1, the padding size is 1, the number of input channels is 1, and the number of output channels is 8; the convolution kernel size of the second layer of convolution is 3×3, the stride is 4, the padding size is 1, and the number of input and output channels is the same; the last layer of convolution uses a 1×1 convolution kernel and the output channel is 1; the convolution kernel size of each layer of convolution of G2 and G3 is the same as that of G1, the difference is that the input of the first layer of convolution of G2 comes from the output of the second layer of convolution of G1, its output channel number is 16, and the stride of the second layer of convolution kernel of G2 is 2; the input of the first layer of convolution of G3 comes from the output of the second layer of convolution of G2, its output channel number is 32, and the stride of the second layer of convolution kernel of G3 is 2; Step 4: Implement tongue segmentation based on tongue segmentation network Training process: The visible light-polarization angle images in the training set are randomly inverted and randomly cropped; then they are sent to the tongue segmentation network in batches, where the encoder of the network has been pre-trained on the ImageNet dataset; after the pixel-by-pixel binary classification probability of a batch of images is obtained by the tongue segmentation network, it is input into the cross entropy loss function with the real tongue segmentation map, as shown in formula (1), Y i represents the true category (0 represents the non-tongue area, 1 represents the tongue area), y i Represents the predicted value of the network output after softmax regression, and N represents the number of pixels; Subsequently, the loss error gradient is back-propagated, and the AdamW optimizer is used to update the network parameters to minimize the loss function L1 and complete a round of optimization. The above process is repeated to make the network output approach the true value. During the experiment, the learning rate warm-up and Poly strategy are used to adjust the learning rate. The momentum parameter is set to 0.9, the weight decay factor is 0.01, and the batch size is set to 4. The training is completed after 500 iterations. The number of iterations in the warm-up phase is 10, and the learning rate lrs1 in this phase increases linearly. The specific formula is as follows: After the warm-up phase, the Poly strategy is used to adjust the learning rate lrs2. The specific formula is as follows: In formulas (2) and (3), the base learning rate base_lr is 3e-5, warmup_epoch is 10, cur_epoch is the current iteration number, num_epoch is 500, and power is 0.9; Testing process: Send the visible light-polarization angle images in the test set to the trained tongue segmentation network one by one to obtain the predicted tongue area; evaluate the segmentation accuracy of the tongue by the intersection over union (IoU) of the predicted value and the true value, as shown in formula (4); TP represents the number of pixels predicted to be 1 and actually 1; FN represents the number of pixels predicted to be 0 and actually 1; FP represents the number of pixels predicted to be 1 and actually 0; If the tongue segmentation accuracy IoU is greater than 95%, proceed to step 5; Step 5: Identify the moistness or dryness of the tongue coating Training process: The visible light-polarization angle images in the training set are input into the trained tongue segmentation network one by one to obtain the tongue segmentation map; then the R1 parameters are frozen, the visible light image is sent to the R1 feature extraction layer, and the features output by R1 are multiplied with the tongue segmentation map to obtain the tongue area features, and the polarization map is multiplied with the segmentation map to obtain the tongue area; the two are then batch-input into the tongue coating moisturizing and dryness recognition network; the moisture bright spot area features in the tongue polarization map are extracted by gated convolution, and the deep features of the moisture bright spot area in the tongue visible light map are mined by a multi-layer neural network; then, the predicted tongue coating moisturizing and dryness category probability distribution is obtained through the fully connected layer and softmax; the predicted value and the actual moisturizing and dryness category are sent to the cross entropy loss function, as shown in formula (5), c represents the category, N represents the number of pixels, and Y represents the number of pixels. ic Indicates the true one-hot encoding of the sample, y ic Represents the predicted probability that the observed sample belongs to category c; The AdamW optimizer is used to update the network parameters to minimize the loss function; the learning rate is 0.001, the batch size is set to 4, and the training is completed after 800 iterations; Testing process: The visible light-polarization angle images in the test set are sent into the trained tongue segmentation network one by one to obtain the tongue segmentation map; then the visible light features and polarization map passed through R1 are multiplied with the tongue segmentation map and sent to the tongue coating moistness recognition network, and the tongue coating moistness prediction category is obtained at the network output end.

Citation Information

Cited By

  • Target detection system, target detection method and corresponding training method

    CN119963814A

  • Target echo classification method and system based on multi-modal data fusion

    CN120524418A

  • Tongue picture segmentation identification method based on edge region dynamic fusion

    CN121616832A

  • Power network fault early warning method based on large model

    CN122020392A

  • Large model-based power network fault early warning method

    CN122020392B