A tongue image segmentation and recognition method based on dynamic fusion of edge regions

Abstract

The application discloses a tongue image segmentation and recognition method based on edge region dynamic fusion, and relates to the technical field of image segmentation, which comprises the following steps: acquiring a tongue image; inputting the tongue image into a tongue image segmentation and recognition network model; extracting main region features from the tongue image through a main region encoder of the tongue image segmentation and recognition network model; dynamically fusing the main region features through a main region feature dynamic fusion module to obtain main region dynamic features and main region feature dynamic fusion features; and performing image segmentation through a main image segmenter on the main region dynamic features and the main region feature dynamic fusion features which are spliced according to channels and subjected to convolution to obtain a tongue image segmentation result. The application can realize accurate segmentation and recognition of special tongue images and ensure that there is no omission in edge regions.

Description

TECHNICAL FIELD

[0001] The application relates to a tongue image segmentation and recognition method based on dynamic fusion of edge regions and belongs to the technical field of image segmentation. BACKGROUND

[0002] Tongue fur is a core observation object of traditional Chinese medicine diagnosis, and its morphology (thickness, dryness), color (white, yellow, gray black), distribution (whole fur, local fur) and other characteristics are directly related to the function and pathological state (such as damp heat, cold, food accumulation, etc.) of human body viscera, but traditional Chinese medicine tongue diagnosis relies on the subjective experience of doctors and is easily affected by light, observation angle and the like.

[0003] At present, some methods use a network model to segment tongue images, and after segmentation, pixel-level features (such as area ratio, color mean value, texture complexity) of tongue fur can be extracted, qualitative description is converted into quantitative data, and a basis is provided for objective diagnosis. As a key preprocessing step for intelligentization of traditional Chinese medicine tongue diagnosis, through the tongue fur features after segmentation, an associated model with diseases (such as gastritis, diabetes, respiratory tract infection, etc.) can be established.

[0004] In the tongue segmentation task, although UNet and its derivative models (such as UNet++, ResUNet) are good at medical image semantic segmentation due to the "encoder-decoder" structure, they still face the following core challenges due to the influence of the characteristics of the tongue itself and the collection environment: the core advantage of UNet is to fuse high and low layer features through a skip connection, but there are two problems in tongue segmentation that weaken its effect: fuzzy boundary leads to inaccurate positioning: the transition zone of the tongue edge and the lips, gums (such as similar mucosa color), and the tongue edge saliva reflection band have no clear pixel difference, and the down-sampling process of UNet easily loses the edge details, and even the skip connection also cannot accurately restore the pixel attribution of the "fuzzy boundary" (such as mistakenly dividing the tongue edge mucosa into the background); the adaptation problem of individual morphological differences: the tongue size of children and adults and the width and thickness of obese and emaciated people differ significantly, and the fixed receptive field and feature extraction scale of UNet cannot be compatible with "multi-scale tongue", for example, for small tongues of children, the model is easy to over-segment (dividing the surrounding lips into the tongue); for wide and thick tongues, the edge region is easy to be missed. SUMMARY

[0005] The purpose of the present application is to overcome the deficiencies in the prior art and provide a tongue image segmentation and recognition method based on dynamic fusion of edge regions, which can realize accurate segmentation and recognition of special tongue image and ensure that there is no omission in the edge region.

[0006] To achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0007] The present application provides a tongue image segmentation and recognition method based on dynamic fusion of edge regions, comprising:

[0008] Obtain tongue image;

[0009] The tongue image is input into the tongue image segmentation and recognition network model;

[0010] The main region features are extracted from the tongue image by the main region encoder of the tongue image segmentation and recognition network model.

[0011] The main region features are dynamically fused using the main region feature dynamic fusion module to obtain the main region dynamic features and the main region feature dynamic fusion features;

[0012] The last feature in the main region features and the dynamic fusion feature of the main region features are concatenated by channel and then convolved together with the dynamic features of the main region. They are then passed through the main image segmenter to perform image segmentation to obtain the tongue image segmentation result.

[0013] Furthermore, the tongue image segmentation and recognition network model also includes an auxiliary region encoder and an edge image extraction module during the pre-training stage;

[0014] The edge image extraction module is used to extract the edge images of the main image and the auxiliary image using a gradient operator to obtain the main edge image and the auxiliary edge image. Its output is connected to the shared edge encoder, which is used to extract the edge features in the main edge image and the auxiliary edge image to obtain the main edge features and the auxiliary edge features.

[0015] The auxiliary region encoder is used to extract features from the auxiliary image to obtain auxiliary region features. Its output is connected to the input of the auxiliary region feature dynamic fusion module. The auxiliary region feature dynamic fusion module is used to dynamically fuse the auxiliary region features to obtain auxiliary region dynamic features and auxiliary region feature dynamic fusion features.

[0016] The outputs of the edge image extraction module, the shared edge encoder, and the auxiliary region encoder are all connected to the input of the auxiliary feature cross-fusion module. The auxiliary edge image, the last auxiliary edge feature, the last auxiliary region feature, and the last main edge feature are cross-fused to obtain the auxiliary feature cross-fusion feature.

[0017] The outputs of the edge image extraction module, the shared edge encoder, and the main region encoder are all connected to the input of the main feature cross-fusion module. The main edge image, the last main edge feature, the last main region feature, and the last auxiliary edge feature are cross-fused to obtain the main feature cross-fusion feature.

[0018] The outputs of the auxiliary region encoder and the auxiliary region feature dynamic fusion module are connected to the auxiliary image segmenter. The last auxiliary region feature and the auxiliary region feature dynamic fusion feature are concatenated by channel and then convolved together with the auxiliary region dynamic feature and passed through the auxiliary image segmenter to obtain the auxiliary image segmentation result.

[0019] Furthermore, the main region encoder, auxiliary region encoder, and shared edge encoder all employ UNet encoders, and the main image segmenter and auxiliary image segmenter both employ UNet decoders.

[0020] Furthermore, both the main region feature dynamic fusion module and the auxiliary region feature dynamic fusion module include multiple processing lines. Each processing line corresponds to a different input feature and includes two branches. One branch includes a dynamic convolution unit within the feature and a 1×1 convolution unit connected in sequence, and the other branch includes a 1×1 convolution unit and a dynamic convolution unit between features connected in sequence. The outputs of the two branches are merged, upsampled, and then spliced ​​and output in the channel dimension.

[0021] The data processing procedures for the intra-feature dynamic convolutional units and inter-feature dynamic convolutional units include:

[0022] The spatial dimensions of the input feature map are averaged, the global context is extracted, and attention weights are calculated using a fully connected layer.

[0023] The learnable convolutional kernels are weighted and combined according to attention weights to obtain dynamic output.

[0024] Furthermore, the data processing procedure of the main feature cross-fusion module includes:

[0025] The main edge image is thresholded and downsampled to the same size as the main edge features to obtain the downsampled main edge image;

[0026] Randomly sample several non-zero pixels from the downsampled main edge image and record their positions;

[0027] Extract the pixel features at non-zero pixel positions of the main edge features to construct the main edge feature map;

[0028] Flatten the main region features and auxiliary edge features to obtain the main region feature map and auxiliary edge feature map, respectively.

[0029] The main edge feature map and the main region feature map are fused within the main edge region to obtain the fused feature within the main edge region.

[0030] The main region feature map and the auxiliary edge feature map are fused together to obtain the fused feature of the auxiliary edge feature and the main region feature.

[0031] The auxiliary feature cross-fusion feature is obtained by adding the fusion features inside the main edge region, the auxiliary edge features, and the fusion features of the main region features. The auxiliary feature cross-fusion feature has the same shape as the main edge feature.

[0032] The data processing procedure of the auxiliary feature cross-fusion module includes:

[0033] The auxiliary edge image is thresholded and downsampled to the same size as the auxiliary edge features to obtain the downsampled auxiliary edge image;

[0034] Randomly sample several non-zero pixels from the downsampled auxiliary edge image and record their positions;

[0035] Extract the pixel features at non-zero pixel positions to construct the auxiliary edge feature map;

[0036] Flattening the auxiliary region features and the main edge features yields the auxiliary region feature map and the main edge feature map, respectively.

[0037] The auxiliary edge feature map and the auxiliary region feature map are fused within the auxiliary edge region to obtain the fused feature within the auxiliary edge region.

[0038] The auxiliary region feature map and the main edge feature map are fused together to obtain the fused feature of the main edge feature and the auxiliary region feature.

[0039] The main feature cross-fusion feature is obtained by adding the fusion features inside the auxiliary edge region and the fusion features of the main edge feature and the auxiliary region feature. The main feature cross-fusion feature has the same shape as the auxiliary edge feature.

[0040] Furthermore, the expression for fusing the main edge feature map and the main region feature map within the main edge region to obtain the fused feature within the main edge region is as follows:

[0041] ;

[0042] in, This represents the attention matrix fused within the main edge region. Represents the normalization function. Represents the feature map of the main region. Represents the dimension of the attention matrix. , , , , , Both represent learnable weight matrices. This represents a learnable bias. Indicates the first edge in the downsampled main edge image A non-zero pixel position, Indicates the location of the main region feature map. pixel features, This indicates the fusion characteristics within the main edge region;

[0043] The expression for fusing the main region feature map and the auxiliary edge feature map, resulting in the fused feature of the auxiliary edge features and the main region features, is as follows:

[0044] ;

[0045] in, This represents the attention matrix that fuses auxiliary edge features with main region features. Represents auxiliary edge feature map, , , Both represent learnable weight matrices. This represents a learnable bias. This represents the fusion of auxiliary edge features and main region features;

[0046] The expression for fusing the auxiliary edge feature map and the auxiliary region feature map within the auxiliary edge region to obtain the fused feature within the auxiliary edge region is as follows:

[0047] ;

[0048] ;

[0049] in, This represents the attention matrix used for fusion within the auxiliary edge regions. Represents the feature map of the auxiliary region. , , , Both represent learnable weight matrices. This represents a learnable bias. The third auxiliary edge image in the downsampled image A non-zero pixel position, Indicates the location of the auxiliary region feature map pixel features, Indicates the fusion features within the auxiliary edge region;

[0050] The expression for fusing the auxiliary region feature map and the main edge feature map to obtain the fused feature of the main edge feature and auxiliary region feature is as follows:

[0051] ;

[0052] in, This represents the attention matrix that fuses the main edge features and auxiliary region features. Represents the main edge feature map. This represents the learnable weight matrix. This represents a learnable bias. This represents the fusion of main edge features and auxiliary region features.

[0053] Furthermore, it also includes a main image decoder and an auxiliary image decoder. The input end of the main image decoder is connected to the output end of the main feature cross-fusion module and the shared edge encoder, and the input end of the auxiliary image decoder is connected to the output end of the auxiliary feature cross-fusion module and the shared edge encoder.

[0054] The main feature cross-fusion feature and the main edge image feature are concatenated by channel and then input into the main image decoder to reconstruct the reconstructed main image;

[0055] The auxiliary feature cross-fusion feature and the auxiliary edge image feature are concatenated by channel and then input into the auxiliary image decoder to reconstruct the reconstructed auxiliary image.

[0056] Both the main image decoder and the auxiliary image decoder use the Unet decoder.

[0057] Furthermore, the pre-training method for the tongue image segmentation and recognition network model includes:

[0058] Obtain the main image dataset and the auxiliary image dataset;

[0059] The main image dataset and the auxiliary image dataset are used as inputs to train the tongue image segmentation and recognition network model. During the training process, the loss function is calculated until the loss function is less than a set threshold, and then the pre-trained tongue image segmentation and recognition network model is obtained.

[0060] Furthermore, the expression for the main image dataset is: ,in, Represents the main image dataset. Indicates the first The main image, Indicates the first Segmentation labels for each main image. Indicates the number of main images;

[0061] The expression for the auxiliary image dataset is: ,in, Represents an auxiliary image dataset. Indicates the first Auxiliary image, Indicates the first Segmentation labels for auxiliary images, Indicates the number of auxiliary images.

[0062] Furthermore, the calculation expression for the loss function is as follows:

[0063] ;

[0064] ;

[0065] ;

[0066] ;

[0067] in, Indicates the total loss. Indicates the partition loss. Represents the cross-entropy loss function. This represents the auxiliary image segmentation result. This represents the segmentation result of the main image. Segmentation labels for auxiliary images, The segmentation label represents the main image. Indicates the losses incurred during reconstruction. Describing the L1 norm, Indicates the main image. Indicates auxiliary image, Indicates the cross-fusion features of the main image. Indicates the main edge features, Indicates the cross-fusion features of auxiliary images. Indicates auxiliary edge features, Indicates the main region decoder. Indicates the auxiliary region decoder, Indicates channel splicing. Indicates distance loss. Represents the L2 norm. Indicates the main region characteristics, This indicates the dynamic fusion of features in the main region. Indicates auxiliary region features, This indicates the dynamic fusion of auxiliary region features. This represents a 1×1 convolution operation.

[0068] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0069] This invention designs a dynamic feature fusion module for the main region. It adaptively focuses on key regions through dynamic convolution within and between features, and achieves multi-scale main region feature fusion by fusing features. The dynamic weight generation submodule within the module automatically calculates the weight distribution based on the input features, which is more adaptable to the complexity of medical images than fixed-weight fusion methods. This design is particularly valuable in tongue image processing because the position and morphology of tongue structures vary significantly among individuals.

[0070] This invention designs a main feature cross-fusion module. The input includes the main edge image, main edge features, main region features, and auxiliary edge features. It calculates the attention map between the main edge features and the main region features and the attention map between the auxiliary edge features and the main region features. It cross-fused the main edge image, main edge features, main region features, and auxiliary edge features, reducing the influence of factors such as lighting, viewing angle, and individual morphological differences between images.

[0071] This invention designs an auxiliary region feature dynamic fusion module, which adaptively focuses on key regions through dynamic convolution within and between features, and achieves multi-scale auxiliary region feature fusion by fusing features.

[0072] This invention designs an auxiliary feature cross-fusion module. The input includes auxiliary edge image, auxiliary edge feature, auxiliary region feature, and main edge feature. It calculates the attention map between auxiliary edge feature and auxiliary region feature and the attention map between main edge feature and auxiliary region feature. It cross-fused the auxiliary edge image, auxiliary edge feature, auxiliary region feature, and main edge feature, reducing the influence of factors such as lighting, viewing angle, and individual morphological differences between images. Attached Figure Description

[0073] Figure 1 This is a schematic diagram of the framework of a tongue image segmentation and recognition method based on dynamic fusion of edge regions in one embodiment of the present invention;

[0074] Figure 2 This is a schematic diagram of the main region feature dynamic fusion module of the tongue image segmentation and recognition network model in an embodiment of the tongue image segmentation and recognition method based on edge region dynamic fusion of the present invention.

[0075] Figure 3 This is a schematic diagram of the auxiliary region feature dynamic fusion module of the tongue image segmentation and recognition network model in a tongue image segmentation and recognition method based on edge region dynamic fusion in one embodiment of the present invention.

[0076] Figure 4 This is a schematic diagram of the main feature cross-fusion module of the tongue image segmentation and recognition network model in the tongue image segmentation and recognition method based on dynamic fusion of edge regions in one embodiment of the present invention;

[0077] Figure 5 This is a schematic diagram of the auxiliary feature cross-fusion module of the tongue image segmentation and recognition network model in a tongue image segmentation and recognition method based on dynamic fusion of edge regions in one embodiment of the present invention.

[0078] Figure 6 This is a schematic diagram showing the test results of a tongue image segmentation and recognition method based on dynamic fusion of edge regions in one embodiment of the present invention, compared with existing methods. Detailed Implementation

[0079] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0080] Example 1:

[0081] like Figure 1 As shown, this embodiment of the invention provides a tongue image segmentation and recognition method based on dynamic fusion of edge regions, including the following steps:

[0082] Constructing a tongue image segmentation and recognition network model:

[0083] In this embodiment, the tongue image segmentation and recognition network model includes a main region encoder connected in sequence. The main region feature dynamic fusion module and the main image segmenter, and the main region encoder. For the main image (In the application phase, the main image is the tongue image to be tested.) Feature extraction is performed to obtain the main region features. , , This indicates the total number of features; the main region feature dynamic fusion module performs feature fusion on the main region. Dynamic fusion is performed to obtain the dynamic features of the main region. Dynamic fusion features with main region features .

[0084] It should be noted that the tongue image segmentation and recognition network model also includes an auxiliary region encoder during the training phase. and edge image extraction module, auxiliary region encoder For auxiliary images Feature extraction is performed to obtain auxiliary region features. Auxiliary region encoder The output of the auxiliary region feature dynamic fusion module is connected to the input of the auxiliary region feature dynamic fusion module. The auxiliary region feature dynamic fusion module dynamically fuses the auxiliary region features to obtain the auxiliary region dynamic features. Dynamic fusion features with auxiliary region features .

[0085] In this embodiment, both the auxiliary image and the main image are tongue images. The main image can be a tongue image of a patient, such as a child, and the auxiliary image can be a tongue image of another patient, such as an adult.

[0086] The edge image extraction module uses gradient operators to extract the main image. Auxiliary images Obtain the main edge image from the edge image Auxiliary edge images The output of the edge image extraction module is connected to the shared edge encoder, which is used to extract the main edge image. Auxiliary edge images The edge features in the middle correspond to the main edge features. Auxiliary edge features .

[0087] The outputs of the edge image extraction module, the shared edge encoder, and the auxiliary region encoder are all connected to the input of the auxiliary feature cross-fusion module, which is the auxiliary edge image. , No. Auxiliary edge features , No. Features of each auxiliary region , No. Main edge features Input auxiliary feature cross-fusion module outputs auxiliary feature cross-fusion feature Main edge image , No. Main edge features , No. Features of each main region , No. Auxiliary edge features Input main feature cross-fusion module outputs main feature cross-fusion features .

[0088] In this embodiment, the main region encoder, auxiliary region encoder, and shared edge encoder all adopt the encoder structure of UNet.

[0089] To ensure the cross-fusion of main features Contains sufficient information, main features cross-fusion features and the Main edge features After being stitched together by channel, the data is input to the main image decoder. The reconstructed main image is obtained. Similarly, to ensure the cross-fusion of auxiliary features... Contains sufficient information, auxiliary features cross-fusion features and the Auxiliary edge features After being stitched together by channel, the data is input to the auxiliary image decoder. The reconstructed auxiliary image is obtained, where the main image decoder... and auxiliary image decoder The decoder structure of Unet can be used.

[0090] No. Features of each auxiliary region Dynamic fusion features with auxiliary region features After concatenation by channel, the data is fed into a 1×1 convolution, and then combined with the dynamic features of the auxiliary region. Input Auxiliary Image Segmenter Obtain auxiliary image segmentation results . No. Features of each main region Dynamic fusion features with main region features After concatenation by channel, the data is fed into a 1×1 convolution, and then combined with the dynamic features of the main region. Input together into the main image segmenter Obtain the main image segmentation result .

[0091] The following is a detailed description of the internal data processing procedures of each module in the tongue image segmentation and recognition network model.

[0092] The main region feature dynamic fusion module and the auxiliary region feature dynamic fusion module have the same structure, as shown below: Figure 2 , Figure 3 As shown.

[0093] The data processing procedure of the auxiliary region feature dynamic fusion module is as follows:

[0094] Combination Figure 3 First, for each auxiliary region feature , ... Perform dynamic convolution within features: auxiliary region features Global average pooling averages the spatial dimensions of the input feature map to extract the global context, and then uses a fully connected layer to calculate the weights. Its expression is:

[0095] ;

[0096] in, This represents the dynamic convolution weights within the auxiliary region features. Represents the normalization function. , All are weights of fully connected layers. This represents the activation function. This indicates global average pooling.

[0097] Next, attention weights are used to weight and combine the K learnable convolutional kernels to obtain a dynamic output, the expression of which is:

[0098] ;

[0099] in, This represents the output of dynamic convolution within the auxiliary region features. Indicates batch normalization, This represents the convolution operation. express The One element, Indicates the first A learnable convolutional kernel, This is a bias term.

[0100] Then, dynamic convolution is performed between features, and the features of each auxiliary region are... After unifying the number of channels through 1×1 convolutions, global average pooling is performed to average the spatial dimensions of the input feature maps, extracting the global context. Then, a fully connected layer is used to calculate the weights.

[0101] ;

[0102] in, This represents the dynamic convolution weights between features in the auxiliary region. , Indicates the total number of features. , All are weights of fully connected layers. Indicates channel splicing. This represents a 1×1 convolution operation. This represents the auxiliary region features after global average pooling in the dynamic convolution between features.

[0103] Next, attention weights are used to weight and combine the K learnable convolutional kernels to obtain a dynamic output, the expression of which is:

[0104] ;

[0105] in, This represents the output of dynamic convolution between features in the auxiliary region. express The One element, Indicates the first A learnable convolutional kernel, This is a bias term.

[0106] The features generated by intra-feature dynamic convolution and inter-feature dynamic convolution are merged and then upsampled before being concatenated along the channel dimension. This ensures that feature information at all scales is not lost. The expression is as follows:

[0107] ;

[0108] in, Indicates the dynamic characteristics of the auxiliary region. , Indicates the total number of features. Indicates upsampling, This indicates the dynamic fusion of auxiliary region features.

[0109] The data processing procedure of the main region feature dynamic fusion module is as follows:

[0110] Combination Figure 2 First, for each main region feature , ... Perform dynamic convolution within features: main region features Global average pooling averages the spatial dimensions of the input feature map to extract the global context, and then uses a fully connected layer to calculate the weights. Its expression is:

[0111] ;

[0112] in, This represents the dynamic convolution weights within the main region's features. , Indicates the total number of features. , All are weights of fully connected layers. Represents the normalization function. This represents the activation function. This indicates global average pooling.

[0113] Next, attention weights are used to weight and combine the K learnable convolutional kernels to obtain a dynamic output, the expression of which is:

[0114] ;

[0115] in, This represents the output of dynamic convolution within the main region's features. Indicates batch normalization, This represents the convolution operation. express The One element, Indicates the first A learnable convolutional kernel, This is a bias term.

[0116] Then, dynamic convolution is performed between features, and the features of each main region are... After unifying the number of channels through 1×1 convolutions, global average pooling is performed to average the spatial dimensions of the input feature maps, extracting the global context. Then, a fully connected layer is used to calculate the weights.

[0117] ;

[0118] in, This represents the dynamic convolution weights between features in the main region. , Indicates the total number of features. , All are weights of fully connected layers. Indicates channel splicing. This represents a 1×1 convolution operation. This represents the main region features after global average pooling in dynamic convolution between features.

[0119] Next, attention weights are used to weight and combine the K learnable convolutional kernels to obtain a dynamic output, the expression of which is:

[0120] ;

[0121] in, This represents the output of dynamic convolution between features in the main region. express The One element, Indicates the first A learnable convolutional kernel, This is a bias term.

[0122] The features generated by intra-feature dynamic convolution and inter-feature dynamic convolution are merged and then upsampled before being concatenated along the channel dimension. This ensures that feature information at all scales is not lost. The expression is as follows:

[0123] ;

[0124] in, Indicates the dynamic characteristics of the main region. , Indicates the total number of features. Indicates upsampling, This indicates the dynamic fusion of features in the main region.

[0125] The structure of the main feature cross-fusion module is as follows: Figure 4 As shown, its internal data processing includes:

[0126] Main edge image , No. Main edge features , No. Features of each main region , No. Auxiliary edge features As input.

[0127] For the main edge image Use threshold segmentation and downsample to the level of the first segment. Main edge features The same size is used, and then several non-zero pixels are randomly sampled from it and their positions are recorded. , will the Main edge features In position The pixel features are extracted to form the main edge feature map. The first Features of each main region , No. Auxiliary edge features Flattening the images yields the feature maps of the main regions. and auxiliary edge feature maps .

[0128] Main edge feature map and main region feature map The fusion output of the main edge region is used to obtain the fusion feature of the main edge region. Its expression is:

[0129] ;

[0130] in, This represents the attention matrix fused within the main edge region. Represents the normalization function. Represents the feature map of the main region. Represents the dimension of the attention matrix. , , , , , Both represent learnable weight matrices. This represents a learnable bias. Indicates the first edge in the downsampled main edge image A non-zero pixel position, Indicates the location of the main region feature map. pixel features, This indicates the fusion characteristics within the main edge region.

[0131] Main region feature map and auxiliary edge feature maps The auxiliary edge features and the main region features are fused together to produce the fused feature output. Its expression is:

[0132] ;

[0133] in, This represents the attention matrix that fuses auxiliary edge features with main region features. Represents auxiliary edge feature map, , , Both represent learnable weight matrices. This represents a learnable bias. This represents the fusion of auxiliary edge features and main region features.

[0134] Fusion features within the main edge region Features fused with auxiliary edge features and main region features After adding, restore to the original state. Main features with the same shape cross-fusion features .

[0135] The structure of the auxiliary feature cross-fusion module is as follows: Figure 5 As shown, its internal data processing includes:

[0136] For auxiliary edge images Use threshold segmentation and downsample to the level of the first segment. Auxiliary edge features The same size is used, and then several non-zero pixels are randomly sampled from it and their positions are recorded. , will the Auxiliary edge features In position The pixel features are extracted to form the main edge feature map. . No. Features of each auxiliary region , No. Main edge features The auxiliary region feature maps were obtained by flattening them out. Main edge feature map .

[0137] Main edge feature map and auxiliary region feature map The auxiliary edge region is fused into its internal fusion output to obtain the auxiliary edge region internal fusion feature. Its expression is:

[0138] ;

[0139] ;

[0140] in, This represents the attention matrix used for fusion within the auxiliary edge regions. Represents the feature map of the auxiliary region. , , , Both represent learnable weight matrices. This represents a learnable bias. The third auxiliary edge image in the downsampled image A non-zero pixel position, Indicates the location of the auxiliary region feature map pixel features, This indicates the fusion features within the auxiliary edge region.

[0141] Auxiliary region feature map Main edge feature map The main edge features and auxiliary region features are fused together to produce the fused feature output. Its expression is:

[0142] ;

[0143] in, This represents the attention matrix that fuses the main edge features and auxiliary region features. Represents the main edge feature map. This represents the learnable weight matrix. This represents a learnable bias. This represents the fusion of main edge features and auxiliary region features.

[0144] fusion features within auxiliary edge regions Features fused with main edge features and auxiliary region features After adding, restore to the original state. Cross-fusion of auxiliary features with the same shape .

[0145] Next, the constructed tongue image segmentation and recognition network model is pre-trained. The pre-training methods include:

[0146] To obtain the main image dataset and the auxiliary image dataset, the expression for the main image dataset is: ,in, Represents the main image dataset. Indicates the first The main image, Indicates the first Segmentation labels for each main image. The number of main images is represented by the expression for the auxiliary image dataset. ,in, Represents an auxiliary image dataset. Indicates the first Auxiliary image, Indicates the first Segmentation labels for auxiliary images, Indicates the number of auxiliary images.

[0147] The main image dataset and auxiliary image dataset are used as input to train the tongue image segmentation and recognition network model. During training, a loss function is calculated until the loss function is less than a set threshold, at which point the pre-trained tongue image segmentation and recognition network model is obtained. The specific expression of the loss function is as follows:

[0148] ;

[0149] ;

[0150] ;

[0151] ;in, Indicates the total loss. Indicates the partition loss. Represents the cross-entropy loss function. This represents the auxiliary image segmentation result. This represents the segmentation result of the main image. Segmentation labels for auxiliary images, The segmentation label represents the main image. Indicates the losses incurred during reconstruction. Describing the L1 norm, Indicates the main image. Indicates auxiliary image, Indicates the cross-fusion features of the main image. Indicates the main edge features, Indicates the cross-fusion features of auxiliary images. Indicates auxiliary edge features, Indicates the main region decoder. Indicates the auxiliary region decoder, Indicates channel splicing. Indicates distance loss. Represents the L2 norm. Indicates the main region characteristics, This indicates the dynamic fusion of features in the main region. Indicates auxiliary region features, This indicates the dynamic fusion of auxiliary region features. This represents a 1×1 convolution operation.

[0152] In the application phase, after acquiring the image of the tongue to be tested, it is input into the pre-trained tongue segmentation and recognition network model, and the output is the tongue segmentation result.

[0153] The performance of the tongue image segmentation and recognition network model in this embodiment is compared and analyzed below.

[0154] A dataset of 6183 tongue images was collected and randomly divided into a training set of 4328 images, a validation set of 619 images, and a test set of 1236 images. All experiments were conducted using Python 3.8 with PyTorch 1.8 on a single NVIDIA GeForce RTX 3090 GPU with 24GB of memory. The code for the comparison method was adapted from the experimental setup of a publicly available implementation. All models were trained using the Adam optimizer with a learning rate of 1.0 × 10⁻⁴ for 200 epochs.

[0155] To quantitatively evaluate the performance of the proposed method, two mainstream and representative evaluation metrics from segmentation tasks were selected to comprehensively measure the accuracy and completeness of the model in image segmentation. Mean Dice Similarity Coefficient (mDSC): This calculates the overlap between the predicted segmented region and the ground truth region, then averages the DSC values ​​for all categories (e.g., tongue, background), primarily reflecting the "overlap accuracy" of the segmentation results. Mean Intersection over Union (mIoU): This calculates the ratio of the intersection to the union of the predicted and ground truth regions, then averages the IoU values ​​for all categories, comprehensively measuring the "accuracy" and "completeness" of the segmentation; it is the most crucial global evaluation metric in semantic segmentation tasks.

[0156] To further verify the advancement of the method of this invention, its segmentation results are compared with four representative existing methods published in recent years on the same task (such as tongue segmentation), including:

[0157] Existing method 1: Tan, Daiqing, et al. "Tongue-litesam: a lightweight model for tongue image segmentation with zero-shot." IEEE Access (2025).

[0158] Existing method 2: Dong, Zhiyu, et al. "GA-TongueNet: tongue imagesegmentation network using innovative DiFP and MDi for stable generalizationability." Frontiers in Physiology 16 (2025): 1617647.

[0159] Existing method 3: Cai, Wenjun, et al. "TSRNet: Tongue image segmentation with global and local refinement." Displays 81 (2024): 102601.

[0160] Existing method 4: Tang, Yan, et al. "RTC_TongueNet: An improved tongue imagesegmentation model based on DeepLabV3." Digital Health 10 (2024):20552076241242773.

[0161] The method of this invention is evaluated based on the same test set as the present invention. The dataset partitioning, data augmentation strategy and hardware testing environment are consistent to ensure the fairness and reliability of the comparison results. The test results are shown in Table 1.

[0162] Table 1: Test results of the method of the present invention and existing methods

[0163]

[0164] As can be seen from Table 1, the method of the present invention is significantly better than the classical basic method and the domain optimization method in both indicators. Compared with the current state-of-the-art method, the mDSC is improved by 2.4% to 8.0%, and the mIoU is improved by 3.3% to 9.4%, which fully demonstrates the advantage of the present invention in segmentation accuracy.

[0165] The segmentation results of the method of this invention and the existing methods are as follows: Figure 6As shown, column 1 represents the image to be segmented, column 2 represents the manually segmented image, column 3 represents the segmented image using the method of this invention, column 4 represents the segmented image using existing method 1, column 5 represents the segmented image using existing method 2, column 6 represents the segmented image using existing method 3, and column 7 represents the segmented image using existing method 4. Row 1 is the first example, and row 2 is the second example. It can be seen that due to uneven lighting conditions and the relative differences in tongue texture and coating, most methods segment the tongue boundary incorrectly. The method of this invention shows more accurate and stable segmentation prediction. The segmented image output by the method of this invention not only meets the requirements but is also more accurate and performs more consistently on datasets with large variations. This indicates that the model has strong robustness.

[0166] To verify the effectiveness of each module proposed in the method architecture of this invention, a refinement module was applied to the segmentation network and trained from scratch. The analysis results are shown in Table 2.

[0167]

[0168] As shown in Table 2, compared with the baseline model, both the main region feature dynamic fusion module and the main feature cross fusion module show relative improvements in mDSC and mIoU, demonstrating the effectiveness of the proposed method in the tongue segmentation task. The mDSC and mIoU of the main region feature dynamic fusion module are 2.3 and 1.3 percentage points higher than the baseline, respectively. Furthermore, the mDSC and mIoU of the main feature cross fusion module are further improved by 2.6 and 3.3 percentage points, respectively. The mDSC and mIoU of the auxiliary region feature dynamic fusion module are further improved by 3.5 and 3.6 percentage points, respectively. Additionally, the mDSC and mIoU of the auxiliary feature cross fusion module are further improved by 1.9 and 2.3 percentage points, respectively. This indicates that the proposed method improves the performance of each model to a certain extent.

[0169] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A tongue image segmentation and recognition method based on dynamic fusion of edge regions, characterized in that, include: Obtain tongue image; The tongue image is input into the tongue image segmentation and recognition network model; The main region features are extracted from the tongue image by the main region encoder of the tongue image segmentation and recognition network model. The main region features are dynamically fused using the main region feature dynamic fusion module to obtain the main region dynamic features and the main region feature dynamic fusion features; The last feature in the main region features and the dynamic fusion feature of the main region features are concatenated by channel and then convolved together with the dynamic features of the main region. Then, they are passed through the main image segmenter to perform image segmentation to obtain the tongue image segmentation result. The tongue image segmentation and recognition network model also includes an auxiliary region encoder and an edge image extraction module in the pre-training stage; The edge image extraction module is used to extract the edge images of the main image and the auxiliary image using a gradient operator to obtain the main edge image and the auxiliary edge image. Its output is connected to the shared edge encoder, which is used to extract the edge features in the main edge image and the auxiliary edge image to obtain the main edge features and the auxiliary edge features. The auxiliary region encoder is used to extract features from the auxiliary image to obtain auxiliary region features. Its output is connected to the input of the auxiliary region feature dynamic fusion module. The auxiliary region feature dynamic fusion module is used to dynamically fuse the auxiliary region features to obtain auxiliary region dynamic features and auxiliary region feature dynamic fusion features. The outputs of the edge image extraction module, the shared edge encoder, and the auxiliary region encoder are all connected to the input of the auxiliary feature cross-fusion module. The auxiliary edge image, the last auxiliary edge feature, the last auxiliary region feature, and the last main edge feature are cross-fused to obtain the auxiliary feature cross-fusion feature. The outputs of the edge image extraction module, the shared edge encoder, and the main region encoder are all connected to the input of the main feature cross-fusion module. The main edge image, the last main edge feature, the last main region feature, and the last auxiliary edge feature are cross-fused to obtain the main feature cross-fusion feature. The outputs of the auxiliary region encoder and the auxiliary region feature dynamic fusion module are connected to the auxiliary image segmenter. The last auxiliary region feature and the auxiliary region feature dynamic fusion feature are concatenated by channel and then convolved together with the auxiliary region dynamic feature and passed through the auxiliary image segmenter to obtain the auxiliary image segmentation result. Both the main region feature dynamic fusion module and the auxiliary region feature dynamic fusion module include multiple processing lines. Each processing line corresponds to a different input feature and includes two branches. One branch includes a dynamic convolution unit within the feature and a 1×1 convolution unit connected in sequence. The other branch includes a 1×1 convolution unit and a dynamic convolution unit between features connected in sequence. The outputs of the two branches are merged, upsampled, and then spliced ​​and output in the channel dimension.

2. The tongue image segmentation and recognition method based on dynamic fusion of edge regions according to claim 1, characterized in that, The main region encoder, auxiliary region encoder, and shared edge encoder all use UNet encoders, and the main image segmenter and auxiliary image segmenter both use UNet decoders.

3. The tongue image segmentation and recognition method based on dynamic fusion of edge regions according to claim 1, characterized in that, The data processing procedures for the intra-feature dynamic convolutional units and inter-feature dynamic convolutional units include: The spatial dimensions of the input feature map are averaged, the global context is extracted, and attention weights are calculated using a fully connected layer. The learnable convolutional kernels are weighted and combined according to attention weights to obtain dynamic output.

4. The tongue image segmentation and recognition method based on dynamic fusion of edge regions according to claim 1, characterized in that, The data processing procedure of the main feature cross-fusion module includes: The main edge image is thresholded and downsampled to the same size as the main edge features to obtain the downsampled main edge image; Randomly sample several non-zero pixels from the downsampled main edge image and record their positions; Extract the pixel features at non-zero pixel positions of the main edge features to construct the main edge feature map; Flatten the main region features and auxiliary edge features to obtain the main region feature map and auxiliary edge feature map, respectively. The main edge feature map and the main region feature map are fused within the main edge region to obtain the fused feature within the main edge region. The main region feature map and the auxiliary edge feature map are fused together to obtain the fused feature of the auxiliary edge feature and the main region feature. The auxiliary feature cross-fusion feature is obtained by adding the fusion features inside the main edge region, the auxiliary edge features, and the fusion features of the main region features. The auxiliary feature cross-fusion feature has the same shape as the main edge feature. The data processing procedure of the auxiliary feature cross-fusion module includes: The auxiliary edge image is thresholded and downsampled to the same size as the auxiliary edge features to obtain the downsampled auxiliary edge image; Randomly sample several non-zero pixels from the downsampled auxiliary edge image and record their positions; Extract the pixel features at non-zero pixel positions to construct the auxiliary edge feature map; Flattening the auxiliary region features and the main edge features yields the auxiliary region feature map and the main edge feature map, respectively. The auxiliary edge feature map and the auxiliary region feature map are fused within the auxiliary edge region to obtain the fused feature within the auxiliary edge region. The auxiliary region feature map and the main edge feature map are fused together to obtain the fused feature of the main edge feature and the auxiliary region feature. The main feature cross-fusion feature is obtained by adding the fusion features inside the auxiliary edge region and the fusion features of the main edge feature and the auxiliary region feature. The main feature cross-fusion feature has the same shape as the auxiliary edge feature.

5. The tongue image segmentation and recognition method based on dynamic fusion of edge regions according to claim 4, characterized in that, The expression for fusing the main edge feature map and the main region feature map within the main edge region to obtain the fused feature within the main edge region is as follows: ； in, This represents the attention matrix fused within the main edge region. Represents the normalization function. Represents the feature map of the main region. Represents the dimension of the attention matrix. , , , , , Both represent learnable weight matrices. This represents a learnable bias. Indicates the first edge in the downsampled main edge image A non-zero pixel position, Indicates the location of the main region feature map. pixel features, This indicates the fusion characteristics within the main edge region; The expression for fusing the main region feature map and the auxiliary edge feature map, resulting in the fused feature of the auxiliary edge features and the main region features, is as follows: ； in, This represents the attention matrix that fuses auxiliary edge features with main region features. Represents auxiliary edge feature map, , , Both represent learnable weight matrices. This represents a learnable bias. This represents the fusion of auxiliary edge features and main region features; The expression for fusing the auxiliary edge feature map and the auxiliary region feature map within the auxiliary edge region to obtain the fused feature within the auxiliary edge region is as follows: ； ； in, This represents the attention matrix used for fusion within the auxiliary edge regions. Represents the feature map of the auxiliary region. , , , Both represent learnable weight matrices. This represents a learnable bias. The third auxiliary edge image in the downsampled image A non-zero pixel position, Indicates the location of the auxiliary region feature map pixel features, Indicates the fusion features within the auxiliary edge region; The expression for fusing the auxiliary region feature map and the main edge feature map to obtain the fused feature of the main edge feature and auxiliary region feature is as follows: ； in, This represents the attention matrix that fuses the main edge features and auxiliary region features. Represents the main edge feature map. This represents the learnable weight matrix. This represents a learnable bias. This represents the fusion of main edge features and auxiliary region features.

6. The tongue image segmentation and recognition method based on dynamic fusion of edge regions according to claim 1, characterized in that, It also includes a main image decoder and an auxiliary image decoder. The input end of the main image decoder is connected to the output end of the main feature cross-fusion module and the shared edge encoder, and the input end of the auxiliary image decoder is connected to the output end of the auxiliary feature cross-fusion module and the shared edge encoder. The main feature cross-fusion feature and the main edge image feature are concatenated by channel and then input into the main image decoder to reconstruct the reconstructed main image; The auxiliary feature cross-fusion feature and the auxiliary edge image feature are concatenated by channel and then input into the auxiliary image decoder to reconstruct the reconstructed auxiliary image. Both the main image decoder and the auxiliary image decoder use the Unet decoder.

7. The tongue image segmentation and recognition method based on dynamic fusion of edge regions according to claim 6, characterized in that, The pre-training method for the tongue image segmentation and recognition network model includes: Obtain the main image dataset and the auxiliary image dataset; The main image dataset and the auxiliary image dataset are used as inputs to train the tongue image segmentation and recognition network model. During the training process, the loss function is calculated until the loss function is less than a set threshold, and then the pre-trained tongue image segmentation and recognition network model is obtained.

8. The tongue image segmentation and recognition method based on dynamic fusion of edge regions according to claim 7, characterized in that, The expression for the main image dataset is: ,in, Represents the main image dataset. Indicates the first The main image, Indicates the first Segmentation labels for each main image. Indicates the number of main images; The expression for the auxiliary image dataset is: ,in, Represents an auxiliary image dataset. Indicates the first Auxiliary image, Indicates the first Segmentation labels for auxiliary images, Indicates the number of auxiliary images.

9. The tongue image segmentation and recognition method based on dynamic fusion of edge regions according to claim 7, characterized in that, The expression for calculating the loss function is as follows: ； ； ； ； in, Indicates the total loss. Indicates the partition loss. Represents the cross-entropy loss function. This represents the auxiliary image segmentation result. This represents the segmentation result of the main image. Segmentation labels for auxiliary images, The segmentation label represents the main image. Indicates the losses incurred during reconstruction. Describing the L1 norm, Indicates the main image. Indicates auxiliary image, Indicates the cross-fusion features of the main image. Indicates the main edge features, Indicates the cross-fusion features of auxiliary images. Indicates auxiliary edge features, Indicates the main region decoder. Indicates the auxiliary region decoder, Indicates channel splicing. Indicates distance loss. Represents the L2 norm. Indicates the main region characteristics, This indicates the dynamic fusion of features in the main region. Indicates auxiliary region features, This indicates the dynamic fusion of auxiliary region features. This represents a 1×1 convolution operation.

Images