Method and device for classifying striation tongues based on a priori regularity

Abstract

The application discloses a tooth mark tongue classification method and device based on priori regularization, and the method comprises the following steps: acquiring a tongue image; processing the tongue image to obtain a mask image; obtaining an attention map through convolutional neural network processing of the tongue image; and combining the mask image and the attention map to construct a tooth mark tongue classification model. The application provides a tooth mark tongue classification method and device based on priori regularization, which overcomes the problem that the model in the prior art does not focus on the target key area, and improves the model performance.

Description

Technical Field

[0001] This invention relates to the field of classification technology, and specifically to a method and apparatus for classifying scalloped tongues based on prior regularization. Background Technology

[0002] Scalloped tongue is characterized by the presence of tooth marks along the edges of the tongue, typically caused by abnormal tongue development. Currently, most methods for identifying scalloped tongue employ deep learning. Specifically, scalloped tongue classification models determine scalloped tongue by judging the presence of tooth marks along the tongue's edges. Therefore, scalloped tongue classification models should focus more on the tongue's edges. The heatmaps of existing models are as follows... Figure 4 As shown in the figure, the model does not focus on the edges of the tongue. Therefore, the performance of this model needs improvement. Summary of the Invention

[0003] To address one or more of the aforementioned technical problems, this invention provides a method and apparatus for classifying serrated tongues based on prior regularization.

[0004] According to one aspect of the present invention, a method for classifying scalloped tongues based on prior regularization is provided, comprising:

[0005] Obtain a tongue image;

[0006] The tongue image is processed to obtain a mask image;

[0007] At least two attention maps are obtained by processing the tongue image using a convolutional neural network.

[0008] Calculate the total loss function, which includes calculating a first loss function and calculating a second loss function;

[0009] The first loss function is calculated based on the attention map, and the second loss function is calculated by combining the mask map and the attention map.

[0010] The tongue classification model of the present invention guides the model to focus on the edge region of the tongue by using the mask image and attention map as input to the model and calculating the loss function, thereby limiting the range that needs to be focused on and thus improving the accuracy and other performance of the model.

[0011] In some embodiments, processing the tongue image to obtain a mask image M further includes:

[0012] The initial mask image M0 is obtained by segmenting the tongue image;

[0013] Convert the initial mask image M0 into a grayscale image;

[0014] After performing a matrix translation operation on the grayscale image using an identity matrix of a specified size, erosion is performed to obtain the eroded image M1.

[0015] The mask image M is calculated by combining the initial mask image M0 and the erosion image M1.

[0016] The mask image M is calculated according to the following formula: M = M0 × (1.0 - M1), resulting in the mask image M. The mask image effectively limits the key candidate regions of the target that the model needs to focus on.

[0017] In some implementations, tongue image segmentation includes first segmenting and labeling the tongue image, and then performing tongue image segmentation algorithm processing. The tongue image segmentation algorithm processing includes: first obtaining segmentation criteria, and then using the segmentation criteria to segment the tongue image. Tongue image segmentation includes segmenting into the tongue body and non-tongue body.

[0018] In some embodiments, obtaining an attention map after processing the tongue image using a convolutional neural network further includes:

[0019] A first convolution operation is performed on the feature map to obtain a first attention map Attention1, and a second convolution operation is performed on the feature map to obtain a second attention map Attention2.

[0020] In some implementations, the backbone network model is a VGG network, a ResNet network, or another CNN network.

[0021] The first loss function is calculated by combining the first attention map Attention1 and the second attention map Attention2, and the second loss function is calculated by combining the mask map M, the first attention map Attention1 and the second attention map Attention2.

[0022] The calculation of the total loss function further includes calculating a third loss function; the first loss function is the different region attention loss function, the second loss function is the mask regularization loss function, and the third loss function is the classification cross-entropy loss function. The classification cross-entropy loss function L... ce It is obtained through the following calculation formula:

[0023] Where q(x) is the 0-1 code of the actual class, p(x) is the probability predicted by the model, and k is the number of actual classes;

[0024] The different regions focus loss functions L cross It is obtained through the following calculation formula:

[0025]

[0026] The mask regularization loss function L exclusion It is obtained through the following calculation formula:

[0027] Where W is the width of the image and H is the height of the image;

[0028] The total loss function is obtained using the following formula:

[0029] L=α×L ce +β×L cross +γ×L exclusion Where the total loss function is L, and α is the classification cross-entropy loss function L ce The weighting coefficients, β, represent the attention loss function L for different regions. cross The weighting coefficients, where γ is the mask regularization loss function L. exclusion The weighting coefficients.

[0030] A priori regularized tongue classification device includes:

[0031] The tongue image acquisition module is used to acquire tongue images.

[0032] The processing module includes a first processing unit for processing the tongue image to obtain a mask image M; and a second processing unit for processing the tongue image through a convolutional neural network to obtain a first attention image Attention1 and a second attention image Attention2.

[0033] The calculation module is used to calculate the total loss function by combining the mask image M, the first attention image Attention1, and the second attention image Attention2. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the training phase of a method for classifying scalloped tongues according to an embodiment of the present invention.

[0035] Figure 2 This is a schematic diagram of the testing phase of a method for classifying scalloped tongues according to an embodiment of the present invention.

[0036] Figure 3 A flowchart illustrating the process of generating a mask image from a tongue image;

[0037] Figure 4 This is a schematic diagram of the thermal model in the existing technology;

[0038] Figure 5 This is a thermal schematic diagram of a model using the tooth-mark tongue sorting device of the present invention. Detailed Implementation

[0039] The present invention will now be described in further detail with reference to the accompanying drawings.

[0040] Figure 1The diagram illustrates the training phase of the scalloped tongue classification method. Figure 2 The diagram illustrates the testing phase of the scalloped tongue classification method. Unlike the training phase, the testing phase does not require generating a mask image of the tongue image.

[0041] The classification methods for scalloped tongue include:

[0042] Obtain a tongue image;

[0043] The tongue image is processed to obtain a mask image;

[0044] At least two attention maps are obtained by processing the tongue image using a convolutional neural network.

[0045] The total loss function is calculated by combining the mask image and the attention image.

[0046] A priori regularized tongue classification device includes:

[0047] The tongue image acquisition module is used to acquire tongue images.

[0048] The processing module includes a first processing unit for processing the tongue image to obtain a mask image M; and a second processing unit for processing the tongue image through a convolutional neural network to obtain a first attention image Attention1 and a second attention image Attention2.

[0049] The calculation module is used to calculate the total loss function by combining the mask image M, the first attention image Attention1, and the second attention image Attention2.

[0050] Methods for generating the mask image M include:

[0051] The initial mask image M0 is obtained from the tongue image acquisition module through the tongue image segmentation algorithm;

[0052] The tongue image is in RGB format.

[0053] Convert the initial mask image M0 into a grayscale image;

[0054] After performing a matrix translation operation on the grayscale image, a 3×3 identity matrix is ​​eroded to obtain the eroded image M1.

[0055] The mask image M is obtained by calculating M = M0 × (1.0 - M1) using the following formula.

[0056] In this embodiment, the original tongue image size is 255×255, meaning both the width and height are 255 pixels, corresponding to a 255×255×3 matrix during model forward computation. The identity matrix used in this embodiment is a 3×3 matrix. The identity matrix can also be 5×5 or 7×7. The matrix translation operation proceeds from left to right and from top to bottom, moving one pixel at a time. The matrix translation operation is essentially multiplying the corresponding values ​​of the 3×3 identity matrix and the corresponding 255×255 matrix of the tongue image grayscale image and summing the results. The greater the number of matrix translation operations, the larger the tongue edge mask on the tongue image. The specific number of operations can be selected according to actual needs.

[0057] Tongue segmentation of a tongue image includes labeling the tongue image using an image annotation tool and processing it using a tongue segmentation algorithm. The tongue segmentation algorithm involves first obtaining segmentation criteria, and then using those criteria to segment the tongue image. An image segmentation model is used for tongue segmentation. In this implementation, the image annotation tool is Labelme, and the image segmentation model is the Unet image segmentation model.

[0058] Methods for generating attention maps include:

[0059] The tongue image obtained from the tongue image acquisition module is input into the backbone network model, and the tongue image feature map of the tongue image is obtained through the backbone network model.

[0060] The backbone network model is a convolutional neural network model used to extract image features. In this embodiment, the backbone network used is the ResNet network.

[0061] Two types of attention maps are obtained by performing two convolutional branch operations on the feature maps respectively.

[0062] Two attention maps were derived based on two image patterns: one with teeth marks and one without. The first attention map for the tongue image with teeth marks is designated Attention1, and the second attention map for the tongue image without teeth marks is designated Attention2.

[0063] In this embodiment, the convolutional branching operation includes global average pooling of the feature map, a fully connected layer, and activation function processing. The activation functions include ReLU and SIGMOID. The first attention map (or second attention map) and the feature map are fused using bilinear attention pooling, a technique employed in existing technologies, to obtain the feature vector of the tongue image. A classification operation is then performed on the feature vector to obtain the classification cross-entropy loss function, such as... Figure 1The feature vector is a one-dimensional vector of a specified length. The specified length can be selected from the following: 64, 128, 256, and 512, or a custom length can be defined. The classification operation involves processing the feature vector through a fully connected layer and a softmax function, as per existing techniques.

[0064] Building a tongue-marking classification model requires not only a mask image and two attention maps, but also the calculation of a total loss function. Calculating the total loss function includes calculating a first loss function, a second loss function, and a third loss function. In this embodiment, the first loss function is the different region attention loss function, the second loss function is the mask regularization loss function, and the third loss function is the classification cross-entropy loss function.

[0065] The different region focus loss functions are used to limit the different regions that the two image modes in the tooth-marked tongue classification model need to focus on. Since the tooth marks are located on the edge of the tongue, the tooth-marked tongue classification model needs to focus on the tooth mark region on the edge of the tongue for tongue images with tooth marks, while it needs to focus on the non-tooth mark region on the edge of the tongue for tongue images without tooth marks.

[0066] The masked regularization loss function is used to limit the focus on the tongue edge region in both image modes of the tooth-marked tongue classification model.

[0067] Different regions focus on loss function L cross It is obtained through the following calculation formula:

[0068]

[0069] Masking regularization loss function L exclusion It is obtained through the following calculation formula:

[0070] Where W is the width of the image and H is the height of the image.

[0071] Classification cross-entropy loss function L ce It is obtained through the following calculation formula:

[0072]

[0073] q(x) is the 0-1 code for the actual category. This embodiment uses 0-1 encoding. p(x) is the probability predicted by the model. Tongue images with teeth marks are encoded as 1, and tongue images without teeth marks are encoded as 0. k is the number of actual categories; in this embodiment, tongue images are divided into two categories: with teeth marks and without teeth marks, therefore k is 2.

[0074] The total loss function is obtained using the following formula:

[0075] L=α×L ce +β×L cross +γ×Lexclusion

[0076] Where the total loss function is L, and α is the classification cross-entropy loss function L ce The weighting coefficients, β, represent the attention loss function L for different regions. cross The weighting coefficients, where γ is the mask regularization loss function L. exclusion The weighting coefficients.

[0077] The constructed tongue classification model with teeth marks can use the backpropagation algorithm to update the model parameters during training, so that the model outputs a value as close as possible to 0 for tongue images without teeth marks and a value as close as possible to 1 for tongue images with teeth marks.

[0078] Figure 3 A schematic diagram of the thermal model in the prior art is shown, while Figure 4 A schematic thermal diagram of a model using the tooth-mark tongue classification device of the present invention is shown. Since the tooth marks are located on the edge of the tongue, the model should focus on the edge of the tongue to determine whether there are tooth marks on the tongue image. Figure 3 and Figure 4 In comparison, it can be seen that Figure 4 We have identified areas that should be given special attention. These areas should be given special attention in... Figure 1 The center area is the white region of the mask image.

[0079] The method and model of this invention are used to determine whether there are teeth marks on a tongue image. The accuracy of this invention's model is improved by approximately 5% compared to existing models. The accuracy is achieved by knowing the true label of each tongue image in advance, having the trained model predict the predicted label for each tongue image, and then statistically analyzing the proportion of predicted labels that match the true labels.

[0080] The above descriptions are merely some embodiments of the present invention. Those skilled in the art can make various modifications and improvements without departing from the inventive concept of the present invention, and these all fall within the scope of protection of the present invention.

Claims

1. A method for classifying striation tongues based on a priori regularity, characterized in that, Comprising: acquiring a tongue image; The tongue image processing obtains a mask image, including: obtaining an initial mask image M0 by tongue image segmentation on the tongue image; converting the initial mask image M0 into a gray image; performing a matrix translation operation on a unit matrix of a specified size on the gray image and then performing corrosion to obtain a corrosion image M1; combining the initial mask image M0 and the corrosion image M1 to obtain the mask image M through . ​ The tongue image is subjected to convolutional neural network processing to obtain at least two attention maps, including: two attention maps are obtained according to two image modes of the tongue image with tooth marks and the tongue image without tooth marks; the first attention map of the image mode of the tongue image with tooth marks is , and the second attention map of the image mode of the tongue image without tooth marks is . calculating a total loss function, the calculating total loss function comprising calculating a first loss function and calculating a second loss function, calculating a third loss function; the first loss function is a different area attention loss function, the second loss function is a mask regularization loss function, and the third loss function is a classification cross entropy loss function; calculating the first loss function according to the attention map, and calculating the second loss function in combination with the mask map and the attention map, comprising: combining the first attention map and the second attention map computing a first loss function in combination with the mask map the first attention map and the second attention map computing a second loss function.

2. The priori-regularity-based tongue print classification method according to claim 1, wherein, The tongue image segmentation comprises tongue image segmentation annotation of the tongue image first, and tongue image segmentation algorithm processing, the tongue image segmentation algorithm processing comprising: obtaining a segmentation standard first, and then using the segmentation standard to perform tongue image segmentation on the tongue image.

3. The priori-regularity-based tongue print classification method of claim 2, wherein, The attention map obtained after the tongue image is processed by the convolutional neural network further comprises: the tongue image extracts features of the tongue image through a backbone network model to obtain a feature map; performing a first convolution operation on the feature map to obtain a first attention map performing a second convolution operation on the feature map to obtain a second attention map .

4. The priori regularity-based tongue print classification method of claim 3, wherein, The backbone network model is a VGG network or a Resnet network or other CNN network.

5. The priori-regularity-based tongue print classification method of claim 1, wherein, the categorical cross-entropy loss function is obtained by the following calculation formula: wherein q(x) is a 0-1 encoding of the actual class, p(x) is the model predicted probability, k is the number of actual classes; The different regions focus on loss functions is obtained by the following calculation formula: ； The mask regular loss function is obtained by the following calculation formula: wherein W is the width of the image, H is the height of the image; The total loss function is obtained by the following calculation formula: The total loss function is: L , α Cross-entropy loss function for classification The weighting coefficients, β Focus on loss function for different regions The weighting coefficients, gamma For masked regularization loss function The weighting coefficients.

6. A device for classifying tongue scrapings based on a priori regularity, characterized by Comprising: a tongue image acquisition module, configured to acquire a tongue image; The processing module comprises a first processing unit, configured to process the tongue image to obtain a mask image , comprising: obtaining an initial mask image M0 by segmenting the tongue image; converting the initial mask image M0 into a gray image; performing matrix translation operation on a unit matrix of a specified size on the gray image and then performing corrosion to obtain a corrosion image M1; combining the initial mask image M0 and the corrosion image M1 to obtain a mask image M by ;​ The second processing unit is further configured to obtain a first attention map by processing the tongue image through a convolutional neural network and a second attention map , including: obtaining two attention maps according to two image modes of the tongue image with tooth marks and the tongue image without tooth marks; the first attention map of the image mode of the tongue image with tooth marks is , and the second attention map of the image mode of the tongue image without tooth marks is ; a computing module configured to combine the mask map , a first attention map , and a second attention map compute a total loss function, the total loss function including a first loss function and a second loss function, a third loss function; the first loss function is a different area attention loss function, the second loss function is a mask regularization loss function, and the third loss function is a classification cross-entropy loss function, comprising: combining the first attention map and the second attention map computing a first loss function combining the mask map , the first attention map and the second attention map computing a second loss function.

Images