Tongue picture data enhancement method based on fractal features

Abstract

The invention belongs to the technical field of image processing, and particularly provides a tongue picture data enhancement method based on fractal features, and tongue picture data enhancement is performed by using the fractal features. The method comprises the following steps: firstly, pre-training a backbone network (MLP-Like model) by using a public descriptive texture data set; then, sending the original tongue picture data into a backbone network for feature extraction to obtain a hierarchical tongue picture feature map, and sending the original tongue picture image and the corresponding tongue picture feature map as target images into an iterative function system to generate a fractal image; and finally, taking the generated fractal image as supplementary data, generating a label of the image according to a label label of the target image, and completing data enhancement of the tongue picture data set. According to the method, the tongue picture image with the fractal feature is generated by using the iterative function system, the tongue picture data set is supplemented, and the method can effectively relieve the problem of data imbalance of the tongue picture data set as a data enhancement method combined with the fractal feature.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and specifically provides a tongue image data enhancement method based on fractal features. Background Art

[0002] Tongue diagnosis mainly uses neural networks to extract tongue images, and then conducts subsequent auxiliary diagnosis and treatment, such as inferring symptoms and constitution. However, in practical applications, there is a long-tail problem in the distribution of tongue images, that is, compared with normal tongue images, there are relatively few samples of cracks, tooth marks, etc. During model training, data imbalance problems will affect the generalization ability of the model, thereby reducing the accuracy of the model. In order to improve model performance, it is usually necessary to adopt data enhancement solutions to increase the sample size and improve the generalization ability of the model.

[0003] At present, the tongue image data enhancement method is still relatively simple, including flipping, mirroring and other methods; for example, the document "S.Lin, X.Song, Z.xu, et al, Moment Invariants with Data Augmentation for Tongue Image Segmentation [C]. in: Proceedings of IEEE International Conference on Bioinformatics and Biomedicine, Houston, 2021, pp. 990-991" proposes a tongue image enhancement method based on moment invariants to assist in tongue segmentation. This method performs different rotation angles, horizontal flipping, noise addition and brightness adjustment on each tongue image to enhance the data. Then, the moment invariant loss is added to the loss, that is, the difference between the samples enhanced by different enhancement methods and the mean of the enhanced samples is used as the loss to prevent the deviation of data enhancement; for example, the document "Y.Zhang, T.Yao, Z.Qiu, et al. Lightweight and Progressively-Scalable Networks for Semantic Segmentation [J]. International Journal of Computer Vision, 2022, 130(4): 994-997” provides a method for pre-training using fractal images instead of real images. A large number of fractal images similar to natural objects are generated using an iterative function system, and then used for model pre-training. This allows the model to learn universal features without using a large-scale real dataset, so that the generalization performance of the model in general fields can be improved, and then the model parameters can be fine-tuned using data from the target field.

[0004] However, in the above two documents, the former is not targeted for enhancing the tongue image features, and does not take into account the texture features of the tongue, such as the tongue coating texture on the tongue surface and the vein texture on the bottom of the tongue. The fractal features of the tongue surface are closely related to the texture, and the fractal features of healthy tongue images and pathological tongue images have been shown to be significantly different; the latter is only targeted at general fields. In the special field of tongue images, simply using non-targeted fractal images for pre-training costs a lot of time and cannot achieve better performance improvement effects. Moreover, this method simply uses an iterative function system for generation and cannot perform learnable generation for the target image. Summary of the invention

[0005] The purpose of the present invention is to provide a tongue image data enhancement method based on fractal features, which utilizes fractal features to enhance tongue image data. Specifically, after gradually collecting tongue image features through a deep neural network (MLP-Like model), an iterative function system is used to generate tongue image images with fractal features to supplement the tongue image data set. As a data enhancement method combining fractal features, the present invention can effectively alleviate the data imbalance problem of the tongue image data set.

[0006] To achieve the above object, the technical solution adopted by the present invention is:

[0007] A tongue image data enhancement method based on fractal features, characterized by comprising the following steps:

[0008] Step 1. Use the public Describable Texture Dataset (DTD) to pre-train the backbone network, where the backbone network adopts the MLP-Like model;

[0009] Step 2. Send the original tongue image data to the backbone network for feature extraction to obtain tongue image feature maps at multiple levels;

[0010] Step 3. The original tongue image and its corresponding tongue feature map are used as target images and sent into the iterative function system to generate a fractal image;

[0011] Step 4. Use the generated fractal image as supplementary data, annotate the label of the generated image according to the label of the target image, and complete the data enhancement of the tongue image dataset.

[0012] Furthermore, in step 1, the MLP-Like model includes a fully connected layer, multiple MLP blocks and a classification head connected in sequence; the texture image x is input into the fully connected layer, the linear embedding Y0 is obtained and input into the first MLP block, which is specifically expressed as:

[0013] Y0=W0x+b0

[0014] Among them, W0 and b0 are the parameters of the fully connected layer;

[0015] Each MLP block adopts the same structure, which is specifically expressed as:

[0016] Y1=W1(LN(input))+b1

[0017] Y2=W2(BLP(LN(input)))+b2

[0018] Y3=W3(Y1+Y2)+b3

[0019] Y=Y3+input

[0020] Y4=W4(LN(Y T ))+b4

[0021] output=(Y4+Y T ) T

[0022] Among them, input and output represent the input and output of the MLP block respectively, LN represents normalization processing, BLP represents bilinear pooling processing; W1, W2, W3, W4 and b1, b2, b3, b4 are all parameters of the fully connected layer;

[0023] The output of the last MLP block, output′, is used as the input of the classification head, which consists of a fully connected layer and a softmax function, specifically expressed as:

[0024] P = softmax(W5(output′)+b5)

[0025] Among them, P is the prediction probability vector, and W5 and b5 are the parameters of the fully connected layer.

[0026] Furthermore, in step 1, the pre-training process is:

[0027] Using cross entropy loss as the loss function, it is specifically expressed as:

[0028]

[0029] Among them, P(x i =c j ) represents the corresponding category c in the predicted probability vector P for the i-th sample j The probability of , t is the number of categories, n is the number of samples; when the label of the i-th sample is category c j When ij =1, otherwise, g ij =0;

[0030] According to the Describable Texture Dataset (DTD), the backbone network is pre-trained by back-propagation technology.

[0031] Furthermore, in step 2, the original tongue image data adopts the public data set BioHit, and the background-free tongue image in the data set is used as the original tongue image, and cracks, punctures, teeth marks, and others are selected as labels for annotation;

[0032] The pre-trained backbone network is used to extract features of the tongue image data, and the output of each MLP block in the backbone network is obtained as the feature extraction result. The MLP blocks from shallow to deep output the tongue image feature maps of the corresponding levels.

[0033] Furthermore, in step 3, the specific process is:

[0034] Step 3.1 Randomly generate N pairs of position transformation parameters {A, b} N , where A∈[-1,1], b∈[-1,1];

[0035] Step 3.2 Randomly sample T pairs from N pairs of position transformation parameters to form a position transformation parameter set {A, b} T , T is the number of steps to generate; according to the position transformation parameter set {A, b} T , the iterative function system generates the pixel coordinates corresponding to the fractal image. The specific process is:

[0036] v (t) =A (t) v (t-1) +b (t) , t=1,2,...,T

[0037] Among them, v (t) represents the pixel coordinates generated in step t, v (0) Indicates the preset initial pixel coordinates, {A (t) , b (t)} represents the position transformation parameter pair used in step t;

[0038] Step 3.3 generates a fractal image by rendering according to the pixel coordinates generated by the iterative function system, which is specifically expressed as:

[0039] I′(h,w)=∑ t exp(-||(h,w)-v (t) || / τ), h∈H, w∈W, t=1,2,...,T

[0040] Where (h, w) represents the pixel coordinates, I′(h, w) represents the brightness of the pixel with the pixel coordinates (h, w) in the generated image, ||·|| represents the L2 norm operator, τ is a hyperparameter, τ∈(0,1];

[0041] Step 3.4 calculates the loss of the iterated function system;

[0042] Convert the target image to a grayscale image and normalize it, which can be expressed as follows:

[0043]

[0044] Among them, I gray Represents a grayscale image, Represents the target image I raw The R, G, and B channel components of the grayscale image; I(h, w) represents the brightness of the pixel with the pixel coordinates (h, w) in the normalized grayscale image. Indicates the minimum and maximum brightness in the grayscale image;

[0045] The fractal image and the target image are subjected to pixel-by-pixel loss calculation, which can be specifically expressed as:

[0046]

[0047] Where D′ and D represent the differential box-counting of the generated image and the target image, respectively, and I′ and represent the brightness matrix of the generated image and the target image, respectively;

[0048] Step 3.5 Update the position transformation parameter set {A, b} using the gradient descent method based on the loss T , repeat steps 3.2 to 3.4 for iteration until the number of iterations is reached or the loss reaches the threshold condition. After the iteration is completed, the fractal image corresponding to the target image is obtained.

[0049] Based on the above technical solution, the beneficial effects of the present invention are:

[0050] The present invention adopts MLP-Lik e Neural network model and iterated function system technology, using MLP-Lik e The model does not change the input image size, using MLP-Lik e The model learns texture features and then uses them to extract tongue texture features. e The shallow and deep intermediate feature maps of the model are used as representations with rich texture information and semantic information, respectively. eThe output representation of the model is used as the target image, and a controllable iterative function system is used to generate fractal images, so that the generated fractal images are similar to the original images, and finally the generated images are used as a supplement to the original tongue image data set, so that the imbalance problem of the number of samples in each category of the original tongue image data set is alleviated, and the tongue image classification model trained using the enhanced data set is not prone to overfitting and has better generalization ability. Compared with traditional image deformation, moment invariant and uncontrollable fractal image generation schemes, the present invention has the advantage of being more in line with the tongue image scene of traditional Chinese medicine. Through controllable fractal image generation, tongue image data enhancement is performed, and the data set contains more diverse texture features. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A schematic flow chart of the fractal feature-based tongue image data enhancement method provided by the present invention.

[0052] Figure 2 This is a model architecture diagram of the fractal feature-based tongue image data enhancement method provided by the present invention.

[0053] Figure 3 This is a schematic diagram of the structure of the MLP-Like model provided by the present invention.

[0054] Figure 4 This is a schematic diagram of the structure of the MLP block in the MLP-Like model provided by the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and beneficial effects of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0056] This embodiment provides a tongue image data enhancement method based on fractal features, and its process is as follows: Figure 1 As shown, the model architecture is as Figure 2 As shown; specifically including the following steps:

[0057] Step 1. Use the publicly available Describabl e Textur e Dataset, DTD) is used to pre-train the backbone network;

[0058] The backbone network adopts MLP-Lik e Models such as Figure 3 As shown, it includes a fully connected layer, multiple MLP blocks and a classification head connected in sequence; the number of MLP blocks can be adaptively matched according to the application scenario. In this embodiment, 8 MLP blocks are used. The MLP blocks play a role in encoding the spatial dimension information and channel dimension information of the input image in the model; specifically:

[0059] The texture image x is input into the fully connected layer, and the linear embedding Y0 is obtained and input into the first MLP block, which is specifically expressed as:

[0060] Y0=W0x+b0 (1)

[0061] Where x∈R S×C , Y0∈R S×C , S = H × W, H and W represent the height and width of the input texture image, C represents the number of channels of the input texture image, W0 and b0 are the parameters of the fully connected layer, W0∈R C×C , b0∈R HW×C ;

[0062] Each MLP block adopts the same structure, such as Figure 4 As shown, the calculation process of the MLP block is expressed as:

[0063] Y1=W1(LN(input))+b1 (2)

[0064] Y2=W2(BLP(LN(input)))+b2 (3)

[0065] Y3=W3(Y1+Y2)+b3 (4)

[0066] Y=Y3+input (5)

[0067] Y4=W4(LN(Y T ))+b4 (6)

[0068] output=(Y4+Y T ) T (7)

[0069] Among them, input and output represent the input and output of the MLP block respectively, input∈R S×C , output∈R S×C ; LN (Layer Normalization) represents normalization processing, BLP (Bilinear Pooling) represents bilinear pooling processing; W1, W2, W3, W4 and b1, b2, b3, b4 are all parameters of the fully connected layer, W1, W2, W3 are calculated in the feature space dimension, W1, W2, W3∈R S×S , W4 is calculated in the feature channel dimension, W4∈R C×C ; Equation (2) to (5) correspond to Figure 4 The left dashed box part in , equation (6) corresponds to equation (7) Figure 4 The right dotted box part in the figure;

[0070] The output of the last MLP block, output′, is used as the input of the classification head, which consists of a fully connected layer and a softmax function, specifically expressed as:

[0071] P=softmax(W5(output′)+b5) (8)

[0072] Where P is the predicted probability vector, P∈R 1×t , t is the number of categories. In the pre-training stage of this embodiment, t is 47, corresponding to the number of texture types; W5 and b5 are the parameters of the fully connected layer, W5∈R t×SC , b5∈R 1×t ;

[0073] Finally, the cross entropy loss is used as the loss function, and the backbone network is pre-trained by back propagation technology based on the public Describable Texture Dataset (DTD); the loss function is expressed as:

[0074]

[0075] Among them, P(x i =c j ) represents the corresponding category c in the predicted probability vector P for the i-th sample j The probability of , t is the number of categories, n is the number of samples; when the label of the i-th sample is category c j When ij =1, otherwise, g ij =0;

[0076] The above pre-training process uses a public texture dataset. The Describable Texture Dataset (DTD) contains 47 textures, 120 images per category, and a total of 5640 images. The dataset is divided into three parts: training, validation, and testing. Each part contains 10 subsets, and each subset contains 1880 images, including all categories, and 40 images per category.

[0077] Step 2. Send the original tongue image data to the backbone network for feature extraction to obtain tongue image feature maps at multiple levels;

[0078] The original tongue image data adopts the public data set BioHit, which is collected in a good environment. Its original annotations are tongue image area and background area annotations, which are used to train tongue image segmentation. In the present invention, the tongue image without background is obtained by using the tongue image area originally manually annotated in the data set as the original tongue image, and cracks, punctures, tooth marks, and others are selected as labels for annotation;

[0079] The pre-trained backbone network is used to extract features from the tongue image data. The output of each MLP block in the backbone network is obtained as the feature extraction result. The MLP blocks from shallow to deep output the tongue image feature maps of the corresponding levels. The tongue image feature maps of each level contain semantic information to varying degrees.

[0080] Step 3. The original tongue image and its corresponding tongue feature map are used as the target image and sent into the iterative function system to generate a fractal image. The specific process is as follows:

[0081] Step 3.1 Randomly generate N pairs of position transformation parameters {A, b} N , where A∈[-1,1], b∈[-1,1];

[0082] Step 3.2 Randomly sample T pairs from N pairs of position transformation parameters to form a position transformation parameter set {A, b} T , T is the number of generation steps;

[0083] According to the position transformation parameter set {A, b} T , the iterative function system generates the pixel coordinates corresponding to the fractal image. The specific process is:

[0084] v (t) =A (t) v (t-1) +b (t) , t=1,2,...,T (6)

[0085] Among them, v (t) represents the pixel coordinates generated in step t, v (0) Indicates the preset initial pixel coordinates, v 0 ∈R 1×2 , can be set to (0,0); {A (t) , b (t)} represents the position transformation parameter pair used in step t;

[0086] Step 3.3: Rendering and generating a fractal image according to the pixel coordinates generated by the iterative function system;

[0087] In step 3.1, T pixel coordinates v(1)~v(T) are generated through T steps. The image is rendered based on these pixel coordinates. This process is equivalent to calculating the distance of each pixel from the T pixel coordinates on a H×W black canvas to represent the brightness of the pixel. The pixel values ​​exceeding 1 are set to 1. Specifically, it is expressed as:

[0088] I′(h,w)=∑ t exp(-||(h,w)-v (t) || / τ), h∈H, w∈W, t=1,2,...,T (7)

[0089] Where (h, w) represents the pixel coordinates, I′(h, w) represents the brightness of the pixel with the pixel coordinates (h, w) in the generated image, and ||·|| represents the L2 norm operator; τ is a hyperparameter, τ∈(0,1], which is used to control the brightness range of each pixel in the generated image. The larger τ is, the higher the brightness of each pixel will be.

[0090] Step 3.4 calculates the loss of the iterated function system;

[0091] The target image is an RGB color image. The target image is converted into a grayscale image and normalized to [0, 1]. The conversion process and normalization process are respectively expressed as:

[0092]

[0093] Among them, I gray Represents a grayscale image, I raw represents the target image, Represents the R, G, and B channel components of the target image; I(h, w) represents the brightness of the pixel with pixel coordinates (h, w) in the normalized grayscale image. Indicates the minimum and maximum brightness in the grayscale image;

[0094] The rendered fractal image and the target image are subjected to pixel-by-pixel loss calculation, which is specifically expressed as:

[0095]

[0096] Where D′ and D represent the differential box-counting of the generated image and the target image, respectively. The ratio of the two makes the generated image and the target image closer in fractal dimension. I′ and, respectively represent the brightness matrices of the generated image and the target image, I, I′∈R H×W ;

[0097] Step 3.5 Update the position transformation parameter set {A, b} using the gradient descent method based on the loss T , that is, repeat steps 3.2 to 3.4, iterate until the number of iterations is reached or the loss reaches the threshold condition, and after the iteration is completed, a fractal image similar to the target image and with fractal features is obtained;

[0098] Step 4. Use the generated fractal image as supplementary data, annotate the generated image label according to the label of the target image, and enhance the tongue image dataset. The enhanced dataset can be sent to an independent tongue image classification model for training to improve the performance of the tongue image classification model.

[0099] The above description is only a specific implementation mode of the present invention. Any feature disclosed in this specification, unless otherwise stated, can be replaced by other alternative features that are equivalent or have similar purposes; all the disclosed features, or all the steps in the methods or processes, except for mutually exclusive features and / or steps, can be combined in any way.

Claims

1. A tongue image data enhancement method based on fractal features, characterized in that: The following steps are involved: Step 1. Use the public Describable Texture Dataset (DTD) to pre-train the backbone network, where the backbone network adopts the MLP-Like model; Step 2. Send the original tongue image data to the backbone network for feature extraction to obtain tongue image feature maps at multiple levels; Step 3. The original tongue image and its corresponding tongue feature map are used as target images and sent into the iterative function system to generate a fractal image; Step 4. Use the generated fractal image as supplementary data, annotate the label of the generated image according to the label of the target image, and complete the data enhancement of the tongue image dataset.

2. The tongue image data enhancement method based on fractal features according to claim 1 is characterized in that: In step 1, the MLP-Like model includes a fully connected layer, multiple MLP blocks and a classification head connected in sequence; The texture image x is input into the fully connected layer, and the linear embedding Y0 is obtained and input into the first MLP block, which is specifically expressed as: Y0=W0x+b0 Among them, W0 and b0 are the parameters of the fully connected layer; Each MLP block adopts the same structure, which is specifically expressed as: Y1=W1(LN(input))+b1 Y2=W2(BLP(LN(inpuy)))+b2 Y3=W3(Y1+Y2)+b3 Y=Y3+input Y4=W4(LN(Y T ))+b4 output=(Y4+Y T ) T Among them, input and output represent the input and output of the MLP block respectively, LN represents normalization processing, BLP represents bilinear pooling processing; W1, W2, W3, W4 and b1, b2, b3, b4 are all parameters of the fully connected layer; The output of the last MLP block, output′, is used as the input of the classification head, which consists of a fully connected layer and a softmax function, specifically expressed as: P = softmax(W5(output′)+b5) Among them, P is the prediction probability vector, and W5 and b5 are the parameters of the fully connected layer.

3. The tongue image data enhancement method based on fractal features according to claim 1 is characterized in that: In step 1, the pre-training process is: Using cross entropy loss as the loss function, it is specifically expressed as: Among them, P(x i =c j ) represents the corresponding category c in the predicted probability vector P for the i-th sample j The probability of , t is the number of categories, n is the number of samples; when the label of the i-th sample is category c j When ij =1, otherwise, g ij =0; According to the Describable Texture Dataset (DTD), the backbone network is pre-trained by back-propagation technology.

4. The tongue image data enhancement method based on fractal features according to claim 1 is characterized in that: In step 2, the original tongue image data adopts the public data set BioHit, and the background-free tongue image in the data set is used as the original tongue image, and cracks, punctures, teeth marks, and others are selected as labels for annotation; The pre-trained backbone network is used to extract features of the tongue image data, and the output of each MLP block in the backbone network is obtained as the feature extraction result. The MLP blocks from shallow to deep output the tongue image feature maps of the corresponding levels.

5. The tongue image data enhancement method based on fractal features according to claim 1 is characterized in that: In step 3, the specific process is: Step 3.1 Randomly generate N pairs of position transformation parameters {A, b} N , where A∈[-1,1], b∈[-1,1]; Step 3.2 Randomly sample T pairs from N pairs of position transformation parameters to form a position transformation parameter set {A, b} T , T is the number of steps; according to the position transformation parameter set {A, b} T , the iterative function system generates the pixel coordinates corresponding to the fractal image. The specific process is: v (t) =A (t) v (t-1) +b (t) ,t=1,2,…,T Among them, v (t) represents the pixel coordinates generated in step t, v (0) Indicates the preset initial pixel coordinates, {A (t) ,b (t) } represents the position transformation parameter pair used in step t; Step 3.3 generates a fractal image by rendering according to the pixel coordinates generated by the iterative function system, which is specifically expressed as: I′(h,w)=∑ t exp(-||(h,w)-v (t) || / τ),h∈H,w∈W,t=1,2,…,T Where (h, w) represents the pixel coordinates, I′(h, w) represents the brightness of the pixel with the pixel coordinates (h, w) in the generated image, ||·|| represents the L2 norm operator, τ is a hyperparameter, τ∈(0,1]; Step 3.4 calculates the loss of the iterated function system; Convert the target image to a grayscale image and normalize it, which can be expressed as follows: Among them, I gray Represents a grayscale image, Represents the target image I raw The R, G, and B channel components of the grayscale image; I(h,w) represents the brightness of the pixel with the pixel coordinates (h,w) in the normalized grayscale image. Indicates the minimum and maximum brightness in the grayscale image; The fractal image and the target image are subjected to pixel-by-pixel loss calculation, which can be specifically expressed as: Where D′ and D represent the differential box-counting of the generated image and the target image, respectively; I′ and I represent the brightness matrix of the generated image and the target image, respectively; Step 3.5 Update the position transformation parameter set {A, b} using the gradient descent method based on the loss T , repeat steps 3.2 to 3.4 for iteration until the number of iterations is reached or the loss reaches the threshold condition. After the iteration is completed, the fractal image corresponding to the target image is obtained.

Images