A tongue image data enhancement method based on fractal features

Abstract

The application belongs to the technical field of image processing, and specifically provides a tongue image data enhancement method based on fractal features, which uses fractal features to enhance tongue image data; first, a public texture dataset is used to pre-train a backbone network (MLP-Like model); then, original tongue image data is input into the backbone network for feature extraction to obtain a hierarchical tongue feature map, and the original tongue image and the corresponding tongue feature map are taken as a target image, which is input into an iterative function system to generate a fractal image; finally, the generated fractal image is taken as supplementary data, and the labels of the generated image are labeled according to the labels of the target image, so that data enhancement of the tongue image dataset is completed. The fractal image with fractal features is generated by using the iterative function system, and the tongue image dataset is supplemented, so that the tongue image dataset is supplemented as a data enhancement method combined with fractal features, and the tongue image dataset imbalance problem can be effectively alleviated.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and specifically provides a tongue image data enhancement method based on fractal features. Background Art

[0002] Tongue diagnosis mainly uses neural networks to extract tongue images, and then conducts subsequent auxiliary diagnosis and treatment, such as inferring symptoms and physical constitution. However, in actual applications, there is a long-tail problem in the distribution of tongue images, that is, compared with normal tongue images, there are relatively few samples of cracks, tooth marks, etc. During the model training process, data imbalance problems will cause the generalization ability of the model to be affected, which in turn leads to a decrease in model accuracy. In order to improve model performance, it is usually necessary to adopt data enhancement solutions to increase the sample size and improve the generalization ability of the model.

[0003] At present, the tongue image data enhancement method is still relatively simple, including flipping, mirroring and other methods; for example, the document "S.Lin,X.Song,Z.xu,et al,Moment Invariants with Data Augmentation for Tongue Image Segmentation[C].in:Proceedings of IEEE International Conference onBioinformatics and Biomedicine,Houston,2021,pp.990-991" proposes a tongue image enhancement method based on moment invariants to assist in tongue segmentation. This method performs different rotation angles, horizontal flipping, noise addition and brightness adjustment on each tongue image to enhance the data. Then, by adding moment invariant loss to the loss, the difference between the samples enhanced by different enhancement methods and the mean of the enhanced samples is used as the loss to prevent the deviation of data enhancement; for example, the document "Y.Zhang,T.Yao,Z.Qiu,et al.Lightweight and Progressively-Scalable Networksfor Semantic Segmentation[J].International Journal of Computer Vision, 2022, 130(4): 994-997” provides a method for pre-training using fractal images instead of real images. An iterative function system is used to generate a large number of fractal images similar to natural objects, which are then used for model pre-training. This allows the model to learn universal features without using large-scale real data sets, thereby improving the generalization performance of the model in general fields. The model parameters are then fine-tuned using data from the target field.

[0004] However, in the above two documents, the former is not targeted in enhancing the features of tongue images and does not consider the texture features of tongue images, such as the tongue coating texture on the tongue surface and the vein texture on the tongue bottom. The fractal features of the tongue surface are closely related to the texture, and the fractal features of healthy tongue images and pathological tongue images have been shown to have obvious differences; the latter is only targeted at general fields. In the special field of tongue images, simply using non-targeted fractal images for pre-training consumes a lot of time and cost, and cannot achieve better performance improvement effects. Moreover, this method simply uses an iterative function system for generation and cannot perform learnable generation for the target image. Summary of the Invention

[0005] The purpose of the present invention is to provide a tongue image data enhancement method based on fractal features, which utilizes fractal features to enhance tongue image data. Specifically, after gradually collecting tongue image features through a deep neural network (MLP-Like model), an iterative function system is used to generate tongue image images with fractal features to supplement the tongue image dataset. As a data enhancement method combining fractal features, the present invention can effectively alleviate the data imbalance problem of tongue image datasets.

[0006] To achieve the above object, the technical solution adopted by the present invention is:

[0007] A tongue image data enhancement method based on fractal features, characterized by comprising the following steps:

[0008] Step 1. Use the public Describable Texture Dataset (DTD) to pre-train the backbone network, which adopts the MLP-Like model;

[0009] Step 2: The original tongue image data is fed into the backbone network for feature extraction to obtain tongue image feature maps at multiple levels;

[0010] Step 3. The original tongue image and its corresponding tongue feature map are used as target images and sent into the iterative function system to generate a fractal image;

[0011] Step 4. Use the generated fractal image as supplementary data, annotate the generated image with the label according to the label of the target image, and complete the data enhancement of the tongue image dataset.

[0012] Furthermore, in step 1, the MLP-Like model includes a fully connected layer, multiple MLP blocks, and a classification head connected in sequence; the texture image x is input to the fully connected layer, and the linear embedding Y0 is obtained and input to the first MLP block, which is specifically expressed as:

[0013] Y0=W0x+b0

[0014] Among them, W0 and b0 are the parameters of the fully connected layer;

[0015] Each MLP block uses the same structure, specifically expressed as:

[0016] Y1=W1(LN(input))+b1

[0017] Y2=W2(BLP(LN(input)))+b2

[0018] Y3=W3(Y1+Y2)+b3

[0019] Y=Y3+input

[0020] Y4=W4(LN(Y T ))+b4

[0021] output=(Y4+Y T ) T

[0022] Where input and output represent the input and output of the MLP block respectively, LN represents normalization, and BLP represents bilinear pooling. W1, W2, W3, W4 and b1, b2, b3, b4 are all parameters of the fully connected layer.

[0023] The output of the last MLP block, output′, is used as the input of the classification head, which consists of a fully connected layer and a softmax function, specifically expressed as:

[0024] P = softmax(W5(output′)+b5)

[0025] Among them, P is the prediction probability vector, W5 and b5 are the parameters of the fully connected layer.

[0026] Furthermore, in step 1, the pre-training process is:

[0027] The cross entropy loss is used as the loss function, which is specifically expressed as:

[0028]

[0029] Among them, P(x i =c j ) represents the corresponding category c in the predicted probability vector P for the i-th sample j The probability of t is the number of categories, n is the number of samples; when the label of the i-th sample is category c j When g ij =1, otherwise, g ij =0;

[0030] According to the Describable Texture Dataset (DTD), the backbone network is pre-trained by back-propagation technology.

[0031] Furthermore, in step 2, the original tongue image data adopts the public dataset BioHit, and the background-free tongue image in the dataset is used as the original tongue image, and cracks, punctures, teeth marks, and other are selected as labels for annotation;

[0032] The pre-trained backbone network is used to extract features from the tongue image data. The output of each MLP block in the backbone network is obtained as the feature extraction result. The MLP blocks from shallow to deep output the tongue image feature maps of the corresponding levels.

[0033] Furthermore, in step 3, the specific process is:

[0034] Step 3.1 Randomly generate N pairs of position transformation parameters {A, b} N , where A∈[-1,1], b∈[-1,1];

[0035] Step 3.2 Randomly sample T pairs from N pairs of position transformation parameters to form a position transformation parameter set {A, b} T , T is the number of steps; according to the position transformation parameter group {A, b} T , the iterative function system generates the pixel coordinates corresponding to the fractal image. The specific process is:

[0036] v (t) =A (t) v (t-1) +b (t) , t=1,2,...,T

[0037] Among them, v (t) Indicates the pixel coordinates generated in step t, v (0) Indicates the preset initial pixel coordinates, {A (t) , b (t)} represents the position transformation parameter pair used in step t generation;

[0038] Step 3.3 generates a fractal image by rendering based on the pixel coordinates generated by the iterative function system, which is specifically expressed as follows:

[0039] I′(h, w)=∑ t exp(-||(h,w)-v (t) || / τ), h∈H, w∈W, t=1,2,...,T

[0040] Where (h, w) represents the pixel coordinates, I′(h, w) represents the brightness of the pixel with the pixel coordinates (h, w) in the generated image, ||·|| represents the L2 norm operator, and τ is a hyperparameter, τ∈(0,1];

[0041] Step 3.4 calculates the loss of the iterated function system;

[0042] The target image is converted into a gray image and normalized, and is specifically expressed as:

[0043]

[0044] wherein I gray represents a gray image, represents R, G, B channel components of the target image I raw ; I(h, w) represents the brightness of a pixel point with pixel coordinates (h, w) in the normalized gray image, represents the minimum and maximum values of the brightness in the gray image;

[0045] The fractal image and the target image are subjected to pixel-by-pixel loss calculation, and is specifically expressed as:

[0046]

[0047] wherein D' and D respectively represent differential box-counting of the generated image and the target image, and I' and I respectively represent brightness matrices of the generated image and the target image;

[0048] Step 3.5. The position transformation parameter set {A, b} is updated according to the loss using a gradient descent method. T Steps 3.2 to 3.4 are repeated for iteration until the iteration number or the loss reaches a threshold condition, and the fractal image corresponding to the target image is obtained after the iteration is completed.

[0049] Based on the above technical solution, the application has the following advantages:

[0050] The MLP-Lik e neural network model and the iterative function system technology are adopted, the MLP-Lik e model does not change the size of the input image, the MLP-Lik e model is used for learning texture features, and then the tongue image texture features are extracted by using the MLP-Lik e model, the intermediate feature maps of the shallow layer and the deep layer of the MLP-Lik eThe model's output representation serves as the target image, and a controllable iterative function system is used to generate a fractal image, making the generated fractal image similar to the original image. Ultimately, the generated image is used as a supplement to the original tongue image dataset, thereby alleviating the imbalance in the number of samples in each category of the original tongue image dataset. Using the enhanced dataset to train a tongue image classification model is less prone to overfitting and has better generalization capabilities. Compared to traditional image deformation, moment invariants, and uncontrollable fractal image generation schemes, the present invention has the advantage of being more suitable for traditional Chinese medicine tongue image scenarios. Through controllable fractal image generation, tongue image data is enhanced, and the dataset contains more diverse texture features. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a flow chart of the fractal feature-based tongue image data enhancement method provided by the present invention.

[0052] Figure 2 This is a model architecture diagram of the tongue image data enhancement method based on fractal features provided by the present invention.

[0053] Figure 3 This is a schematic diagram of the structure of the MLP-Like model provided by the present invention.

[0054] Figure 4 This is a schematic diagram of the structure of the MLP block in the MLP-Like model provided by the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and beneficial effects of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0056] This embodiment provides a tongue image data enhancement method based on fractal features, the process of which is as follows: Figure 1 As shown, the model architecture is as Figure 2 As shown; specifically including the following steps:

[0057] Step 1. Use the public Describabl e Textur e Dataset, DTD) is used to pre-train the backbone network;

[0058] The backbone network adopts MLP-Lik e Models, such as Figure 3 As shown in FIG, it includes a fully connected layer, multiple MLP blocks and a classification head connected in sequence. The number of MLP blocks can be adaptively matched according to the application scenario. In this embodiment, 8 MLP blocks are used. The MLP blocks in the model play the role of encoding the spatial dimension information and channel dimension information of the input image. Specifically:

[0059] The texture image x is input to the fully connected layer, and the linear embedding Y0 is obtained and input to the first MLP block, which is specifically expressed as:

[0060] Y0=W0x+b0 (1)

[0061] Where x∈R S×C , Y0∈R S×C , S = H × W, H and W represent the height and width of the input texture image, C represents the number of channels of the input texture image, W0 and b0 are the parameters of the fully connected layer, W0∈R C×C , b0∈R HW×C ;

[0062] Each MLP block adopts the same structure, such as Figure 4 As shown, the calculation process of the MLP block is expressed as:

[0063] Y1=W1(LN(input))+b1 (2)

[0064] Y2=W2(BLP(LN(input)))+b2 (3)

[0065] Y3=W3(Y1+Y2)+b3 (4)

[0066] Y=Y3+input (5)

[0067] Y4=W4(LN(Y T ))+b4 (6)

[0068] output=(Y4+Y T ) T (7)

[0069] Among them, input and output represent the input and output of the MLP block respectively, input∈R S×C , output∈R S×C ; LN (Layer Normalization) represents normalization processing, BLP (Bilinear Pooling) represents bilinear pooling processing; W1, W2, W3, W4 and b1, b2, b3, b4 are all parameters of the fully connected layer, W1, W2, W3 are calculated in the feature space dimension, W1, W2, W3∈R S×S , W4 is calculated in the feature channel dimension, W4∈R C×C ; Equations (2) to (5) correspond to Figure 4 In the left dotted box, equations (6) and (7) correspond to Figure 4 The right dotted box part in the figure;

[0070] The output of the last MLP block, output′, is used as the input of the classification head, which consists of a fully connected layer and a softmax function, specifically expressed as:

[0071] P=softmax(W5(output′)+b5) (8)

[0072] Where P is the predicted probability vector, P∈R 1×t , t is the number of categories. In the pre-training stage of this embodiment, t is 47, corresponding to the number of texture types; W5 and b5 are the parameters of the fully connected layer, W5∈R t×SC , b5∈R 1×t ;

[0073] Finally, the cross entropy loss is used as the loss function, and the backbone network is pre-trained by back propagation technology based on the public Describable Texture Dataset (DTD); the loss function is expressed as:

[0074]

[0075] Among them, P(x i =c j ) represents the corresponding category c in the predicted probability vector P for the i-th sample j The probability of t is the number of categories, n is the number of samples; when the label of the i-th sample is category c j When g ij =1, otherwise, g ij =0;

[0076] The pre-training process uses a publicly available texture dataset. The Describable Texture Dataset (DTD) contains 47 textures, with 120 images per category, for a total of 5,640 images. The dataset is divided into three parts: training, validation, and testing. Each part contains 10 subsets, and each subset contains 1,880 images, covering all categories, with 40 images per category.

[0077] Step 2: The original tongue image data is fed into the backbone network for feature extraction to obtain tongue image feature maps at multiple levels;

[0078] The original tongue image data uses the public dataset BioHit. This dataset was collected under a good environment and its original annotations included the tongue image region and background region annotations, which were used to train tongue image segmentation. In this paper, the tongue image without background was obtained using the original manually annotated tongue image region of this dataset as the original tongue image, and cracks, punctures, tooth marks, and other labels were selected for annotation.

[0079] The pre-trained backbone network is used to extract features from the tongue image data. The output of each MLP block in the backbone network is used as the feature extraction result. From shallow to deep MLP blocks, the output of the tongue image feature map of the corresponding level is obtained. The tongue image feature map of each level contains different levels of semantic information.

[0080] Step 3. The original tongue image and its corresponding tongue feature map are used as target images and sent into the iterative function system to generate a fractal image. The specific process is as follows:

[0081] Step 3.1 Randomly generate N pairs of position transformation parameters {A, b} N , where A∈[-1,1], b∈[-1,1];

[0082] Step 3.2 Randomly sample T pairs from N pairs of position transformation parameters to form a position transformation parameter set {A, b} T , T is the number of generation steps;

[0083] According to the position transformation parameter set {A, b} T , the iterative function system generates the pixel coordinates corresponding to the fractal image. The specific process is:

[0084] v (t) =A (t) v (t-1) +b (t) , t=1,2,...,T (6)

[0085] Among them, v (t) Indicates the pixel coordinates generated in step t, v (0) Indicates the preset initial pixel coordinates, v 0 ∈R 1×2 , can be set to (0, 0); {A (t) , b (t)} represents the position transformation parameter pair used in step t generation;

[0086] Step 3.3: Rendering and generating a fractal image according to the pixel coordinates generated by the iterative function system;

[0087] The T pixel coordinates v(1) to v(T) are generated by T steps in step 3.1. The image is rendered based on these pixel coordinates. This process is equivalent to calculating the distance between each pixel and the T pixel coordinates on a H×W black canvas to represent the brightness of the pixel. The pixel values ​​exceeding 1 are set to 1. Specifically, it is expressed as:

[0088] I′(h, w)=∑ t exp(-||(h,w)-v (t) || / τ), h∈H, w∈W, t=1,2,...,T (7)

[0089] Where (h, w) represents the pixel coordinates, I′(h, w) represents the brightness of the pixel with coordinates (h, w) in the generated image, and ||·|| represents the L2 norm operator. τ is a hyperparameter, τ∈(0,1], which controls the brightness range of each pixel in the generated image. The larger τ is, the higher the brightness of each pixel will be.

[0090] Step 3.4 calculates the loss of the iterated function system;

[0091] The target image is an RGB color image. The target image is converted into a grayscale image and normalized to [0, 1]. The conversion process and normalization process are expressed as follows:

[0092]

[0093] Among them, I gray Represents a grayscale image, I raw represents the target image, Represents the R, G, and B channel components of the target image; I(h, w) represents the brightness of the pixel with pixel coordinates (h, w) in the normalized grayscale image. Indicates the minimum and maximum brightness in the grayscale image;

[0094] The rendered fractal image and the target image are subjected to pixel-by-pixel loss calculation, which can be expressed as follows:

[0095]

[0096] Where D′ and D represent the differential box-counting of the generated image and the target image, respectively. The ratio of the two makes the generated image and the target image closer in fractal dimension. I′ and , respectively represent the brightness matrices of the generated image and the target image, I, I′∈R H×W ;

[0097] Step 3.5 Update the position transformation parameter set {A, b} using gradient descent based on the loss T , that is, repeat steps 3.2 to 3.4 until the number of iterations is reached or the loss reaches the threshold condition. After the iteration is completed, a fractal image similar to the target image and with fractal features is obtained;

[0098] Step 4. Use the generated fractal image as supplementary data, annotate the generated image with the label according to the label of the target image, and enhance the tongue image dataset. The enhanced dataset can be sent to an independent tongue image classification model for training to improve the performance of the tongue image classification model.

[0099] The above description is only a specific embodiment of the present invention. Any feature disclosed in this specification, unless otherwise stated, can be replaced by other equivalent or alternative features with similar purposes; all disclosed features, or all steps in the methods or processes, except for mutually exclusive features and / or steps, can be combined in any way.

Claims

1. A tongue image data enhancement method based on fractal features, characterized in that: The following steps are involved: Step 1. Use the public Describable Texture Dataset (DTD) to pre-train the backbone network, which adopts the MLP-Like model; Step 2: The original tongue image data is fed into the backbone network for feature extraction to obtain tongue image feature maps at multiple levels; Step 3. The original tongue image and its corresponding tongue feature map are used as target images and sent into the iterative function system to generate a fractal image. The specific process is as follows: Step 3.1 Randomly generate N pairs of position transformation parameters {A, b} N , where A∈[-1,1], b∈[-1,1]; Step 3.2 Randomly sample T pairs from N pairs of position transformation parameters to form a position transformation parameter set {A, b} T , T is the number of steps; according to the position transformation parameter group {A, b} T , the iterative function system generates the pixel coordinates corresponding to the fractal image. The specific process is: v (t) =A (t) v (t-1) +b (t) ,t=1,2,…,T Among them, v (t) Indicates the pixel coordinates generated in step t, v (0) Indicates the preset initial pixel coordinates, {A (t) ,b (t) } represents the position transformation parameter pair used in step t generation; Step 3.3 generates a fractal image by rendering based on the pixel coordinates generated by the iterative function system, which is specifically expressed as follows: I′(h,w)=∑ t exp(-||(h,w)-v (t) || / τ),h∈H,w∈W,t=1,2,…,T Where (h, w) represents the pixel coordinates, I′(h, w) represents the brightness of the pixel with the pixel coordinates (h, w) in the generated image, ||·|| represents the L2 norm operator, and τ is a hyperparameter, τ∈(0,1]; Step 3.4 calculates the loss of the iterated function system; Convert the target image into a grayscale image and normalize it, which can be expressed as follows: Among them, I gray Represents a grayscale image, Represents the target image I raw The R, G, and B channel components of the grayscale image; I(h,w) represents the brightness of the pixel with the pixel coordinates (h,w) in the normalized grayscale image. Indicates the minimum and maximum brightness in the grayscale image; The pixel-by-pixel loss calculation of the fractal image and the target image is expressed as: Where D′ and D represent the differential box-counting of the generated image and the target image, respectively; i′ and i represent the brightness matrix of the generated image and the target image, respectively; Step 3.5 Update the position transformation parameter set {A, b} using the gradient descent method based on the loss T , repeat steps 3.2 to 3.4 for iteration until the number of iterations is reached or the loss reaches the threshold condition, and after the iteration is completed, the fractal image corresponding to the target image is obtained; Step 4. Use the generated fractal image as supplementary data, annotate the generated image with the label according to the label of the target image, and complete the data enhancement of the tongue image dataset.

2. The tongue image data enhancement method based on fractal features according to claim 1 is characterized in that: In step 1, the MLP-Like model consists of a fully connected layer, multiple MLP blocks, and a classification head connected sequentially; The texture image x is input to the fully connected layer, and the linear embedding Y0 is obtained and input to the first MLP block, which is specifically expressed as: Y0=W0x+b0 Among them, W0 and b0 are the parameters of the fully connected layer; Each MLP block uses the same structure, specifically expressed as: Y1=W1(LN(input))+b1 Y2=W2(BLP(LN(input)))+b2 Y3=W3(Y1+Y2)+b3 Y=Y3+input Y4=W4(LN(Y T ))+b4 output=(Y4+Y T ) T Where input and output represent the input and output of the MLP block, respectively; LN represents normalization; BLP represents bilinear pooling; W1, W2, W3, W4 and b1, b2, b3, b4 are all parameters of the fully connected layer; The output of the last MLP block, output′, is used as the input of the classification head, which consists of a fully connected layer and a softmax function, specifically expressed as: P = softmax(W5(output′)+b5) Among them, P is the prediction probability vector, W5 and b5 are the parameters of the fully connected layer.

3. The tongue image data enhancement method based on fractal features according to claim 1, characterized in that: In step 1, the pre-training process is: The cross entropy loss is used as the loss function, which is specifically expressed as: Among them, P9x i =c j ) represents the corresponding category c in the predicted probability vector P for the i-th sample j The probability of t is the number of categories, n is the number of samples; when the label of the i-th sample is category c j When g ij =1, otherwise, g ij =0; According to the Describable Texture Dataset (DTD), the backbone network is pre-trained by back-propagation technology.

4. The tongue image data enhancement method based on fractal features according to claim 1, characterized in that: In step 2, the original tongue image data adopts the public dataset BioHit. The background-free tongue image in the dataset is used as the original tongue image, and cracks, punctures, teeth marks, and other are selected as labels for annotation; The pre-trained backbone network is used to extract features from the tongue image data. The output of each MLP block in the backbone network is obtained as the feature extraction result. The MLP blocks from shallow to deep output the tongue image feature maps of the corresponding levels.

Images