Tongue color correction method based on deep convolutional neural network

Abstract

The tongue color correction method based on a deep convolutional neural network relates to the field of medical image color correction, and solves the problems of color deviation of tongue image data generated by mobile devices in non-standard light source environment, tongue color image color distortion and the like, and the like.The present application collects picture data under standard illumination with a mobile device, uniformly scales, and enhances data; then collects data under a color-biased light source environment, and uniformly scales. The dataset under the standard light source is trained using a deep convolutional neural network, and the output result of the neural network is given to the downstream to perform a classification task on each pixel point. Each pixel point is converted from an RGB space to an HSV space, the distance of each current pixel point from the nearest point on the HSV space is calculated, the nearest pixel value is taken as a standard value, the RGB value corresponding to the HSV is finally calculated for color space restoration, and finally, tongue image color correction is realized. The present application restores the true color of the tongue image to the greatest extent.

Description

Technical Field

[0001] This invention relates to the field of medical image color correction, and more specifically to a tongue color correction method based on deep neural networks. Background Technology

[0002] Tongue diagnosis, as a distinctive and effective diagnostic method in Traditional Chinese Medicine, continues to play an important role in clinical practice today. Currently, color correction primarily employs two methods: one is image analysis-based correction, such as mean white balance, perfect reflection, and the gray-world hypothesis; the other is color correction based on deep fully convolutional networks. The problems with these two methods are as follows:

[0003] 1. They did not refer to a specific standard, such as a standard color patch, during the calibration process, so the final calibration result cannot be measured by the pixel value of the standard color patch;

[0004] 2. Even when correction is based on a fully convolutional network, the true color of each pixel cannot be predicted in many cases due to the influence of the network structure. Some images may even have pixel value overflow due to local exposure problems.

[0005] 3. Both of the above methods are based on the idea of ​​overall correction, and cannot perform one-to-one mapping correction for each pixel, so the correction accuracy is not ideal.

[0006] In summary, the novel tongue image correction method proposed in this invention has certain reference value for research combining deep learning and tongue diagnosis. Summary of the Invention

[0007] The purpose of this invention is to solve the problem of color deviation in tongue image data generated by mobile devices under non-standard light source environments, and to provide a tongue color correction method based on deep neural networks.

[0008] A tongue color correction method based on deep neural networks, comprising the following steps:

[0009] Step 1: Acquire image data and divide the image data into a pre-trained dataset and a test set to be calibrated;

[0010] Step 2: Construct a Unet-based deep neural network model for the pre-training dataset;

[0011] Step 3: Input the pre-trained dataset into the neural network model constructed in Step 2 for training to obtain the trained neural network model;

[0012] Step 4: Input the test set to be corrected into the neural network model trained in Step 3 for prediction, and obtain the prediction results; specifically:

[0013] Step 4: Map the output prediction results from RGB space to HSV space, calculate the similarity distance between the predicted HSV pixel value and the HSV pixel in space, and find the nearest pixel.

[0014] Step 42: Use the nearest pixel in the HSV space as the standard pixel for color correction, and finally convert the HSV space to RGB space to achieve color correction.

[0015] The present invention has the following beneficial effects:

[0016] The tongue color correction method described in this invention proposes a network structure based on Unet. First, in the encoding-decoding stage, a deep fully convolutional network is used to extract color block information that is close to the tongue color under a standard light source. Then, at the output end, the softmax multi-classification idea is used to classify each pixel information of the extracted tongue image features and map the pixels to a standard color space to calculate the similarity distance. Then, the pixel value of the nearest standard color block is used as the standard value for correction. Through the combination of fully convolutional network and standard color block mapping idea, the tongue color can be corrected to the greatest extent.

[0017] The tongue color correction method described in this invention combines the advantages of various classic networks in a deep neural network structure. It treats color correction as a multi-classification task for prediction. By comparing the predicted classification result with the distance of the standard pixel in the HSV space, the nearest standard pixel is selected as the pixel for color correction, thereby maximizing the restoration of the true color of the tongue image. Attached Figure Description

[0018] Figure 1 This is a flowchart of the tongue color correction method based on deep neural networks described in this invention;

[0019] Figure 2 This is a schematic diagram of the structure of a deep neural network. Detailed Implementation

[0020] Specific Implementation Method 1: Combination Figure 1 and Figure 2 This embodiment describes a tongue color correction method based on deep neural networks, which is implemented through the following steps:

[0021] Step 1: Acquire image data.

[0022] Step 1.1: Data collection. First, the dataset collected by mobile devices under standard light sources is scaled to a uniform size and data augmented to serve as a pre-training dataset. Then, test datasets to be corrected are collected by mobile devices under various color-biased light sources.

[0023] Step 1.2: Data Processing. The image data is uniformly scaled. General scaling methods can lead to image distortion, so the letterbox method is used for scaling. Then, the scaled data is augmented using methods such as flipping, translating, and rotating. The test dataset collected under a color-biased light source is scaled in the same way, but without data augmentation. Specifically, the pre-training dataset is flipped horizontally and vertically; the tongue image is rotated by 5°, 10°, 15°, 30°, and 45° respectively for data augmentation.

[0024] Step 2: Construct a deep neural network based on the characteristics of the Unet network for training.

[0025] Step 2.1: The entire deep neural network is built using the TensorFlow deep learning framework. The entire neural network consists of three parts: backbone (downsampling layers), neck (depthiably separable convolutional layers), and head (upsampling layers). The construction of the entire convolutional neural network is as follows: Figure 2 As shown.

[0026] The specific implementation of the backbone downsampling layer is as follows: The backbone downsampling uses the NIN module from GooglenetInceptionV1 for feature fusion and extraction. The advantages of using this module are: using convolutions of different sizes in parallel within the same layer increases the network's adaptability to scale while increasing the width; convolutional kernels with multiple views are arranged side by side, allowing the network to learn information useful to itself; and the 1×1 convolutional kernel reduces the number of channels and fuses information from each channel, significantly reducing the computational cost of parameters while achieving dimensionality reduction. In other words, the characteristics of NIN can effectively fuse objects of different sizes in the feature map, providing better local abstraction capabilities. Furthermore, each convolution is processed by batch normalization after passing through the ReLU activation function, ensuring the data distribution before and after convolution is as consistent as possible. This stage uses the idea of ​​three consecutive NINs, each followed by a max pooling operation with a stride of 2 for dimensionality reduction and downsampling. The number of NIN concatenations is flexible, allowing for adjustments to the number of NIN modules as needed to control the network depth.

[0027] In this implementation, to prevent gradient vanishing or explosion during backpropagation parameter updates due to increased network depth, the shortcut concept from residual networks is employed. The convolution kernel size is set to 1×1 during shortcut, and the stride can be customized based on the input image size, allowing an add operation to be performed each time the NIN module outputs.

[0028] The specific implementation of the Neck-depth-separable convolutional layer is as follows: it employs three consecutive depth-separable convolutions. Compared to ordinary convolution, this method requires fewer parameters. Crucially, depth-separable convolution separates channels and regions, making model deployment more lightweight. The specific parameters for the three depth-separable convolutions are set as follows: the first depth-separable convolutional layer has 256 kernels, a kernel size of 1×1, a stride of 1, and outputs 2 channels in the depth direction for each input channel with a dilation factor of 1; the second depth-separable convolutional layer has 256 kernels, a kernel size of 1×1, a stride of 1, and outputs 2 channels in the depth direction for each input channel with a dilation factor of 2; the third depth-separable convolutional layer has 256 kernels, a kernel size of 1×1, a stride of 1, and outputs 2 channels in the depth direction for each input channel with a dilation factor of 4. Finally, a skip connection is established between the input and output ends.

[0029] The head-upsampling layer specifically employs three consecutive upsampling operations, with an input block size of 2. Upsampling is achieved through PixelShuffle, a method that significantly addresses the issue of low resolution compared to standard upsampling. This is because PixelShuffle's primary function is to reconstruct high-resolution feature maps from low-resolution ones through multi-channel convolutional kernels, thus becoming an effective upsampling technique for solving super-resolution problems.

[0030] In this implementation, concatenation (or stacking) is performed along the channel dimension during each upsampling step to fuse feature maps of different scales. The purpose of this upsampling method is to improve the accuracy of downstream classification tasks. The resulting feature maps are then subjected to global average pooling and finally connected to a fully connected layer to perform multi-class classification.

[0031] Step 3: Input the pre-trained dataset in batches into the constructed neural network model for training. The specific implementation is as follows:

[0032] The dataset is divided in a 5:1 ratio. 5 / 6 of the data is used as the training set to update model parameters. 1 / 5 of the training set is used as the validation set to select model parameters. If overfitting occurs, training can be terminated early. 1 / 6 is used as the test set to prevent data contamination when selecting the final model for testing. Cross-validation is used during training to verify whether the model is overfitting.

[0033] In this implementation, the number of training epochs is set to 50, the batch size of each input to the network model is set to 32, and the learning rate is set to dynamic, that is, the learning rate is initially large and then small as the number of training epochs increases, making the model convergence process more flexible. The loss function adopted is the cross-entropy loss function.

[0034] In this embodiment, the gradient optimizer is Adam. The reason for choosing this optimizer is that it has great advantages in non-convex function optimization: parameter updates are not affected by gradient scaling transformations; hyperparameters have good interpretability and require little or no adjustment; the update step size can be limited to a general range; it can naturally implement the annealing process (automatically adjust the learning rate); and it is suitable for unstable objective functions, etc. In other words, the Adam algorithm is different from the traditional gradient descent algorithm. The traditional gradient descent algorithm maintains a single learning rate to update weights, which does not change during training. Adam, on the other hand, designs independent adaptive learning rates for different parameters by calculating the first-order moment estimate and the second-order moment estimate of the gradient.

[0035] The formula for the cross-entropy loss function is as follows:

[0036]

[0037] In the formula, y is the label for the predicted value, and y is the label for the actual value.

[0038] Step 4: Input the data obtained under the color-biased light source into the trained model for prediction. Calculate the distance between the predicted pixel and the standard pixel in the HSV space, and use the closest pixel as the standard pixel for color correction, ultimately achieving color correction of the tongue image. The specific implementation is as follows:

[0039] The color cast data, used for model color correction, is only scaled to a uniform size without data augmentation to avoid interference during testing and to better test the model's color correction performance. The specific correction process for the tongue image is as follows:

[0040] First, multi-classification is performed based on the model's prediction results, with each pixel classified according to 143 categories in the color lookup table.

[0041] First, the pixels of the 143 color categories are converted from RGB space to HSV space. Then, the positions of the pixels of the 143 color categories are determined in the three-dimensional space formed by HSV.

[0042] The calculation model predicts which color category a pixel's HSV value is closest to in the HSV 3D space. The pixel with the closest color category is then used as the standard pixel for correction. Finally, the RGB value corresponding to the HSV is calculated for color space restoration, thus achieving color correction of the tongue image.

[0043] The formula for converting RGB color space to HSV color space is as follows:

[0044] R′=R / 255

[0045] G′=G / 255

[0046] B′=B / 255

[0047] C max =max(R′,G′,B′)

[0048] C min =min(R′,G′,B′)

[0049] Δ=C max -C min

[0050] In the formula: R is the red channel of the image, G is the green channel, and B is the blue channel. R′, G′, and B′ are the red, green, and blue channels respectively converted to the HSV color space. C max and C min These represent the maximum and minimum values ​​of the three color channels, respectively, with Δ representing the maximum value C. max With minimum value C min The difference.

[0051] Hue (H) calculation:

[0052]

[0053] Saturation (S) calculation:

[0054]

[0055] Brightness Value (V) calculation:

[0056] V = C max

[0057] The specific implementation of the HSV to RGB conversion formula is as follows:

[0058] C = V × S

[0059] X = C × (1 - |(H / 60)) 0 )mod2-1|)m=VC

[0060]

[0061] (R, G, B) = ((R′+m)×255, (G′+m)×255, (B′+m)×255)

[0062] In the formula: V represents brightness, S represents saturation, H represents hue, C represents the product of brightness V and saturation S, X represents the product of C and hue H after arithmetic operations, and m represents the difference between brightness V and saturation S.

[0063] The spatial distance formula used is as follows, where the letter subscript p represents the predicted value in the HSV space, and the subscript r represents the standard value of the color category in the HSV space:

[0064]

[0065] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. The scope of patent protection of the present invention shall be determined by the claims. Similarly, any equivalent structural changes made based on the description and drawings of the present invention shall also be included within the scope of protection of the present invention.

Claims

1. A tongue color correction method based on deep neural networks, characterized by: This method is implemented by the following steps: Step 1: Acquire image data and divide the image data into a pre-trained dataset and a test set to be calibrated; Step 2: Construct a Unet-based deep neural network model for the pre-training dataset; The specific process of constructing a deep neural network model based on Unet is as follows: Step 2: First, define the neural network model as consisting of downsampling layers, depthwise separable convolutional layers, and upsampling layers. The downsampling layer uses the NIN module for multi-scale feature fusion four times in a row, and the shorting operation in the residual network is used at the input and output of each NIN module. An addition operation is performed each time the NIN module outputs. The depthwise separable convolutional layer employs three consecutive depthwise separable convolutions while simultaneously skipping the input and output terminals of this part; The upsampling layer employs three consecutive upsampling operations, achieved through PixelShuffle. After each upsampling, the layers are concatenated and stacked along the channel dimension to fuse feature maps of different scales. Step 3: Input the pre-trained dataset into the neural network model constructed in Step 2 for training to obtain the trained neural network model; Step 4: Input the color cast data as the test set to be corrected into the neural network model trained in Step 3 for prediction, and obtain the prediction results; specifically: Step 4: Map the output prediction results from RGB space to HSV space, calculate the similarity distance between the predicted HSV pixel value and the pixel in HSV space, and find the nearest pixel. Step 42: Use the nearest pixel in the HSV space as the standard pixel for color correction, and finally convert the HSV space to RGB space to achieve color correction.

2. The tongue color correction method based on deep neural networks according to claim 1, characterized in that: The specific process in step one is as follows: Step 11: Data Collection. First, the dataset collected under standard light source is scaled to a uniform size using a mobile device and then augmented to serve as a pre-training dataset. The test dataset to be calibrated was collected using mobile devices under various color-biased light sources; Steps 1 and 2: Data processing. The pre-trained dataset is scaled to a uniform size using the letterbox method. Then, the scaled data is augmented by flipping, translating, and rotating. The test dataset to be corrected, collected under various color-biased light sources, was scaled using the letterbox method. The specific flipping method for the pre-trained dataset is: horizontal and vertical flipping; The specific rotation method is as follows: the pre-trained dataset is rotated by 5°, 10°, 15°, 30° and 45° respectively to augment the data.

3. The tongue color correction method based on deep neural networks according to claim 1, characterized in that: The specific process of step three is as follows: The dataset is divided in a 5:1 ratio; Of the data, 5 / 6 is used as the training set to update the model parameters; 1 / 5 of the training set is designated as the validation set to select model parameters. If overfitting occurs, training is terminated early. One-sixth of the data was used as the test set, and cross-validation was used during training to verify whether the model was overfitting.

4. The tongue color correction method based on deep neural networks according to claim 3, characterized in that: The training epochs are set to 50, the batch size of each input to the network model is set to 32, the learning rate is set to dynamic, and the cross-entropy loss function is used.

Images