A method and system for automatic color correction of children's traditional Chinese medicine tongue diagnosis images based on deep learning

Abstract

The present invention discloses a method and system for automatic color correction of children's traditional Chinese medicine tongue diagnosis images based on deep learning, comprising: (1) shooting a tongue diagnosis image containing a color standard card and collecting a color standard card reference image; (2) constructing an improved GAN model, comprising a generator and a discriminator; during the training process, the generator is responsible for receiving a random noise vector as input and generating an image similar to the color-corrected tongue diagnosis image; the discriminator is responsible for receiving the tongue diagnosis image generated by the generator and comparing the color standard card in the image with the color standard card reference image to distinguish between the tongue diagnosis image with real color and the tongue diagnosis image with color correction; (3) training the GAN model; (4) inputting the tongue diagnosis image containing the color standard card to be corrected into the generator, and outputting the corrected tongue diagnosis image. By using the present invention, the color of the tongue diagnosis image can be corrected to generate the color of the tongue diagnosis image as actually seen by the naked eye.

Description

Technical Field

[0001] The present invention belongs to the field of image correction, and in particular relates to a method and system for automatic color correction of children's traditional Chinese medicine tongue diagnosis images based on deep learning. Background Art

[0002] TCM diagnosis includes observation, auscultation, inquiry and palpation. Tongue diagnosis is an important part of TCM observation. The changes in tongue quality, tongue coating and tongue body can fully reflect the changes in qi, blood, yin and yang in children and the prognosis of diseases. With the development of modern research on TCM and the rise of Internet medicine, the demand for standardized, objective and even intelligent TCM tongue diagnosis is increasing rapidly.

[0003] The Chinese medicine tongue diagnosis instrument can form the tongue diagnosis image data required by Chinese medicine practitioners by photographing the tongue image of the child, thereby achieving standardized auxiliary diagnosis of tongue diagnosis. However, the tongue diagnosis instruments on the market are of varying quality. They are large in size and not easy to move. They are only suitable for offline medical institutions for diagnosis and treatment. They are expensive and cannot meet the growing demand for online and offline children's tongue diagnosis photography. Currently, most medical institutions use mobile phones or digital cameras to capture tongue diagnosis images for online and offline diagnosis and treatment. However, due to differences in environmental and lighting conditions, the color of the captured tongue diagnosis images is difficult to show the actual image color that can be seen by the naked eye. Therefore, a method is needed to color correct these tongue diagnosis images.

[0004] Computer vision technology is often used in the field of fast and intelligent image processing, such as image enhancement, image generation, and super-resolution reconstruction. Computer vision simulates the human visual mechanism and has the advantages of fast generation speed and low cost. In recent years, with the application of deep learning in the field of computer vision, especially in the field of medical imaging, breakthrough progress has been made. Generative Adversarial Networks (GAN) has been widely used in the field of image generation and can generate realistic image samples such as faces, natural scenery or animals. Its image generation ability can generate pictures that are indistinguishable from the real thing. GAN can also be used for image editing, such as target extraction, image restoration and enhancement. For example, converting black and white photos to color photos, or converting low-resolution images to high-resolution images.

[0005] However, the existing generative adversarial network cannot be directly used for color correction of children's tongue diagnosis images. There are mainly the following problems:

[0006] Data problem: Children’s tongue diagnosis image data is scarce and highly specific, which limits the training effect and generalization ability of the GAN model.

[0007] Accuracy requirements: The color correction of tongue diagnosis images needs to be highly accurate, but the instability of the GAN model in color generation makes it difficult to meet this requirement.

[0008] Pathological feature preservation: GAN may blur or change pathological features when reconstructing images, affecting diagnostic accuracy.

[0009] Explainability and transparency: GAN models are complex and difficult to understand intuitively, and do not meet the explainability and transparency requirements in the medical field.

[0010] Resource and time cost: GAN training requires a lot of computing resources and time, which poses an obstacle to institutions with limited medical resources.

[0011] Cross-domain adaptability: Different shooting conditions and individual differences lead to large image differences, which makes it difficult for the GAN model to adapt, affecting the consistency and stability of the correction effect. Summary of the invention

[0012] The present invention provides a method and system for automatic color correction of children's traditional Chinese medicine tongue diagnosis images based on deep learning, which can perform color correction on tongue diagnosis images, generate tongue diagnosis image colors as actually seen by the naked eye, and assist traditional Chinese medicine practitioners in identifying children's tongue diagnosis.

[0013] A method for automatic color correction of children's traditional Chinese medicine tongue diagnosis images based on deep learning, comprising the following steps:

[0014] (1) Taking a tongue diagnosis image including a color standard card and collecting a reference image of the color standard card;

[0015] (2) constructing an improved GAN model, wherein the GAN model includes a generator and a discriminator;

[0016] During the training process, the generator is responsible for receiving a random noise vector as input and injecting prior knowledge into the generator, so that the generator generates an image similar to the color-corrected tongue diagnosis image.

[0017] The discriminator is responsible for receiving the image generated by the generator, comparing the color standard card in the image with the color standard card reference image, and outputting a probability value to distinguish the tongue diagnosis image with real color and the tongue diagnosis image after color correction;

[0018] (3) Training a teacher model. In the forward propagation process, the color standard card reference image is input into the teacher model, and Gaussian noise is gradually added to the image until the image is completely converted into random noise. In the backward propagation process, the teacher model learns the noise distribution at each step through the neural network, gradually removes the noise from the random noise, and restores the original image containing the color standard card.

[0019] The GAN model is trained using the image data from step (1). During the training process, the teacher model and the GAN network are parallel networks. The GAN model imitates the performance of the teacher model in the feature space through the weight sharing technology, thereby obtaining feature extraction capabilities similar to those of the teacher model.

[0020] (4) After the GAN model is trained, the tongue diagnosis image to be corrected including the color standard card is input into the generator, and the color-corrected tongue diagnosis image is output.

[0021] The specific structure of the generator is as follows:

[0022] The generator maps the input random noise and prior knowledge into a tensor of shape 4×4×1024 through a reshape operation; next, feature extraction is performed through a combination of 4 deconvolution layers and CBAM blocks; the convolution kernel size of each deconvolution layer is 4×4, the step size is set to 2, and padding=1. These layers map the input random noise to the data space and generate realistic image samples; the image is input into the batch normalization layer after each deconvolution, and the ReLu function is used as the activation function to speed up the model convergence and introduce nonlinear factors. The last layer is activated using the tanh function to generate a 3-channel 64×64 image.

[0023] The process of acquiring prior knowledge is as follows: first, the nnUnet network model is used to segment the color standard cards in the tongue diagnosis image, and the histogram information of each color is accurately extracted from the segmented color standard cards; these histogram information are compared with the standard color card database in detail, and the difference values ​​between all colors are recorded and saved in the form of tensors; finally, these tensors containing color difference information are injected into the generator as prior knowledge.

[0024] The specific structure of the discriminator is as follows:

[0025] The discriminator receives a 3-channel 64×64 image as input, and gradually reduces the size of the feature map through a combination of 4 convolutional layers and CBAM blocks. The convolution kernel size of each convolutional layer is 4×4, the step size is set to 2, and padding=1. At the same time, the number of channels gradually increases to 128, 256, 512, and 1024 respectively. The last layer is a fully connected layer, which vectorizes the input feature map and uses the Sigmoid nonlinear activation function to obtain the probability that the input sample is real data.

[0026] The CBAM block includes a channel attention module and a spatial attention module, and the channel attention and spatial attention are combined in series.

[0027] In step (3), during the training of the GAN model, Wassertein is defined as the loss function, wherein the loss of the generator is calculated based on the difference between the color standard card in the generated tongue diagnosis image and the color standard card reference image, and the loss of the discriminator is calculated based on the accuracy of its judgment of the authenticity of the input tongue diagnosis image.

[0028] During the training process, the generator uses the stochastic gradient descent algorithm as the optimization algorithm, with the initial learning rate set to 0.002, the momentum to 0.75, and the weight decay to 1e-4; the discriminator uses the Adam optimizer, with the initial learning rate to 0.001, and other hyperparameters are the same as the generator; during network training, the batch size is set to 20 and the cycle to 50, and continuous iterations are performed until the GAN model reaches a stable state.

[0029] A deep learning-based automatic color correction system for children's traditional Chinese medicine tongue diagnosis images, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the above-mentioned automatic color correction method for children's traditional Chinese medicine tongue diagnosis images.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] 1. High-precision correction: Through deep learning technology, the accuracy of color correction is improved and human errors are reduced.

[0032] 2. Strong robustness: Adapt to different lighting, equipment and shooting angles to maintain stable correction effect.

[0033] 3. High degree of automation: fully automatic processing, reducing the burden on doctors and improving diagnostic efficiency

[0034] 4. Popularization application: It is convenient for ordinary users to use and promote the popularization of traditional Chinese medicine tongue diagnosis.

[0035] 5. Integration of science and technology: Promote the combination of traditional Chinese medicine and modern technology, and innovate traditional Chinese medicine diagnostic methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flow chart of a method for automatic color correction of children's traditional Chinese medicine tongue diagnosis images based on deep learning in an embodiment of the present invention;

[0037] Figure 2 A structural diagram of the GAN model constructed for the present invention;

[0038] Figure 3 Schematic diagram of a tongue diagnosis image including a color standard card taken in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be pointed out that the embodiments described below are intended to facilitate the understanding of the present invention and do not have any limiting effect on the present invention.

[0040] like Figure 1 As shown, a method for automatic color correction of children's traditional Chinese medicine tongue diagnosis images based on deep learning mainly includes the following steps:

[0041] 1. Data preparation

[0042] The family members of the child need to hold the X-Rite 48 color standard card and stick out their tongue close to the color standard card. Figure 3 As shown, ensure that the tongue diagnosis image taken by a mobile phone or digital camera and the color standard card are in the same environment and lighting conditions. In addition, it is necessary to collect an ideal 48-color standard card image as a reference image. Use the tongue diagnosis image taken by a mobile phone or digital camera as input data, and the color-corrected tongue diagnosis image as the target output data. The ideal 48-color standard card image is used in the training process to compare with the 48-color standard card in the tongue diagnosis image generated by the generator.

[0043] 2. Image color correction based on deep learning

[0044] Build a GAN model. The GAN model mainly consists of two parts: the generator and the discriminator.

[0045] Generator: Responsible for receiving a random noise vector as input and generating a color-corrected tongue diagnosis image. The goal of the generator is to learn the color statistical characteristics of the 48 color standard cards in the color-corrected tongue diagnosis image to generate the ideal color of the 48 color standard cards, and then generate a real tongue diagnosis image to achieve color correction of the original tongue diagnosis image.

[0046] Discriminator: Receives the ideal 48-color standard card image and the tongue diagnosis image generated by the generator as input, and outputs a probability value indicating whether the input image comes from the output of the generator or the ideal 48-color standard card image. The goal of the discriminator is to distinguish between the tongue diagnosis image with real color and the tongue diagnosis image after color correction by the generator.

[0047] like Figure 2 As shown, the specific structure of the generator is:

[0048] The generator maps the input random noise (or latent space vector) into a tensor (4×4×1024) with a shape of (4,4,1024) through a reshape operation; next, the feature extraction capability is enhanced through a combination of 4 deconvolution layers and CBAM blocks. The convolution kernel size of each layer is 4×4, the step size is set to 2, and padding=1. These layers map the input random noise to the data space and generate realistic samples. After each deconvolution, the image is input into the batch normalization layer (BN) layer, and the ReLu function is used as the activation function to speed up the model convergence and introduce nonlinear factors. The last layer uses the tanh function for activation. The parameters of each layer generated after each convolution operation are as follows: 8×8×512, 16×16×256, 32×32×128, and finally generate a three-channel image of the specified size 64×64×3 (width×height×number of channels).

[0049] The specific structure of the discriminator is:

[0050] The discriminator receives a 3-channel 64×64 image as input, and gradually reduces the size of the feature map through a combination of 4 convolutional layers and CBAM blocks. The convolutional layers all use a 4×4 convolution kernel, padding=1, and a step size of 2. At the same time, the number of channels gradually increases to 128, 256, 512, and 1024, respectively, and the image size gradually decreases to 32×32, 16×16, 8×8, and 4×4, respectively. The last layer is a fully connected layer, which vectorizes the input feature map and uses the Sigmoid nonlinear activation function to obtain the probability that the input sample is real data.

[0051] Before the image is input into the GAN model for processing, a crucial step is to perform the detection and segmentation of the color cards in the image. This task can be accomplished by applying the nnUnet network model, which can automatically segment the 48 color standard cards in the image. Next, the histogram information of each color is accurately extracted from the segmented color card. Subsequently, the histogram information is compared in detail with the standard color plate database, and the difference values ​​between all colors are recorded and saved in the form of tensors. Finally, these tensors containing color difference information are injected into the generator model as prior knowledge.

[0052] The hybrid attention (CBAM) includes a channel attention module and a spatial attention module to improve the quality of the generated image after color correction. In the hybrid attention mechanism, channel attention and spatial attention can be combined in series.

[0053] In addition, another model was trained. During the forward propagation of the model, a standard color card was input into the model, and Gaussian noise was gradually added to the image until the image was completely converted into random noise. During the back propagation process, the model learned the noise distribution at each step through the neural network, gradually removed the noise from the random noise, and restored the original color card image. This step is called the teacher model. The intermediate layer features of the teacher model are regarded as a kind of knowledge, and the GAN model is supervised to learn these features by constructing a loss function. This method aims to enable the GAN model to imitate the performance of the teacher model in the feature space, so as to obtain feature extraction capabilities similar to those of the teacher model. In feature-based knowledge distillation, not only the information of the color plate is restored to the greatest extent, making the generated image clearer and more natural, but also the complexity of the GAN model calculation can be reduced.

[0054] 3. Training process

[0055] Wassertein distance is also called bulldozer distance, which involves the minimum cost of converting one probability distribution into another. During the training process, the present invention defines Wassertein as a loss function to guide the learning of the GAN model. The loss of the generator can be calculated based on the difference between the generated image and the ideal 48-color standard card image, while the loss of the discriminator is calculated based on the accuracy of its judgment of the authenticity of the input image.

[0056] During the training process, the generator uses the stochastic gradient descent (SGD) algorithm as the optimization algorithm, with the initial learning rate set to 0.002, the momentum to 0.75, and the weight decay to 1e-4. The discriminator uses the Adam optimizer, with an initial learning rate of 0.001, and other hyperparameters are the same as the generator model. During network training, the batch size is set to 20 and the cycle is 50, and the CBAM_GAN model is continuously iterated until it reaches a stable state.

[0057] 4. Testing and Evaluation

[0058] After the GAN model is trained, it can be applied to the color calibration of the target image. The generator can be used to color correct the tongue diagnosis image taken by a mobile phone or digital camera to restore the true color of the tongue diagnosis image. In addition, the similarity or error between the color of the 48 color standard card in the color-corrected tongue diagnosis image and the ideal 48 color standard card can be calculated.

[0059] The GAN model can be used to perform color correction on tongue diagnosis images and generate tongue diagnosis image colors that are actually seen by the naked eye, assisting Chinese medicine practitioners in diagnosing and identifying children's tongues. This not only reduces the cost of medical institutions purchasing tongue diagnosis instruments, but also improves the convenience of online and offline medical treatment for children. Ultimately, it provides a basic guarantee for the standardization, objectivity and intelligence of Chinese medicine tongue diagnosis-assisted diagnosis in children.

[0060] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for automatic color correction of children's traditional Chinese medicine tongue diagnosis images based on deep learning, characterized in that: The following steps are involved: (1) Taking a tongue diagnosis image including a color standard card and collecting a reference image of the color standard card; (2) constructing an improved GAN model, wherein the GAN model includes a generator and a discriminator; During the training process, the generator is responsible for receiving a random noise vector as input and injecting prior knowledge into the generator, so that the generator generates an image similar to the color-corrected tongue diagnosis image. The discriminator is responsible for receiving the image generated by the generator, comparing the color standard card in the image with the color standard card reference image, and outputting a probability value to distinguish the tongue diagnosis image with real color and the tongue diagnosis image after color correction; (3) Training a teacher model. In the forward propagation process, the color standard card reference image is input into the teacher model, and Gaussian noise is gradually added to the image until the image is completely converted into random noise. In the backward propagation process, the teacher model learns the noise distribution at each step through the neural network, gradually removes the noise from the random noise, and restores the original image containing the color standard card. The GAN model is trained using the image data from step (1). During the training process, the teacher model and the GAN network are parallel networks. The GAN model imitates the performance of the teacher model in the feature space through the weight sharing technology, thereby obtaining feature extraction capabilities similar to those of the teacher model. (4) After the GAN model is trained, the tongue diagnosis image to be corrected including the color standard card is input into the generator, and the color-corrected tongue diagnosis image is output.

2. The method for automatic color correction of children's TCM tongue diagnosis images based on deep learning according to claim 1 is characterized in that: The specific structure of the generator is as follows: The generator maps the input random noise and prior knowledge into a tensor of shape 4×4×1024 through a reshape operation; next, feature extraction is performed through a combination of 4 deconvolution layers and CBAM blocks; the convolution kernel size of each deconvolution layer is 4×4, the step size is set to 2, and padding=1. These layers map the input random noise to the data space and generate realistic image samples; the image is input into the batch normalization layer after each deconvolution, and the ReLu function is used as the activation function to speed up the model convergence and introduce nonlinear factors. The last layer is activated using the tanh function to generate a 3-channel 64×64 image.

3. The method for automatic color correction of children's TCM tongue diagnosis images based on deep learning according to claim 2 is characterized in that: The process of acquiring prior knowledge is as follows: first, the nnUnet network model is used to segment the color standard cards in the tongue diagnosis image, and the histogram information of each color is accurately extracted from the segmented color standard cards; these histogram information are compared with the standard color card database, and the difference values ​​between all colors are recorded and saved in the form of tensors; finally, these tensors containing color difference information are injected into the generator as prior knowledge.

4. The method for automatic color correction of children's TCM tongue diagnosis images based on deep learning according to claim 1 is characterized in that: The specific structure of the discriminator is as follows: The discriminator receives a 3-channel 64×64 image as input, and gradually reduces the size of the feature map through a combination of 4 convolutional layers and CBAM blocks. The convolution kernel size of each convolutional layer is 4×4, the step size is set to 2, and padding=1. At the same time, the number of channels gradually increases to 128, 256, 512, and 1024 respectively. The last layer is a fully connected layer, which vectorizes the input feature map and uses the Sigmoid nonlinear activation function to obtain the probability that the input sample is real data.

5. The method and system for automatic color correction of children's TCM tongue diagnosis images based on deep learning according to claim 2 or 4, characterized in that: The CBAM block includes a channel attention module and a spatial attention module, and the channel attention and spatial attention are combined in series.

6. The method for automatic color correction of children's TCM tongue diagnosis images based on deep learning according to claim 1 is characterized in that: In step (3), during the training of the GAN model, Wassertein is defined as the loss function, wherein the loss of the generator is calculated based on the difference between the color standard card in the generated tongue diagnosis image and the color standard card reference image, and the loss of the discriminator is calculated based on the accuracy of its judgment of the authenticity of the input tongue diagnosis image.

7. The method for automatic color correction of children's TCM tongue diagnosis images based on deep learning according to claim 6 is characterized in that: During the training process, the generator uses the stochastic gradient descent algorithm as the optimization algorithm, with the initial learning rate set to 0.002, the momentum to 0.75, and the weight decay to 1e-4; The discriminator type uses the Adam optimizer, with an initial learning rate of 0.001 and other hyperparameters the same as the generator. When training the network, the batch size is set to 20 and the cycle is set to 50, and the GAN model reaches a stable state through continuous iterations.

8. A deep learning-based automatic color correction system for children's traditional Chinese medicine tongue diagnosis images, characterized in that: It comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the automatic color correction method of children's traditional Chinese medicine tongue diagnosis image according to any one of claims 1-7.

Images