A tongue image classification method based on a deep learning model and a traditional Chinese medicine diagnosis robot

Abstract

The application provides a tongue image classification method and a traditional Chinese medicine diagnosis robot based on a deep learning model, wherein the method comprises detecting whether an instruction word is received, if yes, taking a picture of the tongue of a user according to the instruction word to obtain a tongue image; performing binaryzation on the tongue image by using an improved Unet model to obtain a binary image; matching the binary image with a tongue contour template to determine a tongue area; inputting the tongue area into a ResNet model for classification to obtain a tongue image type. The decoder of the improved Unet model enhances the local detail recovery capability through FASPP, and can better handle the edge blur problem caused by the reflection of the tongue surface. The binary image is matched with the tongue contour template to determine the tongue area. The cascade design composed of the improved Unet model and the ResNet model is adopted, and on the basis of accurately segmenting the tongue area, the accuracy of tongue image classification is improved.

Description

Technical Field

[0001] This invention relates to the field of data classification technology, and in particular to a tongue image classification method based on a deep learning model and a traditional Chinese medicine diagnostic robot. Background Technology

[0002] Current research on the digitalization of tongue imagery in Traditional Chinese Medicine (TCM) faces multiple technical bottlenecks: First, due to restrictions on medical data privacy, biometric data is strictly controlled, resulting in generally small-scale clinical tongue image databases. For example, data from top-tier hospitals in China shows that publicly available standardized tongue image datasets typically contain only 2000-5000 samples, leading to insufficient feature generalization in deep learning models for tongue segmentation (average intersection-over-union ratio <0.75) and classification (F1-score <0.82). Second, significant individual differences exist in tongue image characteristics; the thickness of the tongue coating can differ by 0.3-0.5 mm between people in dry northern climates and those in hot and humid southern regions. Because there are many indicators for evaluating tongue imagery, such as tongue color, surface texture, tongue coating distribution, and crack morphology, coupled with the influence of external factors such as lighting and camera pixel count during actual photography, certain errors occur when using big data models for tongue image analysis, resulting in a gap compared to traditional TCM tongue image analysis.

[0003] Therefore, there is a need for a method that does not rely on massive amounts of real data, but has good generalization performance and high accuracy in tongue image classification. Summary of the Invention

[0004] To overcome the problems existing in related technologies, the purpose of this invention is to provide a tongue image classification method and a traditional Chinese medicine diagnostic robot based on a deep learning model. The method does not rely on massive amounts of real data and has good generalization performance while achieving high classification accuracy of tongue images.

[0005] A tongue image classification method based on a deep learning model includes:

[0006] If a command word is received, the system will take a picture of the user's tongue according to the command word to obtain a tongue image.

[0007] The tongue image is binarized using the improved Unet model to obtain a binarized image;

[0008] The binarized image is matched with a tongue contour template to determine the tongue region;

[0009] The tongue region is input into a ResNet model for classification to obtain the tongue image type.

[0010] In a preferred embodiment of the present invention, the step of binarizing the tongue image using the improved Unet model to obtain a binarized image includes:

[0011] The feature map extracted by the improved feature extraction module of the Unet model is folded using a 2x2 grid;

[0012] A plurality of parallel cavity convolution units are used to capture multi-scale features of the tongue body;

[0013] A fusion unit is designed to weight and fuse the shallow edge information and the multi-scale features to obtain fused features;

[0014] The fused features are input into a prediction network for binarization to obtain a binarized image.

[0015] In the preferred technical solution of the present application, before the tongue image is binarized using the improved Unet model, the method further comprises:

[0016] The continuous down-sampling module of the decoder of the original Unet model is replaced by a FASPP module; the FASPP module is used to enhance the relevance between different local features in the image.

[0017] In the preferred technical solution of the present application, the matching of the binarized image with the tongue contour template to determine the tongue region comprises:

[0018] The binarized image and the tongue contour template are compared using a normalized square difference matching method; if the similarity is greater than a similarity threshold, it is determined that the binarized image and the tongue contour template match;

[0019] A first coordinate point is selected from the binarized image, and the abscissa of the first coordinate point is added to the width of the tongue contour template to obtain the abscissa of a second coordinate point;

[0020] The ordinate of the first coordinate point is added to the height of the tongue contour template to obtain the ordinate of the second coordinate point;

[0021] The tongue region is drawn from the first coordinate point and the second coordinate point; wherein the tongue region is a rectangle.

[0022] In the preferred technical solution of the present application, after the continuous down-sampling module of the decoder of the original Unet model is replaced by the FASPP module, the method further comprises:

[0023] The phase spectrum of the synthesized data is mixed with the amplitude spectrum low-frequency region of the real tongue image using fast Fourier transform to generate a domain adaptive enhanced image set;

[0024] The domain adaptive enhanced image set is mixed with a real image set to obtain a training set;

[0025] training the Unet model to be trained using the training set, gradually reducing the proportion of the domain adaptation enhanced image set in the training set as the training batch increases;

[0026] detecting whether the number of training times of the Unet model to be trained is greater than a training time threshold, and if so, stopping training to obtain an improved Unet model.

[0027] In the preferred technical solution of the present application, training the Unet model to be trained includes a first training phase, a second training phase and a third training phase. The first training phase includes 50 rounds of training. The encoder weights of the Unet model to be trained are frozen in the first training phase, and the FASPP decoder module is trained. The second training phase includes 100 rounds of training. The AdamW optimizer is used for end-to-end fine-tuning, and the learning rate is decayed from 1e-4 to 1e-6 according to the cosine annealing. The third training phase includes 50 rounds of training. High-frequency perturbation data is generated by Fourier alignment to improve the robustness of the Unet model to be trained.

[0028] In the preferred technical solution of the present application, the photographing of the tongue of the user according to the instruction word to obtain a tongue image includes:

[0029] The light-sensitive resistor is used to detect the brightness of the external environment in real time.

[0030] The brightness of the lighting device is adjusted to the target brightness.

[0031] The tongue of the user is photographed by the camera to obtain a tongue image.

[0032] In the preferred technical solution of the present application, before detecting whether the instruction word is received, the method further includes:

[0033] Detecting whether an input word is received, and if so, extracting the current voiceprint feature of the input word using the mel-frequency cepstral coefficient method;

[0034] Matching the current voiceprint feature with a preset voiceprint feature, and if the matching is successful, determining that the input word is a wake-up word and waking up the traditional Chinese diagnosis robot.

[0035] The present application also provides a traditional Chinese diagnosis robot, a tongue image classification method based on a deep learning model, the robot comprising a shell, a Raspberry Pi assembly arranged in the shell, a touch screen arranged on one side of the shell close to the Raspberry Pi assembly, the touch screen being electrically connected to the Raspberry Pi assembly, a camera arranged on the top of the shell, the camera being electrically connected to the Raspberry Pi assembly, and an output end of the Raspberry Pi assembly being electrically connected to a pc end, the pc end being used for image classification of images collected by the camera.

[0036] In the preferable technical scheme of the present application, the traditional Chinese medicine inspection robot further comprises a PCB, a light adjusting device is arranged on the side of the shell, the side of the shell opposite to the touch screen is provided with the PCB, the input end of the PCB is electrically connected with the output end of the Raspberry Pi component; a power module is further arranged in the shell, and the power module is electrically connected with the Raspberry Pi component, the camera, the touch screen and the PCB.

[0037] The present application has the following advantages:

[0038] The tongue image classification method based on a deep learning model provided by the present application comprises detecting whether an instruction word is received, and if so, taking a picture of the user's tongue according to the instruction word to obtain a tongue image. Each instruction word is bound to a specific 16-bit control code, and the instruction data is transmitted to the master control chip through a serial communication protocol. The tongue image includes the user's tongue, lips and other parts of the face. The improved Unet model is used to binarize the tongue image to obtain a binary image. In the U-Net decoding path, the traditional skip connection is improved by folding the FASPP (Feature Pyramid Space Folding) to capture the multi-scale features of the tongue body through parallel hole convolution, thereby improving the accuracy of tongue body crack detection and the segmentation stability in the tooth blocking scene. The decoder of the improved Unet model enhances the local detail recovery capability through FASPP, which can better handle the edge blur problem caused by tongue surface reflection. The binary image is matched with a tongue contour template to determine the tongue region. The tongue region is cropped by template matching the binary image. The tongue region is input into the ResNet model for classification to obtain the tongue image type. ResNet-50 is a deep residual network that solves the gradient disappearance problem of deep networks through cross-layer connection structure, and can extract local details such as crack direction and global features such as tongue fur color distribution. Compared with the traditional model VGG-16, the parameter amount of ResNet-50 is reduced by 80%, and the cascade design composed of the improved Unet model and the ResNet model is adopted in the present application, which improves the accuracy of tongue image classification on the basis of accurate segmentation of the tongue region. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 is a flowchart of the tongue image classification method based on a deep learning model of the present application;

[0040] Figure 2 is a flowchart of the training of the Unet model to be trained of the present application;

[0041] Figure 3 is a structure diagram of the improved Unet model of the present application;

[0042] Figure 4 is a structure diagram of the traditional Chinese medicine inspection robot of the present application;

[0043] Figure 5 is a front view of a traditional Chinese pulse diagnosis robot of the present application;

[0044] Figure 6 is Figure 5 is a sectional view of the shell in the A-A direction in the traditional Chinese pulse diagnosis robot.

[0045] Reference signs: 1, shell; 2, Raspberry Pi assembly; 3, touch screen; 4, camera; 5, PC end; 6, light adjusting device; 7, PCB board; 8, power module; 9, base; 10, rear cover. DETAILED DESCRIPTION

[0046] Preferred embodiments of the present application will be described in more detail with reference to the drawings. Although preferred embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present application is more thorough and complete, and the scope of the present application can be accurately conveyed to those skilled in the art.

[0047] Example 1

[0048] As shown in Figure 1 , the present embodiment provides a tongue image classification method based on a deep learning model, comprising:

[0049] S1: detecting whether an instruction word is received, and if so, taking a picture of the user's tongue according to the instruction word to obtain a tongue image.

[0050] S2: binarizing the tongue image using an improved Unet model to obtain a binary image.

[0051] S3: matching the binary image with a tongue contour template to determine a tongue region.

[0052] S4: inputting the tongue region into a ResNet model for classification to obtain a tongue type.

[0053] The improved Unet model is used to binarize the tongue image to obtain a binary image, comprising:

[0054] S21: folding the feature map extracted by the feature extraction module of the improved Unet model using a 2x2 grid.

[0055] S22: capturing multi-scale features of the tongue using multiple parallel atrous convolution units.

[0056] S23: designing a fusion unit to weight and fuse the shallow edge information and the multi-scale features to obtain fused features.

[0057] S24: inputting the fused features into a prediction network for binarization to obtain a binarized image.

[0058] Before the tongue image is binarized by the improved Unet model, the method further comprises:

[0059] The continuous down-sampling module of the decoder of the original Unet model is replaced by a FASPP module; the FASPP module is used to enhance the relevance between different local features in the image.

[0060] The original Unet model is improved to obtain the improved Unet model. In the decoding path of the UNet model, the traditional skip connection is improved by folding a FASPP (Folding Atrous Spatial Pyramid). The tongue image is input into the improved Unet model to extract a feature map of the tongue image. After the feature map is folded by a 2x2 grid, parallel atrous convolution with an expansion rate of 2, 4 or 6 is used to capture multi-scale features of the tongue, and after unfolding, the multi-scale features are weighted and fused with shallow edge information. The fused features are input into a prediction network for binarization to obtain a binarized image.

[0061] The structure of the improved Unet model of the application is shown in Figure 3 The network input is a tongue image in RGB format with a size of 572x572 pixels. The preprocessed image with a size of 640x480 is expanded by using a mirror padding method to obtain a tongue image in RGB format, and 18 deep convolution layers are used to gradually extract multi-scale features of the tongue image.

[0062] The encoder part of the improved Unet model is composed of the left half of the contraction path Figure 3 , containing 4-level feature extraction modules. Each level contains two 3x3 convolution kernels, and the number of channels is multiplied by 64-128-256-512 step by step. The output end of each convolution layer is connected with a ReLU activation unit and a batch normalization layer in turn, and a 2x2 max pooling layer is used for down-sampling, and the step of the max pooling layer is 2. The size of the feature map is gradually compressed from 572x572 to 28x28, and the process captures high-order semantic features of the tongue macroscopic morphology such as tongue shape contour and crack direction through hierarchical abstraction.

[0063] The decoder part is composed of Figure 3The right half of the expansion path is composed of each stage, which is multiplied by 2x2 deconvolution to double the size of the feature map, and the feature map of the corresponding level of the encoder is fused by the FASPP module. The FASPP module dynamically calibrates the feature weights of the encoder and the decoder through the channel attention mechanism, and suppresses noise features such as light interference. This cross-layer feature fusion mechanism combines the high-resolution details of the shallow layer, such as moss texture and tooth mark edges, with the semantic information of the deep layer, such as tongue area discrimination, effectively solving the problem of detail loss in medical image segmentation. Finally, 1024-dimensional features are mapped to target categories through 1x1 convolution, and the target categories include tongue and background, outputting a 388x388 pixel binary segmentation mask, which is restored to 640x480 resolution through bilinear interpolation, and is spatially aligned with the original tongue image.

[0064] In deep neural networks, as the number of network layers increases, the gradient may gradually decrease during backpropagation, i.e., gradient vanishing, or the gradient may increase sharply, i.e., gradient explosion. Gradient vanishing will cause the shallow part of the network to update slowly or even fail to update, and gradient explosion will cause the weight update to be too large, resulting in unstable network training. The ResNet model solves this problem by introducing a residual learning framework. The ResNet model assumes that if a shallower network can learn certain features, then adding some additional layers to the shallower network will train these additional layers to be identity mappings, so that the network retains its original learning ability and does not cause gradient vanishing or gradient explosion due to the increase in the number of layers.

[0065] In image recognition and other tasks, shallow networks may only learn simple features such as edges and textures, while deep networks can learn deep features such as object shapes and structures. The ResNet model constructed using the residual learning framework can take into account both simple features and deep features.

[0066] This embodiment provides a tongue image classification method based on a deep learning model, which includes detecting whether a command word has been received. If so, the user's tongue is photographed according to the command word to obtain a tongue image. Each command word is bound to a specific hexadecimal control code, and the command data is transmitted to the main control chip via a serial communication protocol. The tongue image includes the user's tongue, lips, and other facial features. An improved U-Net model is used to binarize the tongue image to obtain a binarized image. In the U-Net decoding path, the traditional skip connections are improved by using Folded Spatial Pyramid (FASPP). After the feature map is folded into a grid, parallel dilated convolution is used to capture multi-scale features of the tongue, thereby improving the accuracy of tongue crack detection and improving segmentation stability in scenarios where teeth are occluded. The decoder of the improved U-Net model enhances the local detail recovery capability through FASPP, which can better handle the edge blurring problem caused by tongue surface reflection. The binarized image is matched with a tongue contour template to determine the tongue region. The tongue region is cropped by template matching of the binarized image. The tongue region is input into the ResNet model for classification to obtain the tongue image type. ResNet-50 is a deep residual network that solves the gradient vanishing problem of deep networks through cross-layer connection structure. It can also extract local details of the tongue image, such as the direction of cracks, and global features, such as the color distribution of the tongue coating. Compared with the traditional model VGG-16, ResNet-50 reduces the number of parameters by 80%. This invention adopts a cascaded design composed of an improved Unet model and a ResNet model, which improves the accuracy of tongue image classification based on accurate segmentation of the tongue region.

[0067] Example 2

[0068] like Figure 1 As shown, this embodiment provides a tongue image classification method based on a deep learning model. The differences between this embodiment and Embodiment 1 are described below. The method includes:

[0069] S1: Detect whether a command word has been received. If so, take a picture of the user's tongue according to the command word to obtain a tongue image.

[0070] S2: The tongue image is binarized using the improved Unet model to obtain a binarized image.

[0071] S3: Match the binarized image with the tongue contour template to determine the tongue region.

[0072] S4: Input the tongue region into the ResNet model for classification to obtain the tongue image type.

[0073] The step of matching the binarized image with the tongue contour template to determine the tongue region includes:

[0074] S31: The normalized square difference matching method is used to compare the binary image with the tongue body contour template, and if the similarity is greater than the similarity threshold, it is determined that the binary image and the tongue body contour template are matched.

[0075] S32: The horizontal coordinate of the second coordinate point is obtained by adding the width of the tongue body contour template to the horizontal coordinate of the first coordinate point.

[0076] S33: The vertical coordinate of the second coordinate point is obtained by adding the height of the tongue body contour template to the vertical coordinate of the first coordinate point.

[0077] S34: The tongue body region is drawn from the first coordinate point and the second coordinate point; wherein the tongue body region is a rectangle.

[0078] The tongue image is input into the improved Unet model, and the tongue body region is extracted from the improved Unet model to obtain a binary tongue body mask with clear anatomical structure, i.e. a binary image. The white pixel domain of the binary image, i.e. the region with a pixel value of 255, represents the tongue body region. The binary image is used as a positioning reference, and the normalized square difference matching method is used to compare it with the pre-set tongue body contour template. Through gray space conversion, template size extraction and similarity matrix calculation, the top-left corner coordinate point of the tongue body region in the tongue image, i.e. the first coordinate point, is determined. The horizontal coordinate of the right-bottom corner coordinate point of the tongue body region in the tongue image, i.e. the horizontal coordinate of the second coordinate point, is obtained by adding the width of the tongue body contour template to the horizontal coordinate of the first coordinate point. The vertical coordinate of the second coordinate point is obtained by adding the height of the tongue body contour template to the vertical coordinate of the first coordinate point.

[0079] A rectangular frame of the tongue body region is drawn from the first coordinate point and the second coordinate point. The tongue body region is cropped from the tongue image in the rectangular frame, and the tongue body region is an image in RGB format. After segmentation, the system calls a third-party graph bed service interface, and uploads the tongue image to the cloud through the HTTP protocol. A multi-format POST request is built using the requests library of Python, carrying the developer authentication token and image binary stream data. The interface response analysis module captures the JSON format return result in real time, extracts the permanent access link when successful, and triggers error log recording when failed. According to the actual measurement, the average time consumption of single image upload is 1.2 seconds, the success rate of link generation is 98.7%, and the demand for multi-terminal tongue diagnosis data review is met.

[0080] The photographing of the tongue according to the instruction word includes:

[0081] S12: The photosensitive resistance is used to detect the brightness of the external environment in real time.

[0082] S13: adjust the brightness of the lighting device to the target brightness.

[0083] S14: take a picture of the user's tongue using the camera to obtain a tongue image.

[0084] Before step S12, there is also step S11: receiving an instruction word.

[0085] The light-dependent resistor is used to realize real-time detection and collection of the brightness of the external environment. By adjusting the proportional coefficient and the integral coefficient, the output brightness gradually approaches the target brightness, thereby realizing adaptive adjustment of the light intensity according to the environmental brightness. The side of the shell of the traditional Chinese medicine diagnosis robot is provided with a lamp sheet, and the brightness of the lamp sheet is controlled in real time according to the environmental brightness and the PWM duty cycle. The instruction words include photographing, please help me check the tongue image, adjusting the brightness up, and adjusting the brightness down. Each instruction word is bound to a specific 16-bit control code, such as the photographing instruction code 0x01, the brightness-up code 0x02, and the brightness-down code 0x03. Instruction data is transmitted to the STM32F103 master control chip through a serial communication protocol. The baud rate of the serial communication protocol is 9600 bps, and the data format is 8N1.

[0086] Before detecting whether the instruction word is received, the method further includes:

[0087] S11': detecting whether an input word is received, and if so, extracting the current voiceprint feature of the input word using the mel-frequency cepstral coefficient method.

[0088] S12': matching the current voiceprint feature with a preset voiceprint feature, and if the matching is successful, determining that the input word is a wake-up word and waking up the traditional Chinese medicine diagnosis robot.

[0089] The mel-frequency cepstral coefficient method is used to extract the current voiceprint feature, including the following steps: pre-emphasis, framing, fast Fourier transform, mel filtering, logarithmic operation, extracting the cepstral coefficient, and dimension reduction. The mel-frequency cepstral coefficient method converts the audio signal into a set of mel-cepstral coefficients. The current voiceprint feature and the preset voiceprint feature are matched by combining the dynamic time warping (DTW) method. Dynamic time warping is an algorithm for measuring the similarity between two time series. By adjusting the time, i.e., stretching or compressing on the time axis, the two time series are aligned to find the best matching path, so that the total distance of the matched sequences under the distance metric is minimized. The distance metric can be Euclidean distance or cosine distance, which is not limited here. This embodiment has been tested 500 times, and the wake-up accuracy rate is 98.5%.

[0090] The embodiment photographs the tongue of the user according to the instruction word to obtain a tongue image, including detecting the brightness of the external environment in real time by using a photoresistor, and adjusting the brightness of the lighting device to the target brightness. The tongue of the user is photographed by using a camera to obtain a tongue image. When receiving the brightness adjustment instruction, a PWM wave is output by a TIM3 timer, the frequency of the PWM wave is 1 kHz, the duty cycle is 5%-95%, and a step adjustment mode is used. The collected environmental illumination is closed-loop controlled in the range of 1200±50 lux by using a light intensity sensor, so as to ensure the consistency of the brightness of different batches of data.

[0091] Embodiment 3

[0092] As shown in Figure 1 , the embodiment provides a tongue image classification method based on a deep learning model, and the differences between the embodiment and embodiment 1 are described based on embodiment 1. The method comprises the following steps:

[0093] S1: detecting whether an instruction word is received, if yes, photographing the tongue of the user according to the instruction word to obtain a tongue image.

[0094] S2: binarizing the tongue image by using an improved Unet model to obtain a binary image.

[0095] S3: matching the binary image with a tongue contour template to determine a tongue region.

[0096] S4: inputting the tongue region into a ResNet model for classification to obtain a tongue type.

[0097] Before the step of binarizing the tongue image by using the improved Unet model, the method further comprises the following steps:

[0098] The continuous downsampling module of the decoder of the original Unet model is replaced by a FASPP module; the FASPP module is used to enhance the relevance between different local features in the image.

[0099] As shown in Figure 2 , after the step of replacing the continuous downsampling module of the decoder of the original Unet model by the FASPP module, the method further comprises the following steps:

[0100] S21': mixing the phase spectrum of the synthesized data with the amplitude spectrum low-frequency region of the real tongue image by using fast Fourier transform to generate a domain adaptation enhanced image set.

[0101] S22': mixing the domain adaptation enhanced image set with a real image set to obtain a training set.

[0102] S23': Use the training set to train the Unet model to be trained. As the training batch increases, gradually reduce the proportion of the domain adaptation enhancement image set in the training set.

[0103] S24': Detect whether the number of training iterations of the Unet model to be trained is greater than the training iteration threshold. If so, stop training and obtain the improved Unet model.

[0104] The low-frequency region of the amplitude spectrum refers to the 50% area near the center of the amplitude spectrum. This invention adopts a cross-domain frequency domain alignment preprocessing method, which effectively alleviates the distribution difference between synthetic data and clinical tongue images. Experiments show that this method narrows the fluctuation range of the Dice coefficient of the segmented tongue region on the real data from ±12.3% to ±3.8%, and reduces the missegmentation rate of tongue coating texture to 4.2%.

[0105] This invention addresses the challenges of cross-domain adaptation and multi-scale feature extraction in traditional Chinese medicine tongue image segmentation by proposing a dual optimization strategy. First, Fourier alignment is introduced as a data preprocessing module. A Fast Fourier Transform is used to mix the phase spectrum of the synthetic training data with the low-frequency components of the amplitude spectrum of the real tongue image, generating a domain-adapted enhanced image. The training data consists of 300 clinical tongue images provided by Guangzhou University of Chinese Medicine, including the face, mouth, and adjacent areas. Image enhancement is then performed on multiple tongue images, resulting in a dataset of 900 images after data enhancement.

[0106] The ratio of the domain-adapted augmented image set to the real image set in the training set is 1:1. As the training batch increases, the ratio of the domain-adapted augmented image set is gradually reduced from 0.5 to 0, so that the Unet model to be trained gradually converges from the domain adaptation to the real data distribution. Preferably, contrast-limited adaptive histogram equalization (CLAHE) is performed on the images in the training set to normalize the images in the training set to the [-1,1] interval, in order to adapt to the multi-scale feature extraction requirements of FASPP.

[0107] The training of the Unet model consists of three phases: Phase 1, Phase 2, and Phase 3. Phase 1 comprises 50 training epochs. During this phase, the encoder weights of the Unet model are frozen, and the FASPP decoder module is trained. The learning rate is fixed at 1e-4, focusing on optimizing multi-scale feature fusion capabilities. Phase 2 comprises 100 training epochs. All parameters are unfrozen, and end-to-end fine-tuning is performed using the AdamW optimizer. The AdamW optimizer parameters include β1 and β2, where β1 = 0.9 and β2 = 0.999. The learning rate decays from 1e-4 to 1e-6 using cosine annealing, and the weight decay coefficient is set to 0.01 to suppress overfitting. Phase 3 comprises 50 training epochs. High-frequency perturbation data is generated through Fourier alignment to improve the robustness of the Unet model.

[0108] The loss function of the training adopts a weighted combination of Dice loss and multi-level cross-entropy, and the loss values are calculated respectively from the 1 / 4, 1 / 8 and 1 / 16 scale feature maps output by FASPP, wherein the weight ratio of the 1 / 4, 1 / 8 and 1 / 16 feature maps is 0.7:0.2:0.1. Gradient clipping is adopted throughout the training, and the Lookahead optimizer is used to accelerate convergence, and the threshold of gradient clipping is 3.

[0109] After 200 rounds of training, the Dice coefficient of the model on the validation set reached 0.963, which was 19.2% higher than that of the traditional UNet, and the training time was optimized to 4.5 hours. The adversarial fine-tuning stage improved the segmentation stability of the model under strong light interference by 37.5%, and the IoU standard deviation of the tongue crack area decreased from ±0.15 to ±0.06. The introduction of the random weight average (SWA) strategy further controlled the cross-domain generalization error of the clinical test set within ±3.2%, which was significantly better than the DeepLabV3+ and other comparative models.

[0110] Traditional Chinese tongue diagnosis observes the color, moisture, thickness, cracks and tooth marks of the tongue to diagnose diseases. From the perspective of digital image analysis, these pathological features are essentially a combination of multi-dimensional features such as color distribution, geometric morphology and texture structure. In this study, ResNet-50 was selected as the classification model for tongue image classification tasks, considering the differences in features such as white and yellow fur, and tongue texture such as cracks and ecchymosis. ResNet-50 is a deep residual network that solves the gradient vanishing problem of deep networks through cross-layer connection structure, and can extract both local details such as crack direction and global features such as fur color distribution. Compared with the traditional model VGG-16, the parameter amount of ResNet-50 is reduced by 80%, and based on the pre-trained weights of ImageNet, the transfer learning effect is significant, which is suitable for model training based on small sample data, such as 100 images per category. Experiments show that the accuracy of ResNet-50 in tongue image classification tasks is 4.21% higher than that of VGG-16.

[0111] The training data contains 6 types of tongue images, namely normal tongue, white fur, yellow fur, cracked tongue, tooth mark tongue and ecchymosis tongue, and the number of images of each type of tongue image is 100. After tongue segmentation in the training set, the image size is reduced to 224x224 pixels. To eliminate device color bias, the input image is white balanced in CIE Lab color space and the contrast is enhanced by histogram equalization. Data augmentation includes random horizontal flip (50% probability), ±15° rotation and HSV color disturbance (hue ±0.1, saturation ±0.2). When training, load the ImageNet pre-trained weights, freeze the shallow convolutional layers (the first 3 residual blocks), fine-tune the deep network and the fully connected layer.

[0112] The optimizer selects AdamW, the initial learning rate is set to 3e-5, the weight decay coefficient is 0.01, the learning rate is dynamically adjusted by the cosine annealing strategy, and the verification accuracy reaches 93.1% after 100 rounds of training. The single training time is 4.5 hours.

[0113] The training of the to-be-trained Unet model in this embodiment includes a first training phase, a second training phase and a third training phase. The first training phase includes 50 rounds of training, the encoder weights of the to-be-trained Unet model are frozen in the first training phase, and the FASPP decoder module is trained; the second training phase includes 100 rounds of training, and the AdamW optimizer is used for end-to-end fine-tuning, and the learning rate is annealed from 1e-4 to 1e-6. The third training phase includes 50 rounds of training, and high-frequency disturbance data is generated by Fourier alignment to improve the robustness of the to-be-trained Unet model. Fourier alignment mixes and synthesizes features similar to real data in the frequency domain, so that the model still maintains a stability of Dice coefficient ≥0.93 when the training set is less than 300 cases. In the improved UNet and ResNet-50 cascade design, accurate segmentation can provide standardized input for subsequent classification, the measurement error of L / a value in tongue color quantization is reduced from 8% to 3%, and the accuracy of non-segmentation direct classification is narrowed from 6.7% to 1.2%. Experiments show that the FASPP module has a segmentation error of ≤0.05mm for images with tongue fur thickness in the range of 0.1-1.2mm, which significantly improves the reliability of traditional Chinese medicine eight syndrome differentiation.

[0114] Embodiment 4

[0115] As Figures 4-6As shown, the embodiment provides a traditional Chinese medicine inspection robot, based on the tongue appearance classification method based on the deep learning model in any one of embodiments 1-3, the robot comprises a shell 1, a raspberry pi component 2 is arranged in the shell 1, a touch screen 3 is arranged on one side of the shell 1 close to the raspberry pi component 2, and the touch screen 3 is electrically connected with the raspberry pi component 2; a camera 4 is arranged on the top of the shell 1, the camera 4 is electrically connected with the raspberry pi component 2, and an output end of the raspberry pi component 2 is electrically connected to a pc end 5, and the pc end 5 is used for image classification on images collected by the camera 4.

[0116] The traditional Chinese medicine inspection robot further comprises a PCB board 7, a light adjusting device 6 is arranged on the side of the shell 1, the PCB board 7 is arranged on the side of the shell 1 opposite to the touch screen 3, an input end of the PCB board 7 is electrically connected with an output end of the raspberry pi component 2; and a power module 8 is further arranged in the shell 1, and the power module 8 is electrically connected with the raspberry pi component 2, the camera 4, the touch screen 3 and the PCB board 7.

[0117] The control chip on the PCB board 7 is STM32F103C8T6, the control chip integrates a vc-01 offline voice module, and can respond to voice instructions in real time. A light sensing resistor is installed on the PCB board 7, which can detect the environmental brightness in real time, and the brightness of the light sheets on both sides of the robot is adjusted in real time through a motor driving module. UART serial port protocol communication is adopted between the PCB board 7 and the raspberry pi to realize the state synchronization between the PCB board 7 and the raspberry pi. An offline voice module is deployed on the PCB board 7 to receive input words and instruction words issued by the user.

[0118] The camera 4 has a 1200-wire-pixel CMOS image sensor, and the maximum resolution of the captured image is 4000*3000. A 7-inch touch screen 3 is adopted, which is used to receive the options clicked by the user and display the tongue appearance information. The raspberry pi serves as an information acquisition module, and the camera 4 is called through the opencv library to acquire images. The raspberry pi sends the obtained photo data to the pc end 5, and the pc end 5 performs image segmentation, template matching and image classification on the tongue appearance image, that is, the pc end 5 executes steps S2-S4 in the tongue appearance classification method based on the deep learning model.

[0119] A rectangular LED fill light sheet is installed on the top of the side of the shell 1 of the traditional Chinese medicine inspection robot, the rectangular light sheet is composed of 48 high-color LEDs (CRI≥95), uniform lighting is realized through the rectangular light sheet, the inner diameter of the rectangular light sheet and the outer diameter of the camera 4 keep a gap of 0.5 mm, so as to accurately match the optical path, thereby better adjusting the environmental light.

[0120] Through the STM32-raspberry pi-PC three-level heterogeneous computing architecture, the dynamic allocation of computing power resources is realized (STM32 handles interactive instructions with a time delay of ≤3 ms, raspberry pi image preprocessing delay ≤50 ms, and PC model inference speed of 0.5S). A hybrid transmission protocol (UART+TCP / IP) is used to ensure transmission rate (peak 5Gbps) while reducing error rate (<1e-6). The dynamic light adjustment circuit cooperates with the ambient light sensor to control the color temperature deviation of the tongue image collection within ±50K.

[0121] The 3D printed shell integrates three-level hardware modules with a volume reduced by 58% compared to traditional devices. The three-level power consumption mode prolongs the standby time to 72 hours, with power consumption ≤0.5W. The expansion interface supports pulse diagnosis modules for plug-and-play.

[0122] The bottom of the shell 1 is connected with a base 9, which supports the shell 1. The side opposite to the touch screen 3 of the shell 1 is provided with a rear cover 10, and the PCB board 7 is installed on the rear cover 10.

[0123] The present embodiment provides a traditional Chinese medicine diagnosis robot based on the tongue image classification method based on the deep learning model described in any one of embodiments 1-3. The robot includes a shell 1, a raspberry pi assembly 2 disposed in the shell 1, a touch screen 3 disposed on one side of the shell 1 close to the raspberry pi assembly 2, and the touch screen 3 is electrically connected to the raspberry pi assembly 2. A camera 4 is disposed on the top of the shell 1, and the camera 4 is electrically connected to the raspberry pi assembly 2. The output end of the raspberry pi assembly 2 is electrically connected to a PC end 5, and the PC end 5 is used for image classification of the image collected by the camera 4. A light sensing resistor is installed on the PCB board 7 to detect the ambient brightness in real time, and the brightness of the light sheets on both sides of the robot is adjusted in real time through a motor drive module. UART serial port protocol communication is used between the PCB board 7 and the raspberry pi to realize the state synchronization between the PCB board 7 and the raspberry pi. An offline voice module is deployed on the PCB board 7 to receive input words and instruction words issued by the user.

[0124] It should be noted that in this document, the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusions, so that a process, device, article or method including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or inherent to such a process, device, article or method. Without more limitations, the element defined by the statement "including a…" does not exclude the presence of another identical element in the process, device, article or method including the element.

[0125] The above merely describes the preferred embodiments of the present application, and is not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation, or direct or indirect application in other related technical fields, which is made according to the content of the present application specification and drawings, is also included in the patent protection scope of the present application.

Claims

1. A tongue image classification method based on a deep learning model, characterized in that, include: If a command word is received, the system will take a picture of the user's tongue according to the command word to obtain a tongue image. The tongue image is binarized using the improved Unet model to obtain a binarized image; The binarized image is matched with a tongue contour template to determine the tongue region; The tongue region is input into a ResNet model for classification to obtain the tongue image type; The step of binarizing the tongue image using the improved Unet model to obtain a binarized image includes: The feature maps extracted by the feature extraction module of the improved Unet model are folded using a 2×2 grid. Multiple parallel dilated convolutional units are used to capture multi-scale features of the tongue. The design incorporates a fusion unit to weightedly fuse shallow edge information with the multi-scale features to obtain fused features. The fused features are input into the prediction network for binarization to obtain a binarized image; Before binarizing the tongue image using the improved Unet model, the process also includes: The continuous downsampling module of the decoder in the original Unet model is replaced with the FASPP module; the FASPP module is used to enhance the correlation between different local features in the image. After changing the continuous downsampling module of the decoder of the original Unet model to the FASPP module, the method also includes: Fast Fourier Transform is used to mix the phase spectrum of the synthetic data with the low-frequency region of the amplitude spectrum of the real tongue image to generate a domain-adaptive enhanced image set; The domain-adapted enhanced image set is mixed with the real image set to obtain the training set; The training set is used to train the Unet model to be trained. As the training batch increases, the proportion of the domain adaptation enhancement image set in the training set is gradually reduced. If the number of training iterations of the Unet model to be trained exceeds a threshold, training is stopped, and the improved Unet model is obtained. The training of the Unet model to be trained includes three training phases: the first training phase, the second training phase, and the third training phase. The first training phase consists of 50 training rounds, during which the encoder weights of the Unet model to be trained are frozen, and the FASPP decoder module is trained. The second training phase consists of 100 training rounds, during which the AdamW optimizer is used for end-to-end fine-tuning, and the learning rate is decayed from 1e-4 to 1e-6 by cosine annealing. The third training phase consists of 50 training rounds, during which high-frequency perturbation data is generated through Fourier alignment to improve the robustness of the Unet model to be trained.

2. The tongue image classification method based on a deep learning model according to claim 1, characterized in that, The step of matching the binarized image with the tongue contour template to determine the tongue region includes: The binarized image is compared with the tongue contour template using the normalized squared difference matching method. If the similarity is greater than the similarity threshold, the binarized image and the tongue contour template are determined to match. Select a first coordinate point from the binarized image, and add the width of the tongue contour template to the x-coordinate of the first coordinate point to obtain the x-coordinate of the second coordinate point; Add the height of the tongue contour template to the ordinate of the first coordinate point to obtain the ordinate of the second coordinate point; The tongue region is drawn using the first coordinate point and the second coordinate point; wherein the tongue region is rectangular.

3. The tongue image classification method based on a deep learning model according to claim 1, characterized in that, The step of taking a picture of the user's tongue according to the instruction word to obtain a tongue image includes: A photoresistor is used to detect the brightness of the external environment in real time; Adjust the brightness of the lighting equipment to the target brightness; A camera is used to take a picture of the user's tongue to obtain a tongue image.

4. The tongue image classification method based on a deep learning model according to claim 1, characterized in that, Before detecting whether an instruction word has been received, the process also includes: If an input word is received, the current voiceprint features of the input word are extracted using the Mel frequency cepstral coefficient method. The current voiceprint features are matched with preset voiceprint features. If the match is successful, the input word is determined to be the wake-up word, and the TCM diagnostic robot is activated.

5. A traditional Chinese medicine diagnostic robot, characterized in that, Based on the tongue image classification method based on a deep learning model according to any one of claims 1-4, the robot includes a shell, a Raspberry Pi component is disposed inside the shell, a touch screen is disposed on the side of the shell near the Raspberry Pi component, and the touch screen is electrically connected to the Raspberry Pi component; a camera is disposed on the top of the shell, the camera is electrically connected to the Raspberry Pi component, and the output end of the Raspberry Pi component is electrically connected to a PC, the PC being used to perform image classification on the images captured by the camera.

6. The TCM diagnostic robot according to claim 5, characterized in that, It also includes a PCB board, a light adjustment device is provided on the side of the housing, the PCB board is provided on the side of the housing opposite to the touch screen, and the input end of the PCB board is electrically connected to the output end of the Raspberry Pi component; a power module is also provided inside the housing, and the power module is electrically connected to the Raspberry Pi component, camera, touch screen and PCB board.

Images