Tongue image recognition system and method based on depth separable convolution

Abstract

The present disclosure relates to the technical field of image processing, and proposes a tongue image recognition system and method based on deep separable convolution. By constructing a tongue image recognition network, deep separable convolution and residual network are fused. After inputting the tongue image to be classified, repeated residual inversion operations are performed to expand the channel number and capture more detailed tongue feature representation information in the image after expanding the feature map receptive field, which is used for tongue recognition and classification. In each residual inversion operation, deep separable convolution operation is performed, and the correlation information of the feature map channel and space is extracted based on the channel attention and spatial attention operation. The objectification of traditional Chinese medicine tongue diagnosis is realized, and in the case of small amount of tongue image data, the deep learning network model is used to quickly and accurately distinguish the tongue image, thereby improving the accuracy and speed of tongue image recognition and classification.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, specifically to a tongue recognition system and method based on depthwise separable convolution. Background Technology

[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.

[0003] Tongue diagnosis, a distinctive diagnostic method in Traditional Chinese Medicine (TCM), directly reflects the state of Qi and blood, organ function, and the strength of pathogenic factors in the body. Furthermore, because the tongue lacks an outer skin, changes in its appearance often precede clinical symptoms. However, due to its strong subjective dependence, quantitative and objective research in tongue diagnosis is still in the exploratory stage, resulting in its value in the diagnosis and treatment of diseases such as hypertension not being fully recognized or explored. Exploring the deeper patterns within tongue image information and establishing an assessment model based on tongue image could maximize the advantages of tongue diagnosis in intelligent medical treatment, achieving the inheritance, transformation, and innovation of TCM tongue diagnosis. Existing tongue image recognition methods are slow and have a large number of parameters, making them unsuitable for application on mobile devices.

[0004] The inventors discovered during their research that the amount of medical tongue image data is too small to support the training of deep neural networks. Furthermore, for general convolutional neural networks, overfitting often becomes severe as the number of network layers increases, making it impossible to obtain a deep neural network that can accurately classify tongue images. Therefore, how to achieve fast, automatic, and accurate classification of medical tongue images using neural network models with limited data is a pressing problem that needs to be solved. Summary of the Invention

[0005] To address the aforementioned problems, this disclosure proposes a tongue image recognition system and method based on depthwise separable convolution. This aims to objectify traditional Chinese medicine tongue diagnosis and, even with limited tongue image data, achieve rapid and accurate tongue image identification using a deep learning network model, thereby improving the accuracy and speed of tongue image recognition and classification. This disclosure also presents a lightweight recognition method with fewer model parameters than existing models and faster operation.

[0006] To achieve the above objectives, the present disclosure adopts the following technical solution:

[0007] One or more embodiments provide a tongue image recognition method based on depthwise separable convolution, including the following steps:

[0008] Obtain the tongue image to be classified and extract features to obtain the input feature map;

[0009] The inverse residual operation is used to sequentially perform channel expansion and depthwise separable convolution operations on the input feature map, and capture the correlation information of the feature map channels and space based on channel attention mechanism and spatial attention mechanism;

[0010] Multiple residual inversion operations are performed in a cascade manner. The output of the previous residual inversion operation is connected to the next cascade operation, thereby increasing the number of channels and expanding the receptive field of the feature map to capture more detailed tongue image feature representation information in the image.

[0011] Based on the tongue image feature representation information, the feature processing channel number is converted and then classified and output.

[0012] One or more embodiments provide a tongue recognition system based on depthwise separable convolution, including:

[0013] Input module: Configured to acquire tongue images to be classified and perform feature extraction to obtain input feature maps;

[0014] Inverse residual module: It is configured to perform inverse residual operation on the input feature map, sequentially perform channel expansion operation and depthwise separable convolution operation, and capture the correlation information of the feature map channels and space based on channel attention mechanism and spatial attention mechanism;

[0015] Multiple residual inversion operations are performed in a cascade manner. The output of the previous residual inversion operation is connected to the next cascade operation, thereby increasing the number of channels and expanding the receptive field of the feature map to capture more detailed tongue image feature representation information in the image.

[0016] Classification output module: It is configured to convert the number of feature processing channels based on the tongue image feature representation information and then perform classification output.

[0017] An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the steps described in the above method.

[0018] A computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the steps described in the above method.

[0019] Compared with the prior art, the beneficial effects of this disclosure are as follows:

[0020] In this disclosure, depthwise separable convolution and residual networks are used to achieve tongue image discrimination, avoiding the problems of overfitting and large number of parameters in convolutional neural network models.

[0021] This disclosure also innovatively proposes an attention mechanism that includes channel attention and spatial attention and integrates it with depthwise separable convolution operations, which ultimately improves the accuracy and speed of model inference. It can be deployed on mobile devices, eliminating the need for manual tongue image recognition, realizing the objectification of tongue image judgment in traditional Chinese medicine, and improving the accuracy of tongue image information recognition.

[0022] The advantages of this disclosure, as well as its additional advantages, will be described in detail in the following specific embodiments. Attached Figure Description

[0023] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute a limitation thereof.

[0024] Figure 1 This is a flowchart of the tongue image recognition network training in Embodiment 1 of this disclosure;

[0025] Figure 2 This is a schematic diagram of the tongue image recognition network of Embodiment 1 of this disclosure;

[0026] Figure 3 This is a schematic diagram of the CASA attention submodule of Embodiment 1 of this disclosure;

[0027] Figure 4 This is an algorithm flowchart of the CASA attention submodule of Embodiment 1 of this disclosure;

[0028] Figure 5 This is a flowchart of the inverse residual module with residual connection in Embodiment 1 of this disclosure;

[0029] Figure 6 This is a flowchart of the inverse residual module without residual connection in Embodiment 1 of this disclosure;

[0030] Figure 7 This is a comparison chart of the parameter quantities of the tongue image recognition network provided in Embodiment 1 of this disclosure and existing models;

[0031] Figure 8 This is a comparison chart of the tongue image recognition network provided in Embodiment 1 of this disclosure with existing models FLOPs (floating-point operands). Detailed Implementation

[0032] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0033] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of this disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0034] It should be noted that the terminology used herein is for descriptive purposes only and is not intended to limit the exemplary embodiments according to this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof. It should be noted that, without conflict, the various embodiments and features within those embodiments can be combined with each other. The embodiments will now be described in detail with reference to the accompanying drawings.

[0035] Example 1

[0036] In one or more of the technical solutions disclosed in the implementation methods, such as Figures 1 to 8 As shown, the tongue recognition method based on depthwise separable convolution includes the following steps:

[0037] Step 1: Obtain the tongue image to be classified and extract features to obtain the input feature map;

[0038] Step 2: Using inverted residual operation, the input feature map is sequentially subjected to channel expansion operation and depthwise separable convolution operation, and the correlation information of the feature map channels and space is captured based on channel attention mechanism and spatial attention mechanism;

[0039] Step 3: Perform multiple inverted residual operations in a cascade manner. The output of the previous inverted residual operation is connected to the next cascaded operation, thereby increasing the number of channels and expanding the receptive field of the feature map to capture more detailed tongue image feature representation information in the image.

[0040] Step 4: Based on the tongue image feature representation information, convert the number of feature processing channels and then perform classification output.

[0041] In this embodiment, a depthwise separable convolutional network and a residual network are fused together. The input tongue image to be classified is repeatedly subjected to the inverse residual operation. In each inverse residual operation, a depthwise separable convolutional operation is performed, and the correlation information between the channels and space of the feature map is extracted based on channel attention and spatial attention operations. Weights are then applied to the feature map after the depthwise separable convolutional operation. The number of channels is increased through multiple inverse residual operations, and the receptive field of the feature map is expanded to capture more detailed tongue image feature representation information in the image for tongue image recognition and classification. By fusing depthwise separable convolutional network and residual network to achieve tongue image discrimination, the overfitting problem of the model and the large number of parameters of the convolutional neural network model are avoided.

[0042] This embodiment also innovatively proposes a CASA attention mechanism that includes channel attention and spatial attention, and integrates it with depthwise separable convolution operations. This ultimately improves the accuracy and speed of model inference, and can be deployed on mobile devices. It eliminates the need for manual tongue image recognition, realizes the objectification of TCM tongue image judgment, and improves the accuracy of tongue image information recognition.

[0043] Step 1 is the feature extraction step. For the tongue image to be classified, convolution operation, batch normalization operation and function activation output operation can be performed in sequence to obtain the input feature map and perform shallow feature extraction, which is conducive to the deeper feature extraction in the subsequent steps.

[0044] In step 4, the number of feature processing channels is converted based on the tongue image feature representation information, and then the classification output is performed, including the following steps:

[0045] The output after the inverted residual operation is subjected to feature processing to convert the number of channels, resulting in a channel-converted feature map.

[0046] Perform global average pooling on the feature map after channel conversion;

[0047] A linear transformation is performed on the feature map after global average pooling to obtain the classification output.

[0048] The above-mentioned tongue image recognition method is implemented by constructing an image recognition network. The network structure of the tongue image recognition network is explained below.

[0049] In some embodiments, such as Figure 2 As shown, the tongue recognition network adopts a depthwise separable convolutional network, which integrates a residual network and a CASA attention submodule.

[0050] In some feasible implementations, a depthwise separable convolutional network, from input to output, includes an input module, multiple inverted residual modules, a convolutional normalized activation module, a global pooling layer, and a classification output module connected in sequence.

[0051] The input module is used to perform convolution and batch normalization on the input image, and after activation, the input feature map is obtained.

[0052] Multiple inverted residual modules are connected through residual connections, and the inverted residual operation is performed repeatedly to increase the number of channels and expand the receptive field of the feature map to capture more detailed tongue image feature representation information in the image.

[0053] Among them, the tongue image feature representation information is used to characterize the tongue color, tongue shape, sublingual veins, coating texture, and coating color. Tongue shape mainly includes features such as tongue weakness, deviation, and shortening; coating texture mainly includes features such as the thickness, moisture, dryness, greasiness, and peeling of the tongue coating.

[0054] The inverted residual module in this embodiment can not only identify basic features such as tongue color, tongue shape, and tongue coating color, but also extract more comprehensive and detailed features about the tongue image.

[0055] Each inverse residual module is configured to perform channel expansion and depthwise separable convolution operations sequentially on the input feature map. Based on channel attention and spatial attention mechanisms, it captures the correlation information between the channels and space of the feature map. After channel compression, a second feature map is obtained, which is then connected to the next inverse residual module or convolution normalization activation module.

[0056] The convolutional normalization activation module is used to convert the number of feature processing channels in the output of the inverse residual module.

[0057] The global pooling layer is used to perform global average pooling on the feature map output by the convolutional normalized activation module.

[0058] The classification output module performs a linear transformation on the feature map after global average pooling to obtain the classification output.

[0059] To address the issue of insufficient training data mentioned in the background section, training the tongue image recognition network also includes a step of constructing an expanded data sample set, as follows:

[0060] Constructing an expanded data sample set includes the following steps:

[0061] Step S11: Obtain tongue image video and extract tongue image images to generate a tongue image image database;

[0062] Specifically, hospitals can use tongue image acquisition equipment to collect videos of clinical patients' tongue images, and then extract tongue images from the videos to generate a tongue image database.

[0063] Step S12: Perform manual dataset annotation on the images in the tongue image database;

[0064] Step S13: After annotation, perform preprocessing and data augmentation on the images to establish an extended data sample set;

[0065] Optionally, data augmentation may include data augmentation operations such as random horizontal flipping, random vertical flipping, and random cropping and stretching of the central region;

[0066] This embodiment uses data augmentation processing to increase the sample size of the data, thereby overcoming the technical defect that the training data volume cannot meet the training requirements due to the severe shortage of tongue image images.

[0067] The tongue image recognition network was trained using an expanded sample dataset to obtain the trained tongue image recognition network.

[0068] In the tongue image recognition stage, the tongue image to be classified is input into the tongue image discrimination model to obtain the classification result.

[0069] In some embodiments, the input module of the tongue recognition network includes a first convolutional layer, a first batch normalization layer, and a first activation function layer connected in sequence;

[0070] The input channel count of the first convolutional layer can be set to 3, the output channel count can be set to 16, and the stride can be set to 1.

[0071] Optionally, the first activation function layer can use the HardSwish activation function.

[0072] The HardSwish activation function is an artificial neural network activation function, an improved version of the Swish activation function.

[0073] The mathematical formula for the HardSwish function is:

[0074] f(x) = x * ReLU6(x+3) / 6,

[0075] ReLU6 is a non-linear activation function, short for "Rectified Linear Unit 6".

[0076] The mathematical formula for ReLU6 is: f(x)=min(max(0,x),6), which means that when x is greater than 0, x is output; when x is less than or equal to 0, 0 is output; and when x is greater than 6, 6 is output.

[0077] The HardSwish activation function outputs values ​​between 0 and 1. Unlike the standard ReLU (Rectified Linear Unit, which outputs x when x is greater than 0 and 0 otherwise) activation function, it avoids the vanishing and exploding gradient problems. Furthermore, HardSwish is more stable and faster than the Swish activation function, making it more efficient in deep learning tasks. The advantage of using HardSwish is that it reduces computational complexity while maintaining model performance, making it highly valuable in practical applications. Additionally, a batch normalization layer is used between the convolutional and activation layers. By using a batch normalization layer, the normal distribution of the data can be normalized during each forward propagation.

[0078] In some embodiments, the inverse residual module includes a first convolutional normalized activation submodule, a second convolutional normalized activation submodule, a CASA attention submodule, a third convolutional normalized activation submodule, and a residual connection module connected in sequence.

[0079] In this embodiment, among all the inverted residual modules, the first two convolutional normalized activation sub-modules, namely the first convolutional normalized activation sub-module and the second convolutional normalized activation sub-module, can both use ReLU6 as the activation function, and the third convolutional normalized activation sub-module can use the HardSwish activation function.

[0080] In this embodiment, the inverse residual module includes three convolutional batch normalization sub-modules. Specifically, each convolutional batch normalization sub-module contains a convolutional layer, a batch normalization layer, and a HardSwish activation function. The structure of the convolutional batch normalization sub-module can be the same as that of the convolutional normalization activation module.

[0081] The convolutional normalization activation module includes a second convolutional layer, a second batch normalization layer, and a second activation function layer connected in sequence; the second activation function layer can use the HardSwish activation function.

[0082] The global pooling layer contains a global average pooling operation;

[0083] The classification output module consists of a third convolutional layer, a third activation function layer, a random dropout layer, and a linear output layer. The third activation function layer can use the HardSwish activation function.

[0084] In this embodiment, referencing residual connections in the inverse residual module can effectively avoid the gradient vanishing problem. The inverse residual modules with and without residual connections are as follows: Figure 5 and Figure 6 As shown, if the number of input channels is the same as the number of output channels, then... Figure 5 The structure shown is (i.e., connected by residuals); the opposite is... Figure 6 ;

[0085] The CASA attention submodule proposed in this embodiment enables the network to pay more attention to important details. The CASA attention submodule learns to automatically acquire important features and spatially effective features in each feature channel, selectively enhancing useful features in the channel and space while suppressing useless features.

[0086] Optionally, the structure of the CASA attention submodule can be as follows: Figure 3 As shown, it includes a channel attention component and a spatial attention component connected in sequence.

[0087] The channel attention component is used to adaptively weight each input channel;

[0088] The spatial attention component is used to adaptively weight each pixel.

[0089] In this embodiment, by combining channel attention and spatial attention, the CASA attention submodule can better capture important information in both space and channel.

[0090] Optionally, the channel attention part specifically adopts the Coordinate Attention (CA) attention mechanism, introducing a CA attention submodule;

[0091] Furthermore, the channel attention component is reduced and the channel number rate is fixed at 16.

[0092] The improvement of the CA attention submodule in this embodiment is to fix the reduction channel rate at 16, which results in better processing of tongue image data.

[0093] The CA attention submodule embeds positional information into channel attention, overcoming the shortcoming of the classic SE module ignoring positional information and further improving the performance of this model. While SE channel attention transforms feature tensors into single feature vectors through two-dimensional global pooling, CA attention decomposes channel attention into two one-dimensional feature encoding processes, thus more effectively capturing positional information and channel relationships to enhance the network's feature representation.

[0094] The CA attention submodule includes two global average pooling layers, one convolutional layer, one batch normalization operation, and one HardSwish activation function layer.

[0095] The CA attention submodule is used to pool the input feature matrix and then fuse the resulting high and wide axis feature maps to generate the final high and wide axis attention mechanism, thus obtaining the channel attention feature matrix. Specifically, the processing includes the following steps:

[0096] Step A1: The input data x is processed by two global average pooling operations to obtain two matrices: a high feature map x_h and a wide-axis feature map x_w. The matrices are then concatenated.

[0097] like Figure 4 As shown in the figure, horizontal average pooling is performed to obtain the high feature map x_h, and vertical average pooling is performed to obtain the wide feature map x_w.

[0098] Step A2: Perform convolution on the concatenated matrix, normalize it according to the normal distribution, and then process it through an activation function;

[0099] Specifically, the concatenated matrix undergoes a 1×1 convolution (conv1) and batch normalization, and is then activated by the HardSwish activation function.

[0100] Step A3: Perform convolution operations on the feature map after activation function processing to calculate the attention scores for the height and width axes respectively. Based on the scores, obtain the channel attention feature matrix, and adjust the weights based on the channel attention scores.

[0101] Specifically, the output matrices out_h and out_w are calculated using two 1×1 convolutions (conv2 and conv3), and the product of the input matrix and the original input is output.

[0102] One feasible implementation uses three convolutional layers, conv1, conv2, and conv3, to fuse the feature maps of the high and wide axes after average pooling, respectively, to generate the final attention mechanisms for the high and wide axes. The first convolutional layer, conv1, has an output channel count of 1 / 16 of the input channels, a kernel size of 1, a stride of 1, and no padding. A batch normalization layer and a HardSwish activation function follow the first convolutional layer, conv1. The second convolutional layer, conv2, and the third convolutional layer, conv3, both have an input channel count of 16 / 16, and their output channel count is set to the original input channel count for ease of connection to the spatial attention submodule. They also have a kernel size of 1, a stride of 1, and again, no padding.

[0103] Spatial attention complements channel attention. Unlike channel attention, spatial attention focuses more on where the most information-rich parts are.

[0104] Optionally, the spatial attention component consists of a multilayer perceptron, which mainly includes a compression component and a spatial component.

[0105] The compression part uses a channel pooling operation, while the spatial part uses a convolutional layer and a batch normalization layer.

[0106] In the spatial attention part, the input data, i.e., the channel attention feature matrix, is compressed, and the correlation between spatial features is established through convolution operations. Then, feature extraction and function activation are performed sequentially to obtain the spatial mask of each pixel, and the product of the spatial mask and the original input is returned; specifically, such as Figure 4 As shown, the spatial attention process, after adjusting the weights through channel attention, includes the following:

[0107] Step B1: Compress the input channel attention feature matrix to obtain two-channel output, and extract the correlation between spatial features;

[0108] The input data first goes through a compression process. Channel pooling is achieved by combining the maximum and mean values ​​along the channel dimension. This mainly involves concatenating the maximum and mean values ​​of each channel to obtain an output containing two channels. The global average pooling and max pooling operations obtain spatial attention features, and the correlation between spatial features is established through two convolutions.

[0109] Step B2: Extract features from the two channel outputs;

[0110] The output from the compressed section enters the spatial section. Feature extraction is performed on the output of the two channels of the compressed section. In this embodiment, a convolution operation with a kernel size of 7×7, a stride of 1, and padding of 3, and a batch normalization operation are used. By using a convolution operation with a kernel size of 7×7, the number of parameters and computational cost are greatly reduced, which is beneficial for establishing high-dimensional spatial feature correlations.

[0111] Step B3: Activate the extracted two-channel features using an activation function to obtain the spatial attention feature matrix;

[0112] Activation is performed using the sigmoid function to obtain the spatial mask for each pixel, and then the product of the spatial mask and the original input is calculated.

[0113] In this embodiment, when the feature matrix is ​​fed into the CASA attention submodule, it first passes through the channel attention part to obtain a channel attention feature matrix, and then this feature matrix is ​​fed into the spatial attention part as input. After this processing, the network model will be able to capture more important information in the channel and space more easily, improve the connection between various features in the channel and space, and thus be more conducive to extracting effective features of the target and improving the recognition accuracy.

[0114] The random dropout layer of the classification output module randomly disconnects the input neurons each time the parameters are updated during training, and deletes part of the layer with a certain probability.

[0115] Optionally, the dropout rate in the random dropout layer can be set to 20%. This prevents overfitting and improves the model's generalization ability. The final linear output layer is used to output the classification result.

[0116] The data processing procedure for the depthwise separable convolutional network described above is as follows:

[0117] In a depthwise separable convolutional network, an input image of 224×224×3 passes through the input module and then enters the inverse residual module. Batch Normalization (BN) is used after each convolutional layer in the model. The main idea of ​​BN is to normalize the data according to the normal distribution during each forward propagation.

[0118] The main principle of the inverse residual module is as follows: First, a convolutional normalization activation submodule is used to perform a 1×1 convolution on the number of channels to expand the number of channels to a specified number. Next, it enters the next convolutional normalization activation submodule to perform a depthwise separable convolution operation, with a kernel size of 3×3 or 5×5. Afterward, the feature map processed by the previous convolutional normalization activation module enters a CASA attention submodule to better capture important channel and spatial information. Its input and output channel numbers are the same as the output channel number of the previous convolutional normalization activation module. Then, a convolutional normalization activation module with a 1×1 kernel size is used to compress the number of channels in the feature map processed by the CASA attention submodule to a specified number. If the number of channels in the input of the inverse residual module is the same as the number of channels in the output, and the specified stride in the inverse residual block is 1, then a residual connection is used to link the beginning and end of the inverse residual module, effectively mitigating the gradient vanishing phenomenon.

[0119] In this network, the inverse residual module will be repeated 15 times, with the number of channels gradually increasing. This allows for the capture of more detailed and important information as the receptive field of the feature map expands. Specifically:

[0120] The first inverse residual structure has 16 input channels, 16 expanded channels, and 16 output channels. The second convolutional normalization activation module in this inverse residual module has a 3×3 kernel size and a stride of 1.

[0121] The second inverted residual structure has 16 input channels, 64 expanded channels, and 24 output channels. The kernel size of the second convolutional normalized activation module in this inverted residual block is 3×3, and the stride is 2.

[0122] The third inverted residual structure has 24 input channels, 72 expanded channels, and 24 output channels. The kernel size of the second convolutional normalized activation module in this inverted residual block is 3×3, and the stride is 1.

[0123] The fourth inverted residual structure has 24 input channels, 72 expanded channels, and 40 output channels. The kernel size of the second convolutional normalized activation module in this inverted residual block is 5×5, and the stride is 2.

[0124] The fifth and sixth inverted residual structures have 40 input channels, 120 expanded channels, and 40 output channels. The kernel size of the second convolutional normalized activation module in this inverted residual block is 5×5, and the stride is 1.

[0125] The seventh inverted residual structure has 40 input channels, 240 expanded channels, and 80 output channels. The kernel size of the second convolutional normalized activation module in this inverted residual block is 3×3, and the stride is 2.

[0126] The eighth inverted residual structure has 80 input channels, 200 expanded channels, and 80 output channels. The kernel size of the second convolutional normalization activation module in this inverted residual block is 3×3, and the stride is 1.

[0127] The ninth and tenth inverted residual structures have 80 input channels, 184 expanded channels, and 80 output channels. The kernel size of the second convolutional normalized activation module in this inverted residual block is 3×3, and the stride is 1.

[0128] The eleventh inverted residual structure has 80 input channels, 480 expanded channels, and 112 output channels. The kernel size of the second convolutional normalized activation module in this inverted residual block is 3×3, and the stride is 1.

[0129] The number of input channels in the twelfth inverted residual structure is 112, the number of channels after expansion is 672, and the number of output channels is 112. The kernel size of the second convolutional normalization activation module in this inverted residual block is 3×3, and the stride is 1.

[0130] The thirteenth inverted residual structure has 112 input channels, 672 expanded channels, and 160 output channels. The kernel size of the second convolutional normalized activation module in this inverted residual block is 5×5, and the stride is 2.

[0131] The fourteenth and fifteenth inverted residual structures have 160 input channels, 960 expanded channels, and 160 output channels. The kernel size of the second convolutional normalized activation module in this inverted residual block is 5×5, and the stride is 1.

[0132] Next, a convolutional normalization activation module is used, which converts the number of output channels to 960 using a 1×1 convolution; then, a global average pooling layer is entered to perform average pooling operations (convolutional kernel size is 7×7, stride is 1).

[0133] Finally, the output module works as follows: First, the number of channels is expanded from 960 to 1280 through convolution (the kernel size is 1×1 and the stride is 1). Then, it goes through a Dropout layer, which randomly disconnects input neurons and deletes some layers with a certain probability each time the parameters are updated during training. Its purpose is to prevent overfitting and improve the generalization ability of the model. Finally, a linear layer is used to convert the output into a classification output.

[0134] Specifically, a concrete implementation of the tongue image recognition network training process can be as follows:

[0135] S1. A tongue image acquisition device can be used to collect videos of clinical patients' tongue images, and then images of the patient's tongue can be extracted from the videos. Due to privacy concerns, only the tongue portion is captured during the extraction process.

[0136] S2, Image annotation.

[0137] The annotation of tongue appearance is carried out by two professors specializing in traditional Chinese medicine diagnosis. If there are differing opinions, they are discussed. If the discussion fails, other professors are consulted to make a judgment, in order to minimize the doctor's personal subjective judgment and obtain a more accurate diagnosis result.

[0138] S3. Image Preprocessing: Establish a labeled dataset of tongue images, removing problematic images due to lighting, improper tongue extension movements, etc.; crop or scale the images to a uniform 224*224 resolution.

[0139] S4. Perform data augmentation on the images: Due to the special nature of tongue images, only random horizontal and vertical flipping and random cropping of 100×100 area are performed and stretched to build an extended tongue image data library.

[0140] S5. The image is input into the tongue image recognition network model. The experimental environment is a single GPU cluster with a GeForce RTX 2080 Ti GPU, a computing power of 7.5, and 12GB of graphics memory. The PyTorch deep learning library is used.

[0141] S6. The model is trained using 80% of the tongue images in the tongue image data expansion library for training and 20% for testing. The training iterations are set to 100, using the Adam optimizer with a batch size of 4 and a fixed learning rate of 0.0003. No further fine-tuning of the learning rate is performed to achieve a more general performance. During training, a callback function is used to shuffle the index order of the expanded data samples in the training set, resulting in a shuffled training set. In this embodiment, the callback function is called during training. This function is automatically executed at the end of each epoch (training period). Its purpose is to randomly shuffle the index order to facilitate the next batch of training data. The shuffle function determines whether the data should be shuffled during data generation.

[0142] S7. After the model training is completed, perform model testing to determine the test accuracy.

[0143] Finally, this model is compared with some classic and lightweight models. The comparison results are shown in Tables 1 and 2 below. Figure 7 and Figure 8As shown in the figure. Ultimately, it was found that this model outperforms other models in performance and is lightweight, enabling faster and more accurate identification of comprehensive tongue information from tongue images.

[0144] Table 1

[0145] Model Parameters She_mobile(Ours) 3.3M Mobilenetv3-large 4.2M MobileViT_Small 4.94M Resnet50 23.51M GoogLenet 5.98M VGG_16 134.27M Alexnet 14.59M

[0146] Table 2

[0147] Model accuracy She_mobile(Ours) 72％ Mobilenetv3-large 58％ MobileViT_Small 57％ Resnet50 65％ GoogLenet 46％ VGG_16 46％ Alexnet 46％

[0148] In the table, She_mobile(Ours) represents the identification method in this embodiment;

[0149] Mobilenetv3-large:

[0150] MobileNetV3 is a lightweight convolutional neural network, an improvement upon MobileNetV2. MobileNetV3 employs innovative designs to enhance model performance, such as reducing computation and memory usage through separable convolutions and linear bottleneck activation functions, and improving the model's expressive power through the Squeeze-and-Excitation module. MobileNetV3-large is a larger model in the MobileNetV3 series, boasting higher accuracy and a more complex network structure.

[0151] MobileViT_Small:

[0152] MobileViT is a lightweight image classification model based on the Transformer architecture, designed to solve the problem of efficient image classification on mobile devices. MobileViT_Small is a small model in the MobileViT series. Due to its lightweight Transformer structure and attention mechanism, it can significantly reduce the model size and computational cost while maintaining high accuracy.

[0153] Resnet50:

[0154] ResNet (Residual Network) is a very popular deep residual network that gained widespread attention due to its outstanding performance in the 2015 ImageNet image classification competition. ResNet employs a residual block design, allowing information to be passed across multiple layers in the network, thus solving problems such as vanishing gradients and model degradation. ResNet50 is a model in the ResNet series with 50 convolutional layers, commonly used for tasks such as image classification and object detection.

[0155] GoogLeNet:

[0156] GoogLeNet, a deep convolutional neural network proposed by Google in 2014, was the first neural network to beat the human visual system in the ImageNet image classification competition. GoogLeNet employs an Inception module design, allowing the network to extract features at different scales and layers, thereby improving accuracy while maintaining fewer parameters and computational cost. GoogLeNet has 22 layers, including 9 Inception modules and 3 fully connected layers.

[0157] VGG_16:

[0158] Visual Geometry Group (VGG) is a classic convolutional neural network proposed by a research team at Oxford University in 2014. A key feature of VGG is its use of multiple small convolutional kernels, allowing for deeper networks with better expressive power. VGG-16 is a model in the VGG series, featuring 16 convolutional layers and 3 fully connected layers, and is commonly used for image classification tasks.

[0159] Alexnet:

[0160] AlexNet is a classic deep convolutional neural network proposed by Alex Krizhevsky et al. in 2012, and it significantly outperformed traditional machine learning methods in the ImageNet image classification competition. AlexNet employs multiple layers of convolutional and pooling layers, resulting in a very deep network structure and a large number of trainable parameters. In addition, AlexNet uses innovative techniques such as the ReLU activation function, Dropout regularization, and data augmentation, enabling the network to generalize better and prevent overfitting.

[0161] The method in this embodiment can quickly and efficiently identify comprehensive tongue features. Its recognition performance is better than traditional convolutional neural network models and recent lightweight convolutional neural network models. Moreover, the model has a small number of parameters and can be well deployed on mobile devices to complete the recognition of tongue images.

[0162] Example 2

[0163] Based on Embodiment 1, this embodiment provides a tongue recognition system based on depthwise separable convolution, including:

[0164] Input module: Acquires the tongue image to be classified and performs feature extraction to obtain the input feature map;

[0165] Inverse residual module: It is configured to perform inverse residual operation on the input feature map, sequentially perform channel expansion operation and depthwise separable convolution operation, and capture the correlation information of the feature map channels and space based on channel attention mechanism and spatial attention mechanism;

[0166] Multiple residual inversion operations are performed in a cascade manner. The output of the previous residual inversion operation is connected to the next cascade operation, thereby increasing the number of channels and expanding the receptive field of the feature map to capture more detailed tongue image feature representation information in the image.

[0167] Classification output module: It is configured to convert the number of feature processing channels based on the tongue image feature representation information and then perform classification output.

[0168] The tongue image recognition system is implemented by constructing an image recognition network. The network structure of the tongue image recognition network is explained below.

[0169] In some implementation examples, the tongue recognition network uses a deep separable convolutional network, which integrates a residual network and a CASA attention submodule.

[0170] A depthwise separable convolutional network consists of, from input to output, a sequentially connected input module, multiple inverted residual modules, a convolutional normalization activation module, a global pooling layer, and a classification output module.

[0171] The input module is used to perform convolution and batch normalization on the input image, and after activation, the input feature map is obtained.

[0172] Multiple inverted residual modules are connected through residual connections, and the inverted residual operation is repeatedly performed to increase the number of channels and expand the receptive field of the feature map to capture more detailed information in the image and the feature representation information of the tongue image.

[0173] The convolutional normalization activation module is used to convert the number of feature processing channels in the output of the inverse residual module.

[0174] The global pooling layer is used to perform global average pooling on the feature map output by the convolutional normalized activation module.

[0175] The classification output module performs a linear transformation on the feature map after global average pooling to obtain the classification output.

[0176] The input module includes a first convolutional layer, a first batch normalization layer, and a first activation function layer connected in sequence;

[0177] The inverse residual module includes a first convolutional normalized activation submodule, a second convolutional normalized activation submodule, a CASA attention submodule, a third convolutional normalized activation submodule, and a residual connection module connected in sequence.

[0178] The convolutional normalization activation module includes a second convolutional layer, a second batch normalization layer, and a second activation function layer connected in sequence; the second activation function layer can use the HardSwish activation function.

[0179] The global pooling layer contains a global average pooling operation;

[0180] The classification output module consists of a third convolutional layer, a third activation function layer, a random dropout layer, and a linear output layer. The third activation function layer can use the HardSwish activation function.

[0181] Optionally, the CASA attention submodule learns to automatically acquire important features and spatially effective features in each feature channel, selectively boosting useful features in the channel and space, and suppressing useless features;

[0182] Specifically, the CASA attention submodule includes a channel attention component and a spatial attention component connected in sequence;

[0183] The channel attention component is used to adaptively weight each input channel;

[0184] The spatial attention component is used to adaptively weight each pixel.

[0185] The channel attention component is specifically a CA attention submodule, which uses the CA attention mechanism to reduce the channel attention component and fix the channel number rate to 16.

[0186] The CA attention submodule includes two global average pooling layers, one convolutional layer, one batch normalization operation, and one HardSwish activation function layer.

[0187] In some embodiments, the CA attention submodule is used to pool the input feature matrix, then fuse the resulting high and wide axis feature maps to generate the final high and wide axis attention mechanism, obtaining the channel attention feature matrix. Specifically, the processing includes the following steps:

[0188] The input data is processed through two global average pooling operations to obtain a high feature map matrix and a wide-axis feature map matrix, and then the two matrices are concatenated.

[0189] The concatenated matrix is ​​convolved and normalized according to a normal distribution, and then processed by an activation function.

[0190] The attention scores for the height and width axes are calculated by performing convolution operations on the feature maps processed by the activation function.

[0191] In some embodiments, the spatial attention part compresses the input data, which is the channel attention feature matrix, and establishes the correlation between spatial features through convolution operations. Then, feature extraction and function activation are performed sequentially to obtain the spatial mask of each pixel and return the product of the spatial mask and the original input.

[0192] In this embodiment, depthwise separable convolution and residual networks are used to achieve tongue image discrimination, avoiding the problems of overfitting and large number of parameters in the convolutional neural network model.

[0193] This embodiment also innovatively proposes a CASA attention mechanism that includes channel attention and spatial attention, and integrates it with depthwise separable convolution operations. This ultimately improves the accuracy and speed of model inference, and can be deployed on mobile devices. It eliminates the need for manual tongue image recognition, realizes the objectification of TCM tongue image judgment, and improves the accuracy of tongue image information recognition.

[0194] Example 3

[0195] This embodiment provides an electronic device, including a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When the processor executes the computer instructions, it performs the steps described in the method of Embodiment 1.

[0196] Example 4

[0197] This embodiment provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the steps described in the method of Embodiment 1.

[0198] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Various modifications and variations can be made to this disclosure by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

[0199] While the specific embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of this disclosure. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this disclosure are still within the scope of protection of this disclosure.

Claims

1. A tongue recognition method based on depthwise separable convolution, characterized in that, Includes the following steps: Obtain the tongue image to be classified and extract features to obtain the input feature map; The inverse residual operation is used to sequentially perform channel expansion and depthwise separable convolution operations on the input feature map, and capture the correlation information of the feature map channels and space based on channel attention mechanism and spatial attention mechanism; The inverse residual module includes a first convolutional normalized activation submodule, a second convolutional normalized activation submodule, a CASA attention submodule, a third convolutional normalized activation submodule, and a residual connection module connected in sequence. Each inverse residual module is configured to perform channel expansion and depthwise separable convolution operations sequentially on the input feature map. Based on channel attention and spatial attention mechanisms, it captures the correlation information between the channels and space of the feature map. After channel compression, a second feature map is obtained, which is then connected to the next inverse residual module or the convolution normalization activation module. The convolutional normalization activation module includes a second convolutional layer, a second batch normalization layer, and a second activation function layer connected in sequence; the second activation function layer uses the HardSwish activation function. The CASA attention submodule includes a channel attention section and a spatial attention section connected in sequence; The channel attention component is specifically a CA coordinate attention submodule, which uses the CA coordinate attention mechanism to reduce the channel attention component and fix the channel number rate to 16; The CA coordinate attention submodule includes two global average pooling layers, one convolutional layer, one batch normalization operation, and one HardSwish activation function layer. Multiple residual inversion operations are performed in a cascade manner. The output of the previous residual inversion operation is connected to the next cascade operation, thereby increasing the number of channels and expanding the receptive field of the feature map to capture more detailed tongue image feature representation information in the image. Based on the tongue image feature representation information, the feature processing channel number is converted and then classified and output.

2. The tongue recognition method based on depthwise separable convolution as described in claim 1, characterized in that: The tongue image recognition method is implemented by constructing a tongue image recognition network. The tongue image recognition network adopts a depthwise separable convolutional network, and a residual network and a CASA attention submodule are integrated into the depthwise separable network. A depthwise separable convolutional network consists of, from input to output, a sequentially connected input module, multiple inverted residual modules, a convolutional normalization activation module, a global pooling layer, and a classification output module. The input module is used to perform convolution and batch normalization on the input image, and after activation, the input feature map is obtained. Multiple inverted residual modules are connected through residual connections, and the inverted residual operation is repeatedly performed to increase the number of channels and expand the receptive field of the feature map to capture more detailed information in the image and the feature representation information of the tongue image. The convolutional normalization activation module is used to convert the number of feature processing channels in the output of the inverse residual module. The global pooling layer is used to perform global average pooling on the feature map output by the convolutional normalized activation module. The classification output module performs a linear transformation on the feature map after global average pooling to obtain the classification output.

3. The tongue recognition method based on depthwise separable convolution as described in claim 2, characterized in that, Training the tongue image recognition network also includes the step of constructing an expanded data sample set, as follows: Acquire tongue images from videos and extract tongue images to generate a tongue image database; Artificial dataset annotation was performed on images in the tongue image database. After annotation, the images are preprocessed and data augmented to create an expanded data sample set; Alternatively, data augmentation processing may include random horizontal flipping, random vertical flipping, or / and random cropping and stretching of the central region.

4. The tongue recognition method based on depthwise separable convolution as described in claim 2, characterized in that: The input module includes a first convolutional layer, a first batch normalization layer, and a first activation function layer connected in sequence; Alternatively, the global pooling layer may contain a global average pooling operation; Alternatively, the classification output module includes a third convolutional layer, a third activation function layer, a random dropout layer, and a linear output layer; the third activation function layer uses the HardSwish activation function.

5. The tongue recognition method based on depthwise separable convolution as described in claim 4, characterized in that: The CASA attention submodule learns to automatically acquire important features and spatially effective features in each feature channel, selectively enhancing useful features in the channel and space, and suppressing useless features. The channel attention component is used to adaptively weight each input channel; The spatial attention component is used to adaptively weight each pixel.

6. The tongue recognition method based on depthwise separable convolution as described in claim 1, characterized in that: The CA coordinate attention submodule is used to pool the input feature matrix, then fuse the resulting high and wide axis feature maps to generate the final high and wide axis attention mechanism, obtaining the channel attention feature matrix. Specifically, the processing includes the following steps: The input data is processed through two global average pooling operations to obtain a high feature map matrix and a wide-axis feature map matrix, and then the two matrices are concatenated. The concatenated matrix is ​​convolved and normalized according to a normal distribution, and then processed by an activation function. The attention scores for the height and width axes are calculated by performing convolution operations on the feature maps processed by the activation function. Alternatively, in the spatial attention part, the input data, which is the channel attention feature matrix, is compressed, and the correlation between spatial features is established through convolution operations. Then, feature extraction and function activation are performed sequentially to obtain the spatial mask of each pixel and return the product of the spatial mask and the original input.

7. A tongue recognition system based on depthwise separable convolution using the method described in claim 1, characterized in that, include: Input module: Acquires the tongue image to be classified and performs feature extraction to obtain the input feature map; Inverse residual module: It is configured to perform inverse residual operation on the input feature map, sequentially perform channel expansion operation and depthwise separable convolution operation, and capture the correlation information of the feature map channels and space based on channel attention mechanism and spatial attention mechanism; Multiple residual inversion operations are performed in a cascade manner. The output of the previous residual inversion operation is connected to the next cascade operation, thereby increasing the number of channels and expanding the receptive field of the feature map to capture more detailed tongue image feature representation information in the image. Classification output module: It is configured to convert the number of feature processing channels based on the tongue image feature representation information and then perform classification output.

8. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the steps of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, perform the steps of any one of claims 1-6.

Images