A tongue feature-oriented lightweight multi-classification method and system thereof

Abstract

This invention proposes a lightweight multi-classification method based on tongue image features. The method includes the following steps: S1, acquiring the original tongue image data dataset; S2, classifying the image data and constructing training and testing sets for different tongue image data based on the dataset; S3, training a lightweight network model, activation function parameters, and a classification threshold optimizer based on the training and testing sets to obtain a lightweight tongue image feature multi-classification model; S4, performing a classification task on the tongue images to be classified based on the obtained lightweight tongue image feature multi-classification model. Targeting the features of traditional Chinese medicine tongue images and addressing the requirements of refined classification tasks at the mobile device level, this method integrates the advantages of depthwise separable convolutional networks and residual networks. The model constructed in this way has a low size and computational cost, making it suitable for lightweight classification tasks on mobile devices, which helps to expand the application scope of the model.

Description

Technical Field

[0001] This invention belongs to the technical field of tongue image data processing, specifically relating to a lightweight multi-classification method and system based on tongue image features. Background Technology

[0002] With the rapid development of the modernization of traditional Chinese medicine (TCM), a crucial task for the development of modern TCM is to deeply explore the scientific connotations of TCM diagnosis and further improve the scientificity and effectiveness of TCM treatment models. In recent years, the State Administration of Traditional Chinese Medicine and relevant local departments have successively issued several guiding documents outlining strategic plans for the development of TCM. These documents point out that departments at all levels should pool their efforts to vigorously inherit, innovate, and develop the TCM industry, improve the level of informatization and standardization, leverage the advantages of big data and artificial intelligence, and build an "Internet + TCM health service" system.

[0003] In the era of big data, cloud computing, big data technology, and artificial intelligence technology have become key technologies supporting the objective diagnosis of traditional Chinese medicine. Introducing these cutting-edge technologies into tongue diagnosis in traditional Chinese medicine can effectively solve many problems such as the description of the inherent principles of syndrome differentiation and treatment in traditional Chinese medicine, the separation between macroscopic representation and microscopic realization, and the discovery of inherent knowledge in multi-modal datasets of tongue diagnosis, so as to achieve scientific qualitative deduction of syndrome differentiation relationships in tongue diagnosis in traditional Chinese medicine.

[0004] This study focuses on lightweight multi-classification models of tongue images in Traditional Chinese Medicine (TCM) and aims to achieve scientific qualitative deduction of syndrome differentiation relationships in TCM tongue diagnosis. This research is of profound significance in promoting the development of intelligent TCM-assisted diagnosis and treatment technology.

[0005] In view of this, it is very meaningful to propose a lightweight multi-classification method and system based on tongue image features. Summary of the Invention

[0006] To address the problems of large model size, high computational cost, limited application scope, and poor applicability in existing tongue feature-based recognition methods, this invention provides a lightweight multi-classification method and system based on tongue feature characteristics to solve the aforementioned technical deficiencies.

[0007] In a first aspect, this invention proposes a lightweight multi-classification method based on tongue image features, which includes the following steps:

[0008] S1. Obtain the original tongue image data dataset;

[0009] S2. Classify the image data and construct training and testing sets for different tongue image data based on the dataset;

[0010] S3. Train a lightweight network model, activation function parameters, and classification threshold optimizer based on the training set and test set to obtain a lightweight tongue image feature multi-classification model.

[0011] S4. Perform a classification task on the tongue image to be classified based on the obtained lightweight tongue image feature multi-classification model.

[0012] Preferably, S1 specifically includes:

[0013] S11. Use image color analysis software to perform quality analysis on the original tongue image, and select qualified images to proceed to the next stage of processing;

[0014] S12. Perform segmentation on the tongue image to obtain an image containing only the tongue body image;

[0015] S13. Reduce the tongue image to a size of 512*512.

[0016] Preferably, the lightweight network models mentioned in S3 include CnnTClNet1.0, MobileTClNet, and TongueIMC-Light V1, TongueIMC-Light V2, and TongueIMC-Light V3, which are obtained by optimizing and modifying MobileTClNet.

[0017] More preferably, the construction of the TongueIMC-Light V1 network includes:

[0018] Treating depthwise separable convolution as a residual function, short-circuit the input of the depthwise convolution and the output of the point-to-point convolution, and then serially connect a Batchnorm and a ReLU-SelfSwish operation layer, the result of which is used as the output of the residual block.

[0019] Preferably, the calculation formula for the activation function parameters in S3 includes:

[0020]

[0021] ReLU 6(x)=min(max(x,0),6)∈[0,6]

[0022]

[0023]

[0024] Where i is the channel number, x is the input value of the activation function on the i-th channel, and a i It is a coefficient that controls the slope of the negative half-axis. When a > 0, it is the slope coefficient of the convex function on the negative axis. The larger the value of a, the greater the convexity of the function.

[0025] Preferably, the design of the classification threshold optimizer in S3 includes:

[0026] S31. Establish an objective function based on the Quadratic Weighted Kappa coefficient to construct an evaluation function for tongue image feature classification results;

[0027] S32. Further optimize the objective function based on the Nelder-Mead method;

[0028] S33. Obtain the tongue image feature quantization classification threshold optimization algorithm to construct a classification threshold optimizer.

[0029] Preferably, the lightweight tongue image feature multi-classification model in S3 includes three parts: image segmentation and image processing, multiple network types of models, and the prediction results of the model based on the threshold determined by the threshold regression classifier.

[0030] In the second part, this invention also proposes a lightweight multi-classification system oriented towards tongue image features, including:

[0031] Dataset Acquisition Module: Used to acquire raw tongue image data datasets;

[0032] Classification module: used to classify the image data and construct training and testing sets for different tongue image data based on the dataset;

[0033] Training module: used to train a lightweight network model, activation function parameters, and classification threshold optimizer based on the training set and test set to obtain a lightweight tongue image feature multi-classification model;

[0034] Builder module: Used to build the classification threshold optimizer;

[0035] Execution module: Used to perform classification tasks on the tongue images to be classified based on the acquired lightweight tongue image feature multi-classification model.

[0036] Thirdly, embodiments of the present invention provide an electronic device, including: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect.

[0037] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the implementations of the first aspect.

[0038] Compared with the prior art, the beneficial results of the present invention are as follows:

[0039] (1) Based on the characteristics of tongue images in traditional Chinese medicine and the requirements of fine classification tasks at the mobile device level, this method integrates the advantages of depthwise separable convolution and residual network. The model constructed in this way has a low size and low computational cost, and is suitable for lightweight classification tasks on mobile devices, which is conducive to expanding the application scope of the model.

[0040] (2) A novel activation function and classification threshold optimizer are proposed for the feature distribution in tongue images, which can enhance the applicability of the model to the fine classification task of tongue images, and can also provide reference for the end classification research of lightweight deep networks. Attached Figure Description

[0041] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and, together with the description, serve to explain the principles of the invention. Other embodiments and many anticipated advantages of the embodiments will be readily recognized as they become better understood through reference to the following detailed description. Elements in the drawings are not necessarily to scale. The same reference numerals refer to corresponding similar parts.

[0042] Figure 1 This is a flowchart illustrating a lightweight multi-classification method based on tongue image features, according to an embodiment of the present invention.

[0043] Figure 2 The graphs of the ReLU-SelfSwish function with different values ​​assigned to parameters a and b in the embodiments of the present invention are shown.

[0044] Figure 3 A graphical comparison of the ReLU-SelfSwish function and the ReLU series activation functions in an embodiment of the present invention;

[0045] Figure 4 This refers to the original image and its feature projection results after being mapped by the activation function in an embodiment of the present invention;

[0046] Figure 5-1 , 5-2 5-3 are the network structures of CnnTClNet, MobileTClNet, and TongueIMC-Light V1 in the embodiments of the present invention, respectively;

[0047] Figure 6-1 , 6-2 These are, respectively, the residual learning network block in TongueIMC-Light V1 and the residual block of the 1×1 convolutional layer transforming the channel dimension in TongueIMC-Light V2, as described in the embodiments of the present invention;

[0048] Figure 7-1 , 7-27-3 are respectively the logical structure diagrams of the optimized and adjusted TongueIMC-Light V1 network, the TongueIMC-Light V2 network structure, and the TongueIMC-Light V3 network structure in the embodiments of the present invention;

[0049] Figure 8 This is a flowchart of the tongue image feature quantization classification threshold optimization process in an embodiment of the present invention;

[0050] Figure 9 This is an algorithm diagram for optimizing the threshold of tongue image feature quantization classification in an embodiment of the present invention;

[0051] Figure 10 This is a diagram illustrating the architecture of a lightweight tongue image feature quantification and classification model in an embodiment of the present invention.

[0052] Figure 11 This is a diagram showing the distribution of output values ​​of each network layer after optimizing the initial values ​​of the model weights in an embodiment of the present invention.

[0053] Figure 12-1 , 12-2 These are comparison charts showing the number of positive examples recalled in model experiments based on the cracked tongue test set and the number of positive examples recalled in model experiments based on the tooth-marked tongue test set, respectively, in embodiments of the present invention.

[0054] Figure 13 This is a visual comparison chart of the accuracy of experimental results for each model in the embodiments of the present invention;

[0055] Figure 14 This is a schematic diagram of the structure of a lightweight multi-classification system based on tongue image features, according to an embodiment of the present invention. Detailed Implementation

[0056] In the following detailed description, reference is made to the accompanying drawings, which form part of the detailed description and are illustrated by specific illustrative embodiments in which the invention may be practiced. In this regard, directional terms such as “top,” “bottom,” “left,” “right,” “up,” “down,” etc., are used with reference to the orientation of the described figures. Because components of the embodiments can be positioned in several different orientations, directional terms are used for illustrative purposes and are by no means limiting. It should be understood that other embodiments may be utilized or logical changes may be made without departing from the scope of the invention. Therefore, the following detailed description should not be taken in a limiting sense, and the scope of the invention is defined by the appended claims.

[0057] In intelligent clinical tongue diagnosis in Traditional Chinese Medicine (TCM), physicians expect intelligent tongue diagnosis systems to provide not only the tongue shape and coating category, but also grading information for each type of tongue feature. For example, based on the length and width of the cracks, a cracked tongue can be labeled as mild, moderate, or severe. Therefore, this invention, based on tongue image segmentation, studies a lightweight multi-classification method oriented towards tongue features. Technically and algorithmically, it aims to propose a network model with smaller size, lower computational cost, and higher classification accuracy. In terms of application, it seeks to expand the application scope of the lightweight model in portable, real-time mobile systems, improving its practical application value. Therefore, based on tongue image segmentation, this invention will focus on the construction of a lightweight multi-classification model for tongue image data, mainly including: optimization of the ReLU activation function, lightweight multi-classification network design, threshold classifier design, and model experiments and analysis.

[0058] Figure 1 An embodiment of the present invention discloses a lightweight multi-classification method oriented towards tongue image features, such as... Figure 1 As shown, the method includes the following steps:

[0059] S1. Obtain the original tongue image data dataset;

[0060] Specifically, this includes: S11, using image color analysis software to perform quality analysis on the original tongue image, and selecting qualified images to proceed to the next stage of processing;

[0061] S12. Perform segmentation on the tongue image to obtain an image containing only the tongue body image;

[0062] S13. Reduce the tongue image to a size of 512*512.

[0063] S2. Classify the image data and construct training and testing sets for different tongue image data based on the dataset;

[0064] S3. Train a lightweight network model, activation function parameters, and classification threshold optimizer based on the training set and test set to obtain a lightweight tongue image feature multi-classification model.

[0065] Specifically, the lightweight network models mentioned in S3 include CnnTClNet1.0, MobileTClNet, and TongueIMC-LightV1, TongueIMC-Light V2, and TongueIMC-Light V3, which are optimized and modified based on MobileTclNet.

[0066] The construction of the TongueIMC-Light V1 network includes: treating depthwise separable convolution as a residual function, short-circuiting the input of the depthwise convolution and the output of the point-to-point convolution, and then sequentially connecting a Batchnorm and a ReLU-SelfSwish operation layer, the result of which is used as the output of the residual block.

[0067] The calculation formula for the activation function parameters includes:

[0068]

[0069] ReLU 6(x)=min(max(x, 0), 6)∈[0, 6]

[0070]

[0071]

[0072] Where i is the channel number, x is the input value of the activation function on the i-th channel, and a i It is a coefficient that controls the slope of the negative half-axis. When a > 0, it is the slope coefficient of the convex function on the negative axis. The larger the value of a, the greater the convexity of the function.

[0073] Furthermore, the design of the classification threshold optimizer includes:

[0074] S31. Establish an objective function based on the Quadratic Weighted Kappa coefficient to construct an evaluation function for tongue image feature classification results;

[0075] S32. Further optimize the objective function based on the Nelder-Mead method;

[0076] S33. Obtain the tongue image feature quantization classification threshold optimization algorithm to construct a classification threshold optimizer.

[0077] Preferably, the lightweight tongue image feature multi-classification model includes three parts: image segmentation and image processing, multiple network types of models, and the prediction results of the model based on the threshold determined by the threshold regression classifier.

[0078] S4. Perform a classification task on the tongue image to be classified based on the obtained lightweight tongue image feature multi-classification model.

[0079] In a specific embodiment, the tongue classification task includes the classification of tongue body and tongue coating. Tongue body can be classified by shape into normal tongue, scalloped tongue, fissured tongue, prickly tongue, swollen / thin tongue, and peeled tongue; and by color into pale white, red, purplish-red, and bluish-purple. Tongue coating can be classified by appearance into thick / thin coating, moist / dry coating, and greasy coating; and by color into white, yellow, gray, and black. Each type of tongue body and tongue coating can be further subdivided into different levels. For example, fissured tongue can be classified into severe, moderate, mild, and normal, and other types are similar. This invention proposes a lightweight multi-classification method with dynamic thresholds to address the requirements of fine-grained classification. The following uses the fine classification of fissured tongue as an example to provide specific operation methods and steps:

[0080] 1. The tongue diagnostic instrument acquires raw tongue image data. First, image color analysis software is used to analyze the quality of the raw tongue image, and qualified images are selected to proceed to the next stage of processing. Second, the tongue image is segmented to obtain an image containing only the tongue body. Finally, the tongue image is reduced to a size of 512*512.

[0081] 2. Train a lightweight network model, activation function parameters, and classification threshold optimizer using the cracked tongue training and test sets.

[0082] 3. Model Application. Based on the well-trained lightweight tongue image classification model and classification threshold set, the classification model and classification threshold set are integrated to form a system for multi-class tasks of cracked tongue. Finally, the cracked tongue image to be classified is input into the system, and the system outputs the final classification result.

[0083] The technical effects of this invention include: targeting the features of tongue images in traditional Chinese medicine and focusing on the requirements of fine-grained classification tasks at the mobile device level, this method integrates the advantages of deep separable convolution and residual networks. The model constructed in this way has a low size and low computational cost, making it suitable for lightweight classification tasks on mobile devices, which is beneficial to expanding the application scope of the model; targeting the feature distribution in tongue images, the proposed novel activation function and classification threshold optimizer can enhance the applicability of the model to fine-grained tongue classification tasks, and can also provide reference for the research on the end classification of lightweight deep networks.

[0084] The overall concept of the technical solution of the present invention will be explained in detail below.

[0085] This invention implements the research and development of a lightweight multi-classification model for tongue image features based on deep learning networks. It is based on a portable and real-time system (CnnTClNet V1.0) of deep CNN. The system basically meets the acceptable standard and can automatically detect the hierarchical classification of a certain type of tongue image features, with good performance.

[0086] The ReLU activation function, f(x) = max(0,x), is a non-linear activation function. It is computationally simple, allows for fast network learning, and is suitable for transforming high-dimensional data. After sufficient layers of function mapping, it can approximate any function well. Compared to Sigmoid, ReLU avoids the gradient approaching zero as the function nears saturation, and also avoids the instability caused by excessive weight updates due to gradient accumulation greater than 1. Furthermore, ReLU sets the output of some neurons to zero, making the network sparser, which is more consistent with the principles of biological neural networks. Therefore, ReLU's many advantages have led to its widespread application in deep learning networks.

[0087] Given that the ReLU activation function exhibits the phenomenon of some neurons "dying" when processing negative input data, which indeed causes distortion or loss of some features, many improvements have been made to ReLU. Representative research results include ReLU6, Hard-Swish, and Randomzied_Leak_ReLU. The following is a brief introduction to these improved methods, followed by a detailed explanation of the construction principle of the new activation function proposed in this invention.

[0088] Based on the ReLU function, a variable parameter is introduced to control the data mapping of the function on the negative half axis. The specific improved formula Randomzied_Leak_ReLU (abbreviated as RReLU) is shown in equation (4-10).

[0089]

[0090] Where i is the channel number, x is the input value of the activation function on the i-th channel, and a i It is a coefficient that controls the slope of the negative half-axis. The coefficient can be different on different channels, allowing for flexible control or autonomous learning by the model. When a i When the value is 0, RReLU degenerates into the traditional ReLU function.

[0091] Leak-ReLU (LReLU for short) has a similar structure to the RReLU function, as shown in equation (4-11). The difference between the two is that the parameter a in the former is different. i The former can be determined during training, while the latter requires assigning a fixed value to the coefficient 'a', which is usually a large value.

[0092]

[0093] From equations (4-10) and (4-11), it can be seen that when α iWhen the value of is a small number, equation (4-10) can be transformed into equation (4-11), that is, the RReLU function is transformed into the LReLU function. These two improved functions only increase the number of parameters by a small amount compared to the traditional ReLU function, but reduce the phenomenon of "neuron death" and effectively improve the prediction performance of the network.

[0094] To enable deep learning models to better meet the requirements of low-precision embedded devices, avoid excessively large activation values ​​of input values ​​that could affect model stability, and reduce the precision requirements of the device, the maximum output of the ReLU activation function can be limited to 6. The modified activation function is called ReLU6. Its calculation method is shown in equation (4-12).

[0095] ReLU 6(x)=min(max(x, 0), 6)∈[0, 6] (4-12)

[0096] To address the application requirements of deep learning models on mobile devices, an improved Swish self-gating function, known as the Hard-Swish function, was designed based on activation functions such as ReLU6 and RReLU. Its complete formula is shown in equation (4-13). This function is similar to the Swish function, but its advantages are more pronounced. For example, it saves significant computational resources, and the forward and backward propagation calculations are simpler and faster.

[0097]

[0098] Based on the deployment and operation requirements of deep learning models on mobile and lightweight devices, and comparing the performance and characteristics of existing activation functions, the ReLU activation function was selected as the basic function for improvement in this invention. By fully absorbing the advantages of the above activation functions, a brand-new activation function, ReLU-SelfSwish, is proposed. The formula of this function is shown in equation (4-14).

[0099]

[0100] Where a > 0 is the slope coefficient of the convex function on the negative axis, and the larger the value of a, the greater the convexity of the function.

[0101] The following proves that the function is convex on the negative semi-axis when x ≤ 0. The theorem and its proof are given below:

[0102] Assuming x≤0, then It is a convex function.

[0103] Proof: Let f(x) = ReLU - SelfSwish(x), then we have:

[0104] Since the second derivative of f(x) is always greater than 0, f(x) is a convex function in the negative half-axis interval, thus proving the original theorem.

[0105] Therefore, when the activation function receives a negative input, it can perform inference at a lower cost because it is a convex function, and convex functions also have the characteristic of global optimality.

[0106] Figure 2 Graphs of the ReLU-SelfSwish function for different values ​​of a and b. Figure 2 It can be seen that when a and b are assigned different parameter values, the ReLU-SelfSwish function exhibits different shapes in the negative half-axis region. 'a' controls the convexity of the graph, and 'b' controls the convexity of the graph at the coordinate of its symmetry axis perpendicular to the x-axis. Therefore, introducing two varying parameters can enhance the robustness of the model and alleviate the information problem in the activation operation.

[0107] Figure 3 The graphs of several ReLU activation functions are given. By comparison, it is found that the ReLU-SelfSwish function proposed in this invention is similar to the Hard-Swish and Swish functions. However, the ReLU-SelfSwish function retains the activation mode of ReLU positive value input, which is more conducive to function convergence. On the negative half axis, its graph is a non-monotonic convex function.

[0108] To verify the feature recovery effect after number mapping of the present invention, the pixel value distribution of the simulated image was used to randomly generate a three-channel spatial matrix with a size of 56×56×3 and the element values ​​in the matrix were distributed in the range of [-255, 255]. Figure 4 (a) is the feature distribution map of the original input information projected onto the X and Y planes. Figure 4 (b) is the projection map after mapping with the ReLU activation function. Figure 4 (c) to Figure 4 (f) shows the projection of the ReLU-SelfSwish function for different values ​​of parameters a and b. The graph shows that 4(e) and 4(f) have the best results, exhibiting the lowest degree of activation disruption to the original information.

[0109] CnnTongueClNet V1.0 is built upon standard convolutional layers and consists of fifteen layers. Its input image resolution is 448×448. Each standard convolutional layer is followed by Batchnorm (BN) normalization and ReLU non-linear activation. Except for the first Conv1 layer, all other layers in this network use max pooling with a stride of 2 for downsampling. When max pooling performs downsampling and halves the feature map size, CnnTongueClNet V1.0 doubles the depth of the convolutional layers by using the output of the pooling layer as input, ensuring more thorough mapping and extraction of high-level features. Except for the first 5×5 layer, the downsampling architecture consists of three convolutional layers with a kernel size of 3×3. Finally, a 1×1 convolutional layer is used to transform the number of channels, and average pooling is used to compress the feature map size until a 1×1×1 regression value is output. CnnTClNet uses a single regression output with a threshold to classify tongue image features. The CnnTClNet network structure is represented as follows: Figure 5-1 As shown.

[0110] To further promote the use of convolutional neural networks on portable terminal devices, a series of improvements were made to the lightweight MobileNet network based on the requirements of intelligent TCM tongue diagnosis. Subsequent network structures in this invention are all improved networks, categorized into multiple versions. The MobileTClNet network is one of these improved model versions. Specifically, the improvements include: removing the Softmax layer compared to MobileNet, changing the filter shape of the FC layer from 1024×1000 to 1024×1, and using a single regression output to classify tongue image feature quantification, similar to the TongueClNet V1.0 network. Regarding input resolution, the CNNTClNet network was also referenced; the filter size in the average pooling layer was increased, and the spatial feature map size was reduced to 1×1 before executing the fully connected layer. Aside from the improvements and optimizations mentioned above, the rest of the MobileTClNet network is similar to MobileNet. Specifically, the convolutional kernels in each network layer are set to 3×3, using depthwise separable convolution. Following the convolution operation is batch normalization and the activation operation of the ReLU-SelfSwish function proposed in this invention. Its architecture is as follows: Figure 5-2 As shown.

[0111] While replacing the standard convolutions in CNN with depthwise separable convolutions in MobileTClNet yields smaller model size and lower computational cost compared to directly fine-tuning CNN, MobileTClNet encounters a convergence bottleneck during training. This convergence failure may be due to insufficient parameters, low complexity, or incompatibility with the architecture of depthwise separable convolutions. Overall, compared to CNN, MobileTClNet reduces model size and computational cost, and achieves similar performance in feature classification on the tongue image test dataset. The comparison results will be presented and analyzed in subsequent model experiments.

[0112] MobileTClNet is a shallow deep network with 28 layers, similar to the standard MobileNet. Although the capacity of a depthwise separable convolutional network is slightly less than that of a standard convolutional network, MobileTClNet is still a multi-layered deep network. Therefore, adding a residual learning module to MobileTClNet can be explored to achieve better performance. Residual learning can not only alleviate the degradation problem that occurs in deeper models, but also accelerate training time through better convergence.

[0113] Therefore, to fully leverage the classification performance of residual learning in MobileTClNet, this invention modifies and optimizes the MobileTClNet network from three perspectives, defining them as TongueIMC-Light V1, TongueIMC-Light V2, and TongueIMC-Light V3, respectively. This embodiment focuses on the construction process of the TongueIMC-Light V1 network. Although the model size of MobileTClNet is significantly smaller than or equal to that of CNNTClNet and MobileTClNet models, it still possesses a large number of depthwise convolutions and depthwise transfers. If W is directly added... s Short-circuit connection computation increases model size and computational cost, making the model size approach that of CNNTClNet, which contradicts the design goal of a lightweight model. Therefore, this invention applies residual learning only to deeply separable layers with identical input / output channels, requiring no additional parameters and a large number of W... s Short-circuit connection calculation cost. The structure of TongueIMC-Light V1 is as follows: Figure 5-3 As shown.

[0114] Here, depthwise separable convolution is treated as a residual function. The input of the depthwise convolution and the output of the point-to-point convolution are short-circuited and then connected in series with a Batchnorm and a ReLU-SelfSwish operation layer. The result is used as the output of the residual block. Figure 6-1 The process of constructing residual blocks in TongueIMC-Light V1 is presented.

[0115] Although the residual block structure chosen in this invention is not optimal, experiments using tongue image samples verify that embedding this residual block in DWSC yields the best output results. The reason for this is that MobileNet is essentially a lightweight deep neural network. In this network structure, residual learning can better compensate for important information lost during continuous convolution. The feature value obtained by adding the residual information to the corresponding points of the point-to-point convolution output is more conducive to the backpropagation of Batchnorm and ReLU-SelfSwish functions, thus making the model converge faster and improving its performance. Subsequent experimental results also fully demonstrate that TongueIMC-Light V1 outperforms MobileTClNet, almost approaching the classification accuracy of the standard convolutional neural network model CNNTClNet, while the TongueIMC-Light V1 model size and computational cost are reduced by several times.

[0116] While ensuring the model's detection performance, this invention also attempts to construct a lightweight model with smaller size and lower computational cost. It was observed that the last two depthwise separable convolutional filters of the TongueIMC-Light V1 network still have a high channel depth of 1024 layers, which results in a still large model size and computational cost. Therefore, this invention first attempts to find effective methods to compress the TongueIMC-Light V1 network without affecting or improving the overall model performance.

[0117] First, it can be observed that the parameters of the last two depthwise separable convolutional filters have relatively small values. In fact, filters with smaller weights have a relatively smaller impact on the results. Therefore, this invention attempts to directly remove these layers to compress the model size and computational cost. However, contrary to expectations, although it achieves good compression results, its performance significantly decreases. Second, to recover from the loss of removing the last two layers, this invention halves their depth and doubles the number of layers to maintain complexity. This means reducing the depth from 1024 to 512 and duplicating the layer, which also uses a depthwise separable 512 depthwise convolution. Although this method does not compress the model size and computational cost as much as directly removing the two layers, its compression effect is still significant, and the model's performance improves to some extent. Finally, this invention duplicates the first few shallow layers of depthwise separable convolutions with depths of 64, 128, and 256 to fill in the information loss, thereby improving the model's high-level feature extraction capability and aiming for better computational performance. After a series of structural adjustments, the model performance is significantly improved, but still not as expected. Further improvements to reduce computational cost will be described later. The adjusted network structure is as follows Figure 7-1 As shown.

[0118] In such Figure 7-1 Based on the network structure shown, this invention further attempts to modify the residual learning block, utilizing short-circuit real connections and short-circuit virtual connections as residual learning strategies to enhance the model's feature nonlinear mapping capability, making forward and backward propagation easier. Unlike TongueIMC-Light V1, TongueIMC-Light V2 applies short-circuit real connection residual learning when the number of input / output layer channels is the same, and applies short-circuit virtual connection residual learning when the number of input and output channels differs. Experiments have verified that these two hybrid residual learning strategies can significantly improve the model's tongue image feature quantization and classification performance. The final complete structure of TongueIMC-Light V2 is as follows: Figure 7-2 As shown.

[0119] In the residual learning block of TongueIMC-Light V2, when encountering different input / output channels, this invention uses a 1×1 convolution matrix variable to change the number of channels output by the convolutional layer in the graph. Figure 6-2 This is a structural diagram of the residual block in the TongueIMC-Light v2 model. The number of channels in the input layer and the output layer that are short-circuited in this residual block are different. Therefore, it is necessary to perform channel transformation on the input layer to keep it consistent with the number of output feature layers.

[0120] In practice, TongueIMC-Light V2 achieved the best performance on the tongue image test dataset, but it incurred a certain computational cost. The main reason is that copying previous layers and using 1×1 convolutional layers for channel propagation significantly increases computational cost. In terms of computational cost, the TongueIMC-Light V2-based model is higher than the MobileTC1Net and TongueIMC-Light V1 models, but much lower than the CNNTC1Net model. Regarding model size, the TongueIMC-Light V2 model is the smallest because the parameter values ​​of the last two depthwise separable convolutional filters are relatively large; this invention addresses this by removing and adjusting corresponding layers.

[0121] In the TongueIMC-Light V2 network model, replicating previous layers to buffer information loss and utilizing residual learning to directly propagate signals does indeed improve the model's tongue feature classification ability. However, this also incurs a certain computational cost. To find a balance between computational cost and model classification ability, this invention attempts to further explore improvement measures to reduce computational cost. Experimental results show... Figure 5-2 The depthwise separable convolutional filters in the last few layers of the TongueIMC-Light V2 network have small values, and removing them has little impact on the classification performance of the entire model. Inspired by this, this invention directly removes the last few layers of the TongueIMC-Light V2 network to form a new network structure, which this invention calls the TongueIMC-Light V3 network. Figure 7-3 The complete architecture of the TongueIMC-Light V3 network.

[0122] Since the last layer of the network is removed, the downsampling operation needs to be moved to the previous layer to correspond to the input of the next average pooling layer. This invention directly moves the downsampling to the DS-Conv13 layer to average the complexity (input size height and weights) across two resolutions. Because TongueIMC-LightV2 was already the smallest model, and the Res-MobileDRNet V3 model is built upon TongueIMC-Light V2, TongueIMC-LightV3 is smaller in size compared to all previous models, achieving further model size compression. The various versions of the model can be selected based on the specific requirements.

[0123] In this invention, due to the fine-grained classification characteristics of tongue image sample features, the Softmax or Sigmoid functions on the conventional output layer do not achieve ideal classification results. Therefore, this invention proposes a threshold classifier to implement multi-classification tasks for tongue image sample features. This involves converting labels into continuous values ​​within a range, thereby transforming the classification problem into a regression problem. The model is trained using regression values, which are the true classification values ​​of the sample data. This aims to make the model's output as close as possible to the regression values ​​of the true number of classifications of the sample. A threshold is set between all classifications to classify the tongue image. The classification mode proposed in this invention differs from common classification models. The classification model can modify the threshold to enhance its classification ability. This means that when the model training results are unstable, a larger regression and classification discriminant space can be given to the model, and vice versa when the model training is relatively stable. This measure improves the model's predictive ability. The threshold optimization algorithm and related evaluation metrics will be introduced in detail below.

[0124] The objective function of this model's predictor is based on Quadratic Weighted Kappa (QWK), which measures the consistency between any two classes in a multi-class classification of tongue image features. The QWK metric ranges from -1 to 1, but in practical applications, it is typically between 0 and 1. The goal of the QWK metric is to measure the difference between the model's predicted sample results and the true labeled sample distribution; it can also be considered an evaluation metric for the model's prediction quality. A higher QWK metric indicates more accurate predictions. The following describes the construction and calculation process of QWK for the task of quantizing and predicting tongue image features:

[0125] First, we set the quantitative classification levels of tongue image features and establish a quadratic error weight matrix W between different prediction results and the true value for the same label. i，j It is a symmetric square matrix, matrix W i，j The error weight element in the formula is a penalty coefficient for the consistency between the predicted result and the true label. The calculation method is shown in the following formula (4-15).

[0126]

[0127] Where i and j are the label category numbers predicted by the model and the true label category numbers, respectively, corresponding to matrix W. i，j The row and column numbers; N is the number of classification levels for tongue image features, corresponding to the square matrix W. i，j The order of the weight matrix. Below, taking the three classification levels of mild, moderate, and severe cracks in the cracked tongue as examples, the quadratic weight matrix W is given. i，j The intuitive expression is shown in Table 4.2.

[0128] Table 4.2 Quadratic Error Weight Matrix Wi，j

[0129]

[0130] Secondly, assuming the model makes predictions on the severity of cracks in 20 tongue images, the distribution of the predictions and the actual labels is shown in the confusion matrix C in Table 4.3.

[0131] Table 4.3 Confusion matrix C between predicted result distribution and true label distribution

[0132]

[0133] From the total number of samples in the horizontal and vertical directions of the confusion matrix C shown in Table 4.3, the expected rank matrix E can be obtained. The element E(i,j) in matrix E is equal to the product of the corresponding horizontal and vertical components divided by the total number of samples, i.e., E(i,j) = (T... i *T j ) / 20, as shown in Table 4.4.

[0134] Table 4.4 Sample Random Prediction Expectation Baseline Matrix E

[0135]

[0136] Finally, based on the above three matrices, the evaluation of the tongue image feature quantitative prediction results is carried out using Equation (4-15) according to the predicted results of the samples and the actual sample label distribution.

[0137]

[0138] Where k∈[0,1], the higher the value, the better the prediction effect of the model.

[0139] To find a suitable classification threshold, expanding or compressing the range of a class affects the range of neighboring classes. If the range of neighboring classes is compressed, the probability of predicting that class will relatively decrease, thus impacting the model's classification accuracy. Therefore, this invention uses the Nelder-Mead method to optimize the objective function and tongue image feature quantization classification threshold mentioned above. In machine learning, the Nelder-Mead method is often used to optimize objective functions in multidimensional space and is suitable for nonlinear optimization problems with unknown derivatives. This invention applies this algorithm to a tongue image training dataset to obtain an optimized classification threshold.

[0140] The essence of Nelder-Mead is a simplex algorithm. An n-dimensional Nelder-Mead includes n+1 test points. The objective function value of each test point is calculated. The goal is to find a new test point and replace the old test point with the worst result in the objective function with the new test point. Through iterative optimization, the goal of optimization is to obtain the optimal parameter values ​​in the state of minimizing the objective function.

[0141] F:R n Let R be an n-dimensional objective function. We will use the Nelder-Mead method to find the minimum value min(F) of the objective function, setting initial values ​​x1, x2, ..., xn for n+1 test points. n+1 Given a simplex, the Nelder-Mead method uses iteratively generated simplexes to approximate min(F) by using this as an initial value.

[0142] In one iteration: based on the magnitude of the objective function value, the test point set in the simplex is... The elements in the array are sorted in ascending order, as shown in equation (4-17).

[0143] F(x1)≤F(x2)≤…≤F(x n+1 (4-17)

[0144] The Nelder-Mead method comprises four operations: reflection, expansion, compression, and contraction, each associated with a scalar parameter: α (reflection), β (expansion), γ (compression), and δ (contraction). These parameters satisfy the following values: α > 0, β > 1, 0 < γ < 1, 0 < δ < 1. For objective functions with a dimension n ≥ 2, the calculation method for these four parameters can be chosen according to equation (4-18).

[0145]

[0146] This invention, based on the characteristics of the objective function, replaces the worst point with the simplest centroid reflection point. If the reflection point is better than the current point, the search continues along the direction of the reflection point; if the effect is still unsatisfactory, all points are shrunk towards a better direction. The Nelder-Mead method finds F(x) j If the minimum value of ) is found, then x1 can be determined as the optimal test point. n+1 For the worst test point, use equation (4-19) to calculate the centroid.

[0147]

[0148] Furthermore, the following section will focus on the construction process and algorithm implementation of the tongue image feature quantification predictor. Figure 8The flowchart shows the execution process of the tongue image feature quantization classifier. Assuming that the classification threshold is divided into n categories based on the quantization degree of a certain type of tongue image feature, these n categories form a set of parameters that need to be optimized using the Nelder-Mead method. First, an initial point X is set. T1 Additionally, generate n initial points X. T2 X T3 , ..., X Tn+1 , making X T1+i Compared to X on the i-th component T1 The weighted average is 5%, while other components remain the same. Next, regression values ​​are output from the multiple lightweight network models mentioned above, and the evaluation coefficients of the sample classification results are calculated using a quadratic weighted Kappa function. Finally, the Nelder-Mead method is used to iteratively optimize the n classification thresholds. These n classification thresholds will vary depending on the multi-class classification task and all need to be obtained through training.

[0149] The design details of the threshold optimization algorithm are given below. The algorithm mainly consists of two parts: a quadratic weight Kappa function to calculate and evaluate the model's classification performance, and a Nelder-Mead threshold iterative optimization module combined with it. Figure 9 The algorithm shown describes the entire process of optimizing the threshold for tongue image feature quantization classification.

[0150] Furthermore, we propose a new activation function and construct an optimized classification threshold module for the networks of several lightweight compressed models for multi-classification tasks of tongue image features, as described above. Based on this, we combine them to propose a lightweight tongue image feature detection model with lower computational cost and fewer parameters, as intuitively described below. Figure 10 As shown. The model's input is a tongue image captured by specialized tongue diagnostic equipment. The model's output is a classification label for the input image; the specific meaning of these labels will be explained in detail below.

[0151] Figure 10The model shown mainly consists of three parts. The first part is image segmentation and image processing, which removes external factors from the input image that may affect the model's detection accuracy and modifies the format of the input image to suit the lightweight deep network proposed in this invention. The second part integrates models of multiple network types described above (according to the key research task to which this invention is based, the network structure options of the model include CnnTClNet1.0, MobileTClNet, TongueIMC-Light V1, TongueIMC-Light V2, and TongueIMC-Light V3), and outputs classification regression values. The last part outputs the model prediction results using a threshold determined by a threshold regression classifier. Here, the threshold is optimized by executing the Nelder-Mead algorithm during model training. When the model is used to predict the route, the Nelder-Mead algorithm is not executed. Figure 10 The bottom of the diagram contains the classification network modules that can be selected during model training or prediction.

[0152] Specifically, the following sections will focus on experimental data related to model training, initialization of model hyperparameters, definition and setting of loss functions, selection of optimizers, and experimental results.

[0153] For the loss function of this model, the Euclidean loss function is used for training and optimizing the model parameters because the Euclidean function has a significant advantage in solving models where the output value is a regression result. Therefore, it is widely used in training regression output models, and since the output of the lightweight deep network classification model proposed in this invention is a regression value, it is included in the application. The complete loss function Loss is defined as Equation (4-20). The first part of Equation (4-20) is the Euclidean loss, which is used to calculate the sum of squared differences between the true label and the prediction.

[0154]

[0155] Where N is defined as the total number of training samples in a batch, T represents the true label, and P represents the predicted value. Since this invention aims to make the model's output value closer to the quantified level value of tongue appearance characteristics assessed by TCM clinicians, the true label of the tongue appearance is directly used within the quantification range. To alleviate overfitting and improve the model's generalization ability, in addition to using the Euclidean method, the weight decay is calculated using the L2 loss function in the latter part of the formula. Here, M represents the number of layers with weight decay, W represents the weight of the layer, and λ is the weight decay coefficient.

[0156] This invention uses the Nesterov Momentum Optimizer as the optimizer for a tongue image feature quantization classification model. In specific experiments, several high-performing optimizers were selected as the optimizer for this model, but the expected experimental results were not obtained. This may be due to factors such as the applicability of the optimizer or improper settings. The basic principle of the Nesterov Momentum optimizer and its application in this model are briefly outlined below. Before introducing it, momentum update is briefly described here. It is a stochastic gradient descent (SGD) method that incorporates momentum, which helps to accelerate SGD in relevant directions and suppress oscillations. Equation (4-21) shows the calculation method for momentum update. This method is well-suited to the optimization requirements of fine-grained tongue image classification models. It updates the past time vector v... t-1 A portion of the quantity γ is added to the current update vector v. t In this context, Θ represents the model parameters, Loss is defined as the loss function used for model training, and η represents the learning rate.

[0157]

[0158] Nesterov can give momentum updates a new meaning; in standard momentum updates, the momentum term γv is used. t-1 To move the parameter Θ, while in the Nesterov momentum update, it first calculates Θ. t-1 -γv t-1 This is an approximation of the next position of the parameters, and its gradient is calculated; then the gradient with respect to the approximate next position of the model parameters is calculated. Practical results show that this method is more efficient for gradient calculation in some application areas. The calculation equation for Nesterov momentum update is shown in equation (4-22).

[0159]

[0160] For the learning rate η, some modifications were made to allow it to decay exponentially, as shown in equation (4-23). The learning rate after decay is represented, α represents the base of decay, GS represents the global step size, and DS is defined as the decay step size. When the model's weights are updated once and the initial value is zero, the global step size will increase by 1, and it will increase by 1 in each iteration.

[0161]

[0162] Table 4.5 shows the settings for parameters such as Nesterov momentum update coefficient γ, initial learning rate η, learning rate decay base α, decay step size DS, and model weight decay coefficient λ.

[0163] Table 4.5 Parameter settings for the Nesterov momentum update optimizer

[0164]

[0165] From the above parameter value distribution, it can be seen that, in addition to the learning rate increasing with the decrease in model size, the weight decay coefficient also increases with the increase in model complexity, while the values ​​of other parameters remain consistent in each model.

[0166] The quality of weight initialization in a deep learning model has a significant impact on the model's convergence speed and overall quality. Regarding weight initialization, maintaining a relatively constant variance between layers in both forward and backward propagation directions will result in better learning capabilities for the network. Otherwise, an increase or decrease in the variance between input and output values ​​in the network layers will lead to saturation of nonlinear neurons in either the positive or negative directions.

[0167] In this model, ReLU-SelfSwish is used as the activation function. Referring to the He initialization method, which provides a weight parameter initialization scheme that maintains relatively stable variance between layers, the initialization calculation formula for the weight values ​​of each layer of the network can be designed based on the characteristics of tongue image feature distribution, as shown in equation (4-24).

[0168]

[0169] in, Let δ be the mean and δ be the variance. These represent the number of input and output nodes in the i-th layer network, respectively. Figure 11 This section describes the distribution of the first forward propagation output values ​​of a 19-layer fully connected network for a 512×512 tongue image. The activation function used is ReLU, and the initial weights of the i-th layer are calculated using equation (4-24), where the mean... It is assigned a value of 0, and the variance δ is assigned a value of 1.

[0170] Depend on Figure 11 The results show that after optimizing the initial values ​​of the network layer weights of the model using Equation (4-24), the output values ​​of each layer are more evenly and stably distributed. The mean of the output values ​​is concentrated between 0.9 and 0.26, and the variance is between 0.34 and 0.44, which is close to the mean and variance of the input values. The model weight initialization effect is good, which is beneficial to the training of the model.

[0171] Table 4.6 precisely shows the size and computational cost of various network architecture models. Here, the number of training variables is called the model size, and the total number of digit multiplications and additions is called the computational cost, which mainly includes element-wise digit multiplication and addition operations in convolutional layers and residual learning. The units for model size and computational cost are in millions.

[0172] Table 4.6 Model Size and Computational Cost

[0173]

[0174] As shown in Table 4.6, compared to the standard deep convolutional neural network model CnnTClNet, the new network models employing a combination of deep separability and residual learning structure optimization all have smaller sizes and lower computational costs. MobileTClNet and TongueIMC-LightV1 have the same model size and similar computational costs (Note: TongueIMC-LightV1 has additional element summation, but it is less than one million and is not specified in the table). In terms of computational cost, among the multiple sequence models proposed in this invention, the TongueIMC-Light V2 model has the highest computational cost, but compared to the CnnTClNet1.0 model, the TongueIMC-Light V2 model reduces the computational cost by more than half. In addition, the size of this model is smaller than MobileTClNet and TongueIMC-Light V1, approximately one-quarter the size of the CnnTClNet1.0 model. TongueIMC-Light V3 has the smallest model size, which is 1 / 6 the size of the CnnTClNet1.0 model; its computational cost is similar to that of MobileTClNet and TongueIMC-Light V1, and is about three times lower than that of the CnnTClNet1.0 model.

[0175] The experimental results will now be analyzed in a specific embodiment:

[0176] Currently, there are no large-scale public datasets of tongue images on public network platforms. Since the research conducted in this paper is based on a sub-project of a national key research and development project, the experimental data of the model in this invention were all collected by a professional tongue diagnosis instrument developed independently. The tongue images collected manually came from patients with tumors, diabetes, hypertension, stomach diseases, and other diseases, as well as healthy individuals undergoing physical examinations. To date, a total of 41,004 tongue images have been collected, including 5,046 images of peeled tongue and 6,233 images of tongue with teeth marks. Table 4.7 shows the distribution of the two types of tongue images.

[0177] Peeling tongue refers to a tongue coating that has been completely or partially peeled off. A completely peeled tongue is smooth and mirror-like, and is called a smooth peeling tongue, also known as a mirror tongue or a glossy tongue. If the tongue coating is not completely peeled off, the peeled areas are smooth and without coating, while the remaining areas have mottled and clearly defined coating, it is called a mottled peeling tongue. Tooth-marked tongue refers to a tongue with visible teeth marks along its edges, also known as a tooth-marked tongue. This is often caused by the tongue being enlarged and compressed by the edges of the teeth, and is a type of abnormal tongue shape. This invention's model experiment uses two characteristics, fissured tongue and tooth-marked tongue, as examples. Traditional Chinese medicine clinicians rated the fissured or tooth-marked features of the tongue in each image using four levels: 0 to 3 (meaning: Label 0 - "normal", Label 1 - "mild", Label 2 - "moderate", Label 3 - "severe"). Table 4.8 shows the visualization of tongue samples under different quantitative standards rated by TCM professionals.

[0178] Table 4.7 Training and Test Sets of Tongue Images

[0179]

[0180]

[0181] After processing the experimental dataset, firstly, 10% of the samples in the training set are randomly extracted as validation datasets. The distribution of the validation set is shown in Table 4.9. Next, all experimental data are preprocessed, including scaling the original images and initial tongue segmentation. Finally, the tongue segmentation method proposed above is used to segment all tongue data. Specifically, due to the imbalanced and insufficient dataset, the model can only focus on labels with more training samples and less robustness. Furthermore, images may contain residual noise and distortion, affecting the model's learning performance. Therefore, data augmentation techniques are used to address the imbalance and insufficiency of the dataset and to cover external factors that image preprocessing cannot handle, or even problems that image preprocessing may cause. Data augmentation measures mainly include random rotation, shifting, flipping, scaling, and GAN tongue augmentation methods. Through data augmentation, the model can learn on a relatively balanced and sufficient dataset, allowing it to focus on learning key features and problems, minimizing the impact of other factors on the model. It should be noted that augmentation operations are performed only on the data in the training set, and no data augmentation is performed on the validation and test sets, because the originality and authenticity of the validation and test sets must be maintained.

[0182] Table 4.9 Distribution of tongue image training and validation sets

[0183]

[0184] The distribution of the enhanced training dataset is shown in Table 4.10.

[0185] Table 4.10 Distribution of the training set after tongue image enhancement

[0186]

[0187]

[0188] To verify the effectiveness of the ReLU-SelfSwish function, the CIFA-100 public dataset was selected as the experimental sample. Table 4.11 shows the ReLU-SelfSwish-based MobileNet model, which incorporates the activation function proposed in this invention. The corresponding activation functions for comparison include ReLU, RReLU, Swish, and Hard-Swish. The CIFAR-100 dataset contains 60,000 three-channel images. All images in the CIFAR-100 dataset are divided into 100 categories, with 600 images in each category. The training and test sets are split in a 7:3 ratio. In the experimental samples of the CIFAR-100 dataset, the training set contains 42,000 images, and the test set contains 18,000 images, with 180 test images for each category. The experimental network model used was MobileNet V1. The training parameters were set as follows: 100 epochs, 128 batch sizes, and a learning rate of 0.1. Based on the prepared experimental data and parameters, classification experiments were conducted using MobileNet models incorporating five different activation functions. The results show that the activation function proposed in this invention has a significant advantage in classification accuracy. Compared to the traditional ReLU function, it achieves nearly 6 percentage points higher TOP-1 classification accuracy on the CIFAR-100 dataset. Compared to other activation functions, it also shows an accuracy improvement of 2 to 5 percentage points. The experimental results are shown in Table 4.11.

[0189] Table 4.11 Comparison of classification accuracy of MobileNet model under different activation functions

[0190] Model Name Top-1 Top-5 ReLU-based MobileNet 65.02％ 88.29％ RReLU-based MobileNet 69.58％ 89.11％ Swish-based MobileNet 68.79％ 90.26％ Hard-Swished-based MobileNet 68.32％ 89.93％ ReLU-SelfSwish-based MobileNet 71.06％ 89.87％

[0191] (Note: top-1 and top-5 refer to the accuracy of the images with the highest and highest classification probabilities in the model output, respectively, compared with the true labels of the test images.)

[0192] The experimental results of the model will be analyzed below based on the test set. A direct and effective method for evaluating the accuracy of the model results is to statistically analyze the accuracy of tongue image feature classification. The calculation method for the accuracy is shown in Equation (4-25).

[0193]

[0194] Where TP, FP, TN, and FN represent the number of correctly classified and misclassified positive samples, and the number of correctly classified and misclassified negative samples, respectively. TP+FP+TN+FN is the total number of all samples.

[0195] Table 4.12 presents a comparison of the accuracy of the model experiments based on the cracked tongue test set. Due to the imbalance in the amount of data for each type of test label in the tongue image test set, the overall accuracy of all categories and the average accuracy of the four category labels were added to more comprehensively measure the quality of the model's computational results. Table 4.12 shows that TongueIMC-LightV2 achieved the best computational accuracy. The TongueIMC-LightV1 model is quite close to the CnnTClNet1.0 model; however, the TongueIMC-LightV1 computational results are slightly better than the MobileTClNet model. Overall, the TongueIMC-LightV2 model outperforms all other models except for TongueIMC-LightV1.

[0196] Table 4.12 Comparison of accuracy of model experiment results based on cracked tongue test set

[0197]

[0198] To further test the model's effectiveness in correctly classifying positive samples, Figure 12-1 The paper presents a comparison of the number of positive examples recalled in the model experiment based on the cracked tongue test set. Figure 12-1 It can be seen that the number of positive examples recalled by all models differs somewhat from the number of true labeled samples. However, relatively speaking, the TongueIMC-Light V2 model has the highest positive example recall rate, which is closest to the number of true labels for each category. In addition, it can be seen that multi-label classification for tongue image features is quite difficult, and the classification accuracy for each category and the overall classification is not very high.

[0199] Table 4.13 shows the accuracy comparison of the model experiment results based on the scalloped tongue test set. Compared with the model experiment results based on the fissured tongue test set, the model proposed in this invention has a certain degree of reduction in the accuracy of fissured tongue feature quantification classification, which confirms that the feature extraction and mapping of scalloped tongue is more difficult than that of fissured tongue. This is consistent with the fact that it is more difficult for professional doctors to quantify and classify the features of tongue image edge.

[0200] Table 4.13 Comparison of accuracy of model experiment results based on the scalloped tongue test set

[0201]

[0202] and Figure 12-1 Similarly, in Figure 12-2 The paper presents a comparison of the number of positive examples recalled in the model experiment based on the tooth-marked tongue test set. Figure 12-2 It can be seen that the TongueIMC-Light V2 model has the highest positive example recall rate, followed by the TongueIMC-Light V3 model, while the MobileTClNet model has the worst positive example recall rate. Analysis shows that the poor performance of the MobileTClNet model is due to insufficient extraction of tongue image features; some feature information is missed during the shallow-depth separation convolution operation.

[0203] Next, to further test the model's classification ability, the four-label classification problem was changed to a two-label binary classification problem. Labels Label1 to Label3 were merged into a single label Label1, where Label0 represents a normal tongue and Label1 represents an abnormal tongue. Therefore, two metrics for evaluating binary classification performance—sensitivity and specificity—were used to assess the model. Sensitivity, also known as the true positive rate, measures the proportion of correctly identified positive samples, while specificity, known as the true negative rate, is often used to measure the proportion of correctly identified negative samples.

[0204] Equation (4-26) defines the calculation method for the two evaluation indicators, Sensitivity and Specificity.

[0205]

[0206] The meanings of TP, FP, TN, and FN are the same as in equation (4-19), so they will not be elaborated upon.

[0207] Tables 4.14 and 4.15 present a comparison of the binary classification results among the various models for the two test sets: cracked tongue and tooth-marked tongue. The total metric in Tables 4.14 and 4.15 is calculated as (TP+TN) / (TP+TN+FP+FN), and the average metric is the average of the sensitivity and specificity of each model. Table 4.14 shows that the performance of each model when switching to binary classification is similar to that of the same model in four-class classification. TongueIMC-Light V2 also has the best evaluation result. The TongueIMC-Light V3 model is slightly lower than the TongueIMC-Light V2 model, but better than models such as CnnTClNet1.0. Overall, the classification performance of the network models improved using the residual learning method is quite similar.

[0208] Table 4.14 Comparison of experimental results of binary classification model based on cracked tongue test set

[0209]

[0210] Table 4.15 Comparison of experimental results of binary classification of the model based on the tooth-marked tongue test set

[0211]

[0212] The design process of the classification threshold optimization module was described above. Based on this, the Softmax or Sigmoid classification function was removed from the end of the lightweight tongue image feature multi-classification model, allowing the model to output quantifiable regression values. The threshold extracted by the tongue image feature quantization multi-classification optimizer was then used for classification. To verify the effectiveness of the classification optimization threshold extractor, several ablation experiments were conducted. Tables 4.16 and 4.17 show the ablation experiment results for two test sets: one based on a cracked tongue and the other on a tongue with teeth marks. As can be seen from the tables, the classification performance of each model was significantly improved after using the classification optimization threshold extractor, with the classification accuracy increasing by 1 to 5 percentage points. In Tables 4.16 and 4.17, "before" indicates the experimental results of the model without using the classification optimization threshold extractor, and "after" indicates the model results after obtaining the classification threshold using the threshold extractor.

[0213] Table 4.16 Comparison of model ablation experimental results based on cracked tongue test set and threshold extractor

[0214]

[0215] Table 4.17 Comparison of model ablation experimental results based on the tooth-marked tongue test set and threshold extractor

[0216]

[0217]

[0218] Tables 4.18 and 4.19 show the experimental results of each comparative model for multi-classification (normal, mild, moderate, severe) and binary classification (normal, abnormal) of tongue features. Experiments were performed on two sets of data: fissured tongue and teeth-marked tongue. The data in the tables are the average values ​​of the four indicators in the two sets of experiments. Figure 13The experimental results provide a visual comparison of the accuracy of each model. The experimental results show that the TongueIMC-Light series models proposed in this invention achieve the highest scores in both accuracy and F1-Score, with positive sample recall rates exceeding 75% under both classification conditions. This demonstrates the significant advantages of the proposed models in the refined classification of tongue features, laying a foundation for subsequent research on intelligent TCM tongue diagnosis.

[0219] Table 4.18 Comparison of Multi-Classification Accuracy of Models

[0220]

[0221] Table 4.19 Comparison of Binary Classification Accuracy of the Models

[0222] To address the requirements of tongue image feature quantification and classification, this invention proposes a network model with smaller size and lower computational cost based on MobileNet and ResNet networks. Simultaneously, a novel activation function is designed to effectively reduce information loss during feature activation, thereby improving the model's classification performance. Considering the characteristics of tongue image feature quantification and classification tasks, a classification threshold optimizer is constructed using the Nelder-Mead algorithm and a quadratic weighted Kappa function. During training, the threshold optimizer is connected to the tail of the deep learning network to obtain a threshold set suitable for different tongue image feature classifications, aiming to further improve the model's classification accuracy.

[0223] Multiple experiments based on public datasets and clinical tongue diagnosis datasets demonstrate that the lightweight tongue image feature quantification classification model proposed in this invention operates stably, showing improvements in several metrics, including model size and classification accuracy. The model essentially meets the application standards of portable, low-precision mobile devices. However, the proposed model also has some shortcomings. For example, the reduction in inference time is not proportional to the reduction in model size and computational cost, possibly due to insufficient optimization in model initialization and CPU parallel control. Therefore, future research aims to find better methods to address this issue, further compress the model's inference time, actively explore smaller structures or compression methods, and further compress the model while maintaining similar performance, in order to continuously improve the model's application value and achieve more accurate classification of tongue images.

[0224] Secondly, embodiments of this application also propose a lightweight multi-classification system oriented towards tongue image features, such as... Figure 14 As shown, it includes:

[0225] Data set acquisition module 141: Used to acquire the original tongue image data dataset;

[0226] Classification module 142: used to classify the image data and construct training and testing sets for different tongue image data based on the dataset;

[0227] Training module 143: used to train a lightweight network model, activation function parameters and classification threshold optimizer based on the training set and test set to obtain a lightweight tongue image feature multi-classification model;

[0228] Module 144: Used to build a classification threshold optimizer;

[0229] Execution module 145: used to perform a classification task on the tongue image to be classified based on the acquired lightweight tongue image feature multi-classification model.

[0230] In another aspect, the present invention also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to perform the method steps as described in the first aspect.

[0231] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention is not limited to the specific combination of the above-described technical features, but also includes other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in this invention.

Claims

1. A lightweight multi-classification method oriented to tongue image features, characterized in that, The method comprises the following steps: S1, obtaining an original tongue image data set; S2, classifying the image data and constructing a training set and a test set of different tongue data according to the data set; S3, training a light network model, an activation function parameter, and a classification threshold optimizer according to the training set and the test set to obtain a light tongue feature multi-classification model; the light network model comprises CnnTClNet1.0, MobileTClNet, and TongueIMC-Light V1, TongueIMC-Light V2 obtained by optimizing and reforming the MobileTclNet network; S4, performing a classification task on a tongue image to be classified according to the obtained light tongue feature multi-classification model; The construction of the TongueIMC-Light V1 network comprises: regarding a depth separable convolution as a residual function, short-circuit connecting the input of a depth convolution and the output of a point convolution, and then serially connecting a Batchnorm and a ReLU-SelfSwish operation layer, and taking the result as the output of the residual block; in the residual learning block of the TongueIMC-Light V2, when different input / output channels are encountered, a 1×1 convolution matrix variable is used to transform the channel number of the output of the convolution layer in the graph; The calculation formula of the activation function parameter is: ； in, Channel number, In the first The input value of the activation function on the channel. The slope coefficient of the convex function on the negative axis. The larger the value, the greater the convexity of the function; The design of the classification threshold optimizer comprises: S31, establishing a target function based on a Quadratic Weighted Kappa coefficient to construct a tongue feature classification result evaluation function; S32, further optimizing the target function based on a Nelder-Mead method; S33, obtaining a tongue feature quantitative classification threshold optimization algorithm to construct a classification threshold optimizer.

2. The light-weight multi-classification method oriented to tongue features according to claim 1, characterized in that, S1 specifically comprises: S11, using image color analysis software to analyze the quality of the original tongue image, and selecting qualified images to enter the next stage of processing; S12, performing segmentation on the tongue image to obtain an image containing only the tongue image; S13, scale the tongue image down to a size image.

3. The light-weight multi-classification method oriented to tongue features according to claim 1, characterized in that, The light tongue feature multi-classification model in S3 comprises three parts of image segmentation and image processing, a plurality of network type models, and a threshold output model prediction result determined by a threshold regression classifier.

4. A lightweight multi-classification system oriented to tongue picture features, characterized in that, For implementing the method as claimed in any one of claims 1-3, comprising: a data set acquisition module for acquiring an original tongue image data set; a classification module for classifying the image data and constructing a training set and a test set of different tongue data according to the data set; a training module for training a light network model, an activation function parameter, and a classification threshold optimizer according to the training set and the test set to obtain a light tongue feature multi-classification model; a construction module for constructing a classification threshold optimizer; an execution module for performing a classification task on a tongue image to be classified according to the obtained light tongue feature multi-classification model.

5. An electronic device, comprising: one or more processors; storage means for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement a method as claimed in any one of claims 1-3.

6. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program, which when executed by a processor, implements a method as claimed in any one of claims 1-3.

Images