A Cross-Domain Tongue Image Classification Method Based on Deep Hierarchical Optimal Transport

Abstract

The present invention discloses a cross-domain tongue image classification method based on deep hierarchical optimal transport, which includes collecting tongue images; constructing a machine learning classification model, where the machine learning classification model uses a deep hierarchical optimal transport model to align the features of tongue images in different domains. The deep hierarchical optimal transport model includes a two-layer network structure. Among them, the first-layer network structure is used to achieve optimal transport between different domains, and the second-layer network structure is used to achieve optimal transport between different samples; classify the tongue images according to the deep hierarchical optimal transport model and output the tongue image categories. The beneficial effect of the present invention is to align tongue images with different distributions while enhancing the classification ability of tongue images.

Description

Technical Field

[0001] The present invention relates to the technical field of tongue image classification for assisting traditional Chinese medicine diagnosis and treatment, and more specifically, to a cross-domain tongue image classification method based on deep hierarchical optimal transport. Background Art

[0002] Most of the existing tongue image classification methods based on machine learning are based on supervised learning. The supervised learning method usually assumes that the training set and the test set follow the same distribution. Therefore, the model trained on the training set can also perform well on the test set. However, in practical applications, such an assumption is difficult to hold. The main idea to solve this kind of problem is to assume that two data distributions are non-linearly mapped to a common latent space between domains to reduce the drift between distributions and make the two distributions more similar after being transformed by the non-linear mapping. This non-linear mapping process is called domain adaptation.

[0003] The existing tongue image classification based on machine learning faces such problems. First, the tongue images of different people have differences, including the edge texture, color, etc. of the tongue images. Secondly, the tongue image acquisition devices in different hospitals may be different, and the collected tongue image data is also affected by the acquisition environment, such as the angle, illumination, etc. In addition, the geographical locations of different hospitals are different, and there are also regional differences in the individuals from whom the tongue images are collected. These factors lead to relatively large distribution differences in the tongue image data of different hospitals. If the tongue image data of each hospital is domain data, then the data distributions of different domains are different, and these differences will cause the model trained on the collected data set to have serious performance degradation when deployed to other hospitals. At the same time, due to the high cost of annotating medical data, it is more difficult when there are no labels in the target domain, that is, only the labels of the source domain are available.

[0004] To solve this problem, the distributions in different domains need to be aligned. Currently, the mainstream methods for solving the alignment of distributions in different domains mainly include two steps: First, make the two distributions closer through non-linear transformation; then, use the label information of the source domain to train a classifier for the target domain on the transformed distribution, so that the model can generalize to the target domain, which is also the process of inter-domain knowledge transfer. It can be seen that how to find this non-linear transformation is the key to solving the domain adaptation problem. In recent years, the optimal transport method has shown great advantages in the domain adaptation problem. It can directly measure the distance between two distributions on the marginal distributions without the need for label information. In the visual field, this distance based on optimal transport is called the EMD (Earth Mover’s Distance) distance. On the one hand, it can directly calculate the distance between two distributions on the discrete empirical distributions (domains). On the other hand, it can also provide meaningful gradients when the support sets of the two domains do not significantly overlap, so it is not easy to cause training failure. In addition, it has good interpretability and can explicitly model the coupling between domains.

[0005] By optimizing the minimum transport cost between the feature distributions of the two domains, the distributions of the source domain and the target domain can be transformed into a common latent space at the minimum cost, and the features in this latent space are domain-invariant. This process is called domain alignment. The classifier trained on such features has the ability to transfer to the target domain. However, domain alignment is not the ultimate goal, classification is. However, in optimal transport, the cost matrix is usually calculated by computing the Euclidean distance (L2 distance) between pairwise samples. In such a metric space, when the support sets of two samples do not overlap, no meaningful distance can be provided. This is manifested in visual problems as when the backgrounds of two samples are too cluttered or have large intra-class appearance variations, images of the same category may be far apart in such a metric space. In other words, the L2 distance is greatly affected by background changes at this time. Although this can be alleviated through neural network modeling, it requires sufficient training data, which is very difficult to achieve in actual scenarios (especially in medical scenarios). It is necessary to emphasize the local features of the target area. At the same time, as a global representation, the L2 distance destroys the spatial structure of image features and loses local information. And local information can provide discriminative and transferable information, which is very important for the classification task. Especially in the traditional Chinese medicine tongue image dataset, the standardization degree of the acquisition process is poor, and the background or environmental factors change greatly. For the above reasons, in the existing domain alignment process, while obtaining domain-invariant features, the class discriminability of the features will also be blurred, that is, over-alignment occurs.

[0006] Therefore, how to avoid the phenomenon of over-alignment in the process of tongue image classification and improve the accuracy of tongue classification images is a technical problem that needs to be urgently solved by those skilled in the art. Summary of the Invention

[0007] To solve these problems, the present invention discloses a cross - domain tongue image classification method based on deep hierarchical optimal transport, enabling the machine learning model to learn invariance features that are more robust to environmental noise, endowing tongue image data with different distributions with adaptive capabilities, and improving the classification accuracy.

[0008] To achieve the above object, the present invention adopts the following technical solutions:

[0009] A cross - domain tongue image classification method based on deep hierarchical optimal transport, comprising:

[0010] S1. Collect tongue image samples from multiple different domains as the training set;

[0011] S2. Use a deep neural network to extract features from the source - domain tongue image samples in the training set, and obtain a source - domain image feature map composed of the corresponding source - domain tongue image sample features;

[0012] Use a deep neural network to extract features from the target - domain tongue image samples in the training set, and obtain a target - domain image feature map composed of the corresponding target - domain tongue image sample features;

[0013] S3. Divide the source - domain tongue image sample features in the source - domain image feature map into blocks to obtain a source - domain image feature set corresponding to the source - domain tongue image samples;

[0014] Divide the target - domain tongue image sample features in the target - domain image feature map into blocks to obtain a target - domain image feature set corresponding to the target - domain tongue image samples;

[0015] S4. Calculate the optimal transport distance between the source - domain image feature set corresponding to each source - domain tongue image sample and the target - domain image feature set corresponding to the target - domain tongue image sample as the sample - optimal transport distance between the source - domain tongue image sample and the target - domain image sample;

[0016] S5. Take the sample - optimal transport distance as the cost metric between the source domain and the target domain, and calculate the inter - domain optimal transport distance between the source domain and the target domain;

[0017] S6. Calculate the softmax cross - entropy loss based on the source - domain tongue image sample feature values extracted in step S2 as a part of the loss function; take the inter - domain optimal transport distance as another part in the loss function, construct a classification loss function, and use the classification loss function to train the classifier;

[0018] S7. Use the trained classifier to classify the tongue image samples to be verified.

[0019] In step S4, calculating the optimal transport distance between the source domain image feature set corresponding to each source domain tongue image sample and the target domain image feature set corresponding to the target domain tongue image sample includes the following methods:

[0020] Jointly using the EMD distance and the L2 distance as the cost function between the source domain image feature set and the target domain image feature set, and calculating the optimal transport distance between the two feature sets. The specific cost function includes:

[0021]

[0022] where g represents the feature extractor of the deep neural network;

[0023] represents the source domain image feature map extracted from the i-th source domain tongue image sample, represents the i-th source domain tongue image sample, H i and W i respectively represent the width and height of the source domain image feature map extracted from the i-th source domain tongue image sample;

[0024] represents the target domain image feature map extracted from the j-th target domain tongue image sample, represents the j-th target domain tongue image sample, H j and W j respectively represent the width and height of the target domain image feature map extracted from the j-th target domain tongue image sample;

[0025] represents the joint image feature map of the source domain image feature map and the target domain image feature map;

[0026] γ in represents the optimal transport plan between any source domain tongue image sample and any target domain tongue image sample with respect to the corresponding image feature set, C in represents the cost matrix between any source domain tongue image sample and any target domain tongue image sample with respect to the corresponding image feature set; <γ in ,C in > F represents γ in and C in 's Frobenius dot product; represents the global average pooling result of the source domain image feature map extracted from the i-th source domain tongue image sample along the spatial dimension, It represents the result of global average pooling of the target domain image feature map extracted from the j-th target domain tongue image sample along the spatial dimension. ch represents the number of channels.

[0027] Preferably, in step S4, calculating the optimal transport distance between the source domain image feature set corresponding to each source domain tongue image sample and the target domain image feature set corresponding to the target domain tongue image sample further includes the following method:

[0028] Jointly use the SWD distance and the L2 distance as the cost function between the source domain image feature set and the target domain image feature set, and calculate the optimal transport distance between the two feature sets. The specific cost function includes:

[0029]

[0030] P represents the permutation matrix. represents the set of all permutation matrices, U i represents converting the features corresponding to the source domain tongue image sample to a common high-dimensional hidden layer space, U j represents converting the features corresponding to the target domain tongue image sample to a common high-dimensional hidden layer space, and T is the matrix transpose symbol.

[0031] Preferably, in step S4, calculating the optimal transport distance between the source domain image feature set corresponding to each source domain tongue image sample and the target domain image feature set corresponding to the target domain tongue image sample further includes the following method:

[0032] Use the SWD distance, the L2 distance, and the cross-entropy of the class-conditional distribution difference as the cost function between the two feature sets, and calculate the optimal transport distance between the two feature sets. The specific cost function includes:

[0033]

[0034] λ swd represents the balance coefficient of the SWD distance, λ l2 represents the balance coefficient of the L2 distance, λ cond represents the balance coefficient of the cross-entropy of the class-conditional distribution difference. represents the label of the source domain tongue image sample [[ID=$$]] represents the difference in class-conditional distribution, M represents the total number of projection matrices, Z i represents mapping the source domain tongue image sample to the feature matrix obtained by mapping to the hidden layer space Z, Z j represents the target domain sample The feature matrix obtained by mapping to the hidden layer space Z represents projecting the source domain tongue image samples or the target domain tongue image samples onto the hidden layer space Z to form the corresponding m-th projection matrix.

[0035] Preferably, in the step S5, a mini-batch strategy is adopted, specifically including randomly extracting mini-batches of size n from each of the source domain tongue image samples and the target domain tongue image samples each time, and calculating the optimal transport between these two mini-batches as the optimal transport distance between domains: where OT n is a matrix representing the optimal transport distance between domains, is a matrix representing the composition of the source domain tongue image sample distribution, is a matrix representing the composition of the target domain tongue sample distribution, represents and 's joint distribution, γ n is an n×n matrix representing the optimal transport scheme between any source domain tongue image sample and any target domain tongue image sample with respect to the corresponding image feature set, C n is an n×n matrix representing the sample optimal transport distance between any source domain tongue image sample and any target domain image sample, <γ n ,C n > F represents γ n and C n 's Frobenius dot product.

[0036] Preferably, in the step S5, calculating the optimal transport distance between domains between the source domain and the target domain further includes adopting unbalanced optimal transport.

[0037] Preferably, in the step S5, calculating the optimal transport distance between domains between the source domain and the target domain further includes adopting unbalanced optimal transport loss and adding the classification cross-entropy loss function of the source domain.

[0038] Through the above technical solutions, it can be seen that compared with the prior art, the present invention discloses a cross-domain tongue image classification method based on deep hierarchical optimal transport, having the following beneficial effects:

[0039] While aligning tongue images with different distributions through deep hierarchical optimal transport and enhancing the classification ability, unbalanced optimal transport is adopted in the first-layer optimal transport for domain alignment, relaxing the marginal constraints of optimal transport, so as to provide a more robust optimization performance for small-batch training; for the second-layer optimal transport, SWD is used instead of EMD distance to enhance the discriminative features of samples. SWD is an approximation of EMD distance, but its computational cost is lower. The accuracy of tongue image classification is improved. Brief Description of the Drawings

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained according to the provided drawings.

[0041] Figure 1 Schematic diagram of the process of the deep hierarchical image classification method provided by the present invention;

[0042] Figure 2 Schematic diagram of the structure of the deep hierarchical optimal transport model provided by the present invention. Detailed Embodiments

[0043] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0044] The embodiment of the present invention discloses a cross-domain tongue image classification method based on deep hierarchical optimal transport, including:

[0045] A cross-domain tongue image classification method based on deep hierarchical optimal transport, including:

[0046] S1. Collect tongue image samples from multiple different domains as the training set;

[0047] S2. Use a deep neural network to extract features from the source domain tongue image samples and the target domain tongue image samples in the training set respectively, and obtain a source domain image feature map composed of the corresponding source domain tongue image sample features and a target domain image feature map composed of the target domain tongue image sample features;

[0048] That is, using a deep neural network to extract features from the source-domain tongue image samples in the training set, and obtaining a source-domain image feature map composed of the corresponding source-domain tongue image sample features;

[0049] Using a deep neural network to extract features from the target-domain tongue image samples in the training set, and obtaining a target-domain image feature map composed of the corresponding target-domain tongue image sample features;

[0050] S3. Divide the source-domain image feature map into blocks to obtain a source-domain image feature set corresponding to the source-domain tongue image samples; divide the target-domain image feature map into blocks to obtain a target-domain image feature set corresponding to the target-domain tongue image samples;

[0051] S4. Calculate the optimal transport distance between the source-domain image feature set corresponding to each source-domain tongue image sample and the target-domain image feature set corresponding to the target-domain tongue image sample, as the sample optimal transport distance between the source-domain tongue image sample and the target-domain image sample;

[0052] In the present invention, the two samples in the sample optimal transport distance are respectively from the source-domain tongue image sample and the target-domain image sample, so as to introduce local information while achieving domain alignment and maintaining the discriminability of its features;

[0053] S5. Taking the sample optimal transport distance as the cost metric between the source domain and the target domain, calculate the inter-domain optimal transport distance between the source domain and the target domain;

[0054] S6. Calculate the softmax cross-entropy loss according to the source-domain tongue image sample feature values extracted in step S2 as a part of the loss function; take the inter-domain optimal transport distance as another part in the loss function, construct a classification loss function, and use the classification loss function to train a classifier;

[0055] S7. Use the trained classifier to classify the tongue image samples to be verified.

[0056] Suppose and are two samples respectively from the source-domain distribution μ s and the target-domain distribution μ t , and Π(μ s , μ t ) is the joint probability distribution of the source-domain distribution μ s and the target-domain distribution μ t . Assume that the number of samples in the two domains are N s and N t respectively, C ≥ 0 and is the cost matrix between μ s , μ t , and each element is composed of The calculated cost is the cost between the two samples, which is used to measure the difference between the two samples. c is the cost function that measures the distance between the two samples, usually using L2 distance. As a cost metric, the optimal transmission distance between domains can be calculated. There are the following methods to calculate .

[0057] In one embodiment, the sample-optimized transfer distance between the source domain tongue image sample and the target domain image sample in step S4 may be a cost function between the source domain image feature set and the target domain image feature set using the EMD distance and the L2 distance:

[0058] First, we design a feature extractor g:x→z based on a deep neural network, which can map the input to a hidden space Z. At the same time, we design a classifier f:z→y, which can map the hidden space to the label space. The image x can be obtained by the feature extractor g. Then, the cost function between the source domain image feature set and the target domain image feature set can be transformed into:

[0059]

[0060] Where g represents the feature extractor of the deep neural network;

[0061] represents the source domain image feature map, represents the source domain tongue image sample, H i and W i Represent the width and height of the source domain image feature map respectively;

[0062] represents the target domain image feature map, represents the target domain tongue image sample, H j and W j Represent the width and height of the target domain image feature map respectively;

[0063] γ in represents the optimal transmission scheme between two samples regarding the image feature set, C in Represents the cost matrix between two samples with respect to the image feature set, γ in ∈R HiWi×HjWj ; C in ∈R HiWi×HjWj ; <γ in ,C in > F Represents γ in and C in Frobenius dot product; It represents the result of global average pooling of the source domain image feature map along the spatial dimension. It represents the result of global average pooling of the target domain image feature map along the spatial dimension. ch represents the number of channels.

[0064] The feature extractor can be implemented by the convolutional layer of a convolutional neural network.

[0065] To further optimize the above technical solution, in another embodiment, step S4 calculates the sample optimal transport distance between the source domain tongue image samples and the target domain image samples in the following manner:

[0066] Jointly use the SWD distance and the L2 distance as the cost function between the source domain image feature set and the target domain image feature set, and calculate the optimal transport distance between the two feature sets. The cost function specifically includes:

[0067]

[0068] P represents the permutation matrix. represents the set of all permutation matrices, U i represents transforming the features corresponding to the sample into a common high-dimensional hidden layer space, U j represents transforming the features corresponding to the sample into a common high-dimensional hidden layer space, and T is the matrix transpose symbol.

[0069] Within a mini-batch of size n, the formula (1) needs to be calculated once for each pair of samples between the source domain tongue image samples and the target domain image samples, and such calculation cost is still too high. Formula (2) uses the SWD distance (Sliced Wasserstein Distance) to approximate the calculation of the cost function between the source domain image feature set and the target domain image feature set:

[0070] Formula (2) introduces the permutation matrix P to match different regions of the images. P contains the associations between regions of the two images, and represents the set of all permutation matrices. represents transforming the features corresponding to the sample into a d-dimensional common hidden layer space.

[0071] To further optimize the above technical solution, in another embodiment, another way to calculate the sample optimal transport distance between the source domain tongue image samples and the target domain image samples in step S4 is as follows:

[0072] The matching problem in formula (2) is an NP-hard problem with a too high time complexity. To solve this problem more efficiently, formula (2) will be approximated by the following algorithm:

[0073]

[0074] Among them, contains M projections. Here, there is no need to explicitly obtain the permutation matrix P. Instead, only sort each projected region and then calculate their corresponding distances. In this way, the sample optimal transport distance between the source domain tongue image samples and the target domain image samples is approximately calculated through formula (3). Each calculation The computational complexity can be reduced from O(N 3 )(if solved by linear programming) or O(JN 2 )(if solved by the Sinkhorn scaling algorithm, J is the number of iterations of the Sinkhorn scaling algorithm) to O(MN), where N is the problem complexity. Since the feature map obtained here is based on the features extracted by the deep feature extractor g and the size of the obtained feature map is not large and the number of regions that can be segmented is limited, N will be relatively small and the computational cost will not increase too much.

[0075] The difference in marginal distributions here is measured by the inner SWD distance and the L2 distance, and the class-conditional distribution is measured by the entropy between sample labels. Since the target domain samples have no label information, the labels predicted by the model are used as proxies here. By jointly aligning the marginal distribution and the class-conditional distribution, more class information can be introduced and the discrimination between classes can be improved. Therefore, formula (3) will be converted to:

[0076]

[0077] Among them, λ swd represents the balance coefficient of the SWD distance, λ l2 represents the balance coefficient of the L2 distance, λ cond represents the balance coefficient of the cross-entropy of the difference in class-conditional distributions, M represents the total number of projection matrices, Z i represents the feature matrix obtained by mapping the source domain tongue image samples to the hidden layer space Z, Z j represents the feature matrix obtained by mapping the target domain tongue image samples to the hidden layer space Z, represents the m-th projection matrix formed by projecting the source domain tongue image samples or the target domain tongue image samples to the hidden layer space Z, represents the source domain tongue image samples Tags, ∈ R is the difference in class-conditional distributions.

[0078] Formula (4) contains three items. The first item is the source domain tongue image sample and target domain image samples Specifically, we will Characteristics of Z i and Characteristics of Z j Projection to multiple Z i and Z j Shared space, in each such shared space, Z i and Z j After sorting the feature subsets, directly calculate their Euclidean distance to obtain Z i and Z j The distance in the shared space. Finally, the Z calculated in multiple shared spaces i and Z j The average distance is taken as Z i and Z i The second term is The function adds a global average pooling operation compared to the g function, which is equivalent to adding Z i or Z j Perform global average pooling, so the second item is Z i The corresponding global average pooling results and Z j The corresponding global average pooling result calculates the L2 distance. The third item is to calculate The corresponding labels and The cross entropy between the labels for classification prediction represents the difference in class conditional distribution. Finally, formula (4) uses three hyperparameters as the balance coefficients of these three terms to weight and sum them.

[0079] It is worth noting that the three terms in formula (4) are complementary: SWD distance and L2 distance form local and global complementary information; SWD distance and L2 distance both calculate the difference in edge distribution; The difference in class conditional distribution is measured. By complementing each other, the three items can be used to maintain the distinguishability of categories while performing domain alignment, thereby improving classification performance.

[0080] In this implementation, the feature extractor uses ResNet-50. The feature maps extracted by feature extractor g maintain their spatial structure and are used to calculate the SWD distance. Feature extractor g is pre-trained on ImageNet, and classifier f is trained from scratch. Therefore, the learning rate of the classifier is 10 times that of feature extractor g. Here, several balancing coefficients in our loss function will be set to λswd = 0.001, λ l2 = 0.001 and λ cond = 1.0.

[0081] The optimizer in this implementation case uses the SGD optimizer, with the momentum set to 0.9. The learning rate change strategy is set to change linearly. The size of the small batch is 65, and it is iterated 10,000 times.

[0082] Step S5 is used to calculate the inter-domain optimal transport distance between the source domain and the target domain according to the sample-optimized transport distance as the cost metric between the source domain and the target domain;

[0083] Optimal Transport (OT) is a method for measuring the distance between two probability distributions, which can utilize the geometric structure of the distributions. Generally speaking, OT will search for possible coupling methods γ ∈ Π(μ s and μ t ), and find the coupling scheme with the minimum transport cost: s , μ t ), and find the coupling scheme with the minimum transport cost:

[0084]

[0085] where and are any two samples from the source domain distribution μ s and the target domain distribution μ t respectively; is the cost between two samples, used to measure the difference between two samples; Π(μ s , μ t ) is the joint probability distribution of the marginal distributions μ s and μ t . The discrete form of OT on the empirical distribution can be defined as:

[0086]

[0087] where μ s , μ t are positive vectors, <·,·> F is the Frobenius dot product. Assume that the number of samples in the two domains are N s and N t respectively, C ≥ 0 and is the cost matrix between μ s , μ t , and each element is composed of It is calculated. c is a cost function for measuring the distance between two samples, and usually the L2 distance is adopted. By optimizing formula (6), that is, minimizing the transportation cost, the optimal transportation flow can be obtained. Formula (6) can be solved by linear programming.

[0088] In one embodiment, in step S5, calculating the inter-domain optimal transportation distance between the source domain and the target domain includes adopting a mini-batch strategy, specifically including randomly extracting a mini-batch of size n from each domain each time, and calculating the optimal transportation between these two mini-batches as the proxy optimal transportation between the domains:

[0089]

[0090] Among them, Considering the calculation cost of formula (6), in the present invention, a mini-batch of size n is randomly extracted from each domain each time, and the optimal transportation between these two mini-batches is calculated as the proxy optimal transportation between the domains, that is, transforming formula (6) into formula (7) C n Each element in is calculated by equation (6), thus forming a hierarchical optimal transportation model.

[0091] As an improved technical solution, in another embodiment, in step S5, another way to calculate the inter-domain optimal transportation distance between the source domain and the target domain is to adopt a mini-batch-based unbalanced optimal transportation method to replace formula (7):

[0092]

[0093] Among them, D φ is Csiszar Divergences, KL is Kullback-Leibler divergence, and is the marginal distribution of γ n Here, τ is the marginal penalty coefficient (Marginal Penalization), ε is the regularization coefficient (Regularization Coefficient), ε≥0, and specifically it can be set as ε = 0.01 and τ = 0.5.

[0094] In this way, in each mini-batch, each domain is a set of samples. Equation (8) is the optimal transportation between the source domain and the target domain at the first layer, and the cost matrix C in equation (8) nEach element in is calculated (4) from the corresponding source-domain tongue image sample and the target-domain image sample. Therefore, the second layer is the optimal transport between the source-domain tongue image sample and the target-domain image sample, where each sample is a collection of spatial regions of the image feature map. Such two-layer optimal transport constitutes a deep hierarchical optimal transport model (Deep Hierarchical Optimal Transport, DeepHOT), as Figure 2 shown. Thus, for a given mini-batch, the objective problem of DeepHOT is:

[0095]

[0096] where

[0097]

[0098] As an improved technical solution, in another embodiment, in step S5, another method for calculating the optimal transport distance between the source domain and the target domain is to use the unbalanced optimal transport loss and add the classification cross-entropy loss function of the source domain:

[0099]

[0100] The purpose is to avoid the "catastrophic forgetting" problem on the source domain. The final optimization objective includes the classification cross-entropy loss L of the source domain.

[0101] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the various embodiments, reference can be made to each other. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple. For the relevant parts, reference can be made to the description in the method section.

[0102] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be obvious to those skilled in the art. The general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but will be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A cross-domain tongue image classification method based on deep hierarchical optimal transmission, characterized by: The method comprises the following steps: S1. Collect tongue image samples from multiple different fields as training sets; S2. Using a deep neural network to extract features from the source domain tongue image samples in the training set, and obtaining a source domain image feature map composed of features of the corresponding source domain tongue image samples; Using a deep neural network to extract features from target domain tongue image samples in the training set, obtaining a target domain image feature map composed of corresponding target domain tongue image sample features; S3. Divide the source domain tongue image sample features in the source domain image feature map into blocks to obtain a source domain image feature set corresponding to the source domain tongue image sample; Divide the target domain tongue image sample features in the target domain image feature map into blocks to obtain a target domain image feature set corresponding to the target domain tongue image sample; S4. Calculate the optimal transmission distance between the source domain image feature set corresponding to each source domain tongue image sample and the target domain image feature set corresponding to the target domain tongue image sample as the sample optimal transmission distance between the source domain tongue image sample and the target domain image sample; specifically, including: The SWD distance and L2 distance are jointly used as the cost function between the source domain image feature set and the target domain image feature set to calculate the optimal transmission distance between the two feature sets. The cost function specifically includes: ； Represents the i-th source domain tongue image sample Extracted source domain image feature map; Represents the jth target domain tongue image sample The extracted target domain image features; g represents the feature extractor of the deep neural network; P represents the permutation matrix, represents the set of all permutation matrices, U i Indicates that the source domain tongue image sample The corresponding features are transformed into a common high-dimensional hidden space, U j Indicates that the target domain tongue image sample The corresponding features are transformed into a common high-dimensional hidden space, where T is the matrix transpose symbol; It represents the global average pooling result of the source domain image feature map extracted from the i-th source domain tongue image sample along the spatial dimension; represents the global average pooling result of the target domain image feature map extracted from the j-th target domain tongue image sample along the spatial dimension; S5. Calculate the inter-domain optimized transmission distance between the source domain and the target domain using the sample optimized transmission distance as a cost metric between the source domain and the target domain; S6. Calculate the softmax cross entropy loss based on the source domain tongue image sample feature values extracted in step S2 as part of the loss function; use the inter-domain optimized transmission distance as another part of the loss function to construct a classification loss function, and train the classifier using the classification loss function; S7. Use the trained classifier to classify the tongue image samples to be verified.

2. The cross-domain tongue image classification method based on deep hierarchical optimal transmission according to claim 1 is characterized in that: The step S4 of calculating the optimal transmission distance between the source domain image feature set corresponding to each source domain tongue image sample and the target domain image feature set corresponding to the target domain tongue image sample comprises the following method: The EMD distance and L2 distance are used together as the cost function between the source domain image feature set and the target domain image feature set to calculate the optimal transmission distance between the two feature sets. The cost function specifically includes: ； Where g represents the feature extractor of the deep neural network; represents the source domain image feature map extracted from the i-th source domain tongue image sample, represents the i-th source domain tongue image sample, , H i and W i Respectively represent the width and height of the source domain image feature map extracted from the i-th source domain tongue image sample; represents the target domain image feature map extracted from the j-th target domain tongue image sample, represents the jth target domain tongue image sample, , H j and W j They represent the width and height of the target domain image feature map extracted from the j-th target domain tongue image sample; A joint image feature map representing the meta-domain image feature map and the target domain image feature map; γ in represents the optimal transmission scheme between any source domain tongue image sample and any target domain tongue image sample regarding the corresponding image feature set, C in Represents the cost matrix between any source domain tongue image sample and any target domain tongue image sample with respect to the corresponding image feature set; Represents γ in and C in Frobenius dot product; It represents the global average pooling result of the source domain image feature map extracted from the i-th source domain tongue image sample along the spatial dimension. ; It represents the global average pooling result of the target domain image feature map extracted from the j-th target domain tongue image sample along the spatial dimension. ;ch represents the channel number.

3. The cross-domain tongue image classification method based on deep hierarchical optimal transmission according to claim 1 is characterized in that: The step S4 of calculating the optimized transmission distance between the source domain image feature set corresponding to each source domain tongue image sample and the target domain image feature set corresponding to the target domain tongue image sample further includes the following method: The SWD distance, L2 distance, and cross entropy of class conditional distribution difference are used as the cost function between the two feature sets to calculate the optimal transmission distance between the two feature sets. The cost function specifically includes: ； λ swd The balance coefficient of SWD distance, λ l2 represents the balance coefficient of L2 distance, λ cond The balance coefficient of the cross entropy representing the difference in class-conditional distributions, Represents a tongue image sample in the source domain Tags, Represents the difference in class conditional distribution, M represents the total number of projection matrices, Z i Indicates that the source domain tongue image sample The feature matrix obtained by mapping to the hidden space Z, Z j Indicates that the target domain tongue image sample The feature matrix obtained by mapping to the hidden space Z is: Indicates that the source domain tongue image sample Or target domain tongue image sample Projected to the hidden space Z to form the corresponding m-th projection matrix.

4. The cross-domain tongue image classification method based on deep hierarchical optimal transmission according to claim 3 is characterized in that: In step S5, the inter-domain optimized transmission distance between the source domain and the target domain is calculated, including adopting a mini-batch strategy, specifically including: Each time, a mini-batch of size n is randomly extracted from each source domain tongue image sample and target domain tongue image sample, and the optimal transmission between the two mini-batches is calculated as the optimal transmission distance between the domains: ； in, , , OT n The matrix representing the optimal transmission distance between domains, Represents the matrix composed of the distribution of tongue image samples in the source domain, Represents the matrix composed of tongue sample distribution in the target domain, express and The joint distribution of γ n represents the n*n matrix composed of the optimal transmission scheme for the corresponding image feature set between any source domain tongue image sample and any target domain tongue image sample, C n represents an n*n matrix composed of the sample optimized transmission distance between any source domain tongue image sample and any target domain image sample, Represents γ n and C n Frobenius dot product.

5. The cross-domain tongue image classification method based on deep hierarchical optimal transmission according to claim 4 is characterized in that: In step S5, calculating the inter-domain optimized transmission distance between the source domain and the target domain further includes adopting unbalanced optimal transmission.

6. The cross-domain tongue image classification method based on deep hierarchical optimal transmission according to claim 1 is characterized in that: In step S5, calculating the inter-domain optimized transmission distance between the source domain and the target domain further includes using an unbalanced optimal transmission loss and then adding a classification cross entropy loss function of the source domain.

Images