Multi-source generalization method based on Markov classifier

Abstract

The invention discloses a multi-source generalization method based on a Markov classifier. The method comprises the following steps: acquiring image data of a plurality of source domains; according to the image data of the plurality of source domains, a training data set is obtained, one image in the image data of each source domain serves as a sample image, and the training data comprises the sample image and a category label corresponding to the sample image; an image detection model is constructed, the model comprises a feature extractor and a generation classifier which are connected in sequence, the feature extractor adopts a convolutional neural network, and the generation classifier adopts a Mahalanobis classifier based on Mahalanobis distance; constructing a target detection loss function; training an image detection model by using the training data set based on the target detection loss function; and obtaining a to-be-classified image, inputting the to-be-classified image into the trained image detection model, and obtaining and outputting a category corresponding to the to-be-classified image. According to the method, cross-domain class change can be fully captured, image data of a transition fitting source domain is avoided, and the classification precision of the model on different source domain images is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a multi-source generalization method based on a Mahalanobis classifier. Background Art

[0002] In recent years, deep learning methods represented by deep neural networks have been widely used in fields such as computer vision and remote sensing image processing, and have achieved good results in tasks such as image classification, semantic segmentation, and target detection.

[0003] In the field of image processing, images obtained from different sources are called multi-source images. Different sources include but are not limited to cameras, satellites, geographic information systems, computer technology, etc. Since these sources have different properties and characteristics, the images obtained from them are also different. For image classification, current deep learning methods are usually supervised and require a large amount of labeled training data to fit the characteristics of the training set. However, among multi-source images, some source images are easy to obtain and sufficient labeled training data can be obtained, while some source images are difficult to obtain and it is difficult to obtain sufficient labeled training data. Therefore, how to extract effective information with generalization ability from other source domain image data with sufficient labels so that the image detection model can achieve good detection and classification performance in the target source domain without labels is an important technical issue in the current image processing field.

[0004] The ERM algorithm in the field of Multi-Source Domain Generalization (MSDG) is a machine learning method used to solve the problems of data sparsity and high dimensionality. The ERM algorithm achieves generalized learning of target data by fusing information from multiple data sources. The ERM algorithm mainly consists of two parts: a feature extractor and a model trainer. The feature extractor is responsible for extracting useful feature information from multiple data sources; the model trainer uses this feature information to train a linear classifier. In the field of multi-source generalization, the ERM algorithm is also applied to the field of image recognition. However, when using the ERM algorithm for multi-source image detection and recognition, it is easy to cause the model to over-fit the source domain data, which in turn leads to poor detection performance of the trained image detection model on the target source domain. Summary of the invention

[0005] In order to solve part or all of the technical problems existing in the above-mentioned prior art, the present invention provides a multi-source generalization method based on Mahalanobis classifier.

[0006] The technical solution of the present invention is as follows:

[0007] A multi-source generalization method based on a Mahalanobis classifier is provided, the method comprising:

[0008] Acquire image data from multiple source domains;

[0009] According to the image data of the plurality of source domains, a training data set is obtained, wherein one image in the image data of each source domain is used as a sample image, and the training data in the training data set includes the sample image and its corresponding category label;

[0010] Construct an image detection model, which includes a feature extractor and a generative classifier connected in sequence. The feature extractor uses a convolutional neural network, and the generative classifier uses a Mahalanobis classifier based on Mahalanobis distance.

[0011] Based on the definition of Mahalanobis distance, an object detection loss function is constructed for training image detection models.

[0012] Based on the target detection loss function, the image detection model is trained using the training data set;

[0013] Obtain the image to be classified, input the image to be classified into the trained image detection model, and obtain and output the category corresponding to the image to be classified.

[0014] In some possible implementations, it is assumed that: the feature extracted by the feature extractor from the image is z, the class label corresponding to the image is y, and each class conforms to a multivariate Gaussian distribution;

[0015] The Mahalanobis distance is defined as:

[0016]

[0017] in, represents the Mahalanobis distance of feature z on the jth class, μ j Represents the multivariate Gaussian distribution that the jth class conforms to The mean value of j Represents the multivariate Gaussian distribution that the jth class conforms to The covariance of , the superscript T indicates the transpose, Represents ∑ j The inverse matrix of .

[0018] In some possible implementations, the target detection loss function is:

[0019]

[0020] Among them, θ represents the parameters of the feature extractor, λ1 represents The corresponding weight, λ2 represents The corresponding weight, λ3 represents The corresponding weights, represents the discriminant loss function based on Mahalanobis distance, represents the intra-class loss function used to minimize the modeled multivariate Gaussian distribution, represents the center loss function, represents the inter-class difference loss function used to maximize the inter-class difference.

[0021] In some possible implementations, N source domain image data are set, each source domain image data includes multiple images, an image and its corresponding category label constitute a training data {x, y}, multiple images of each source domain and their corresponding category labels constitute multiple training data, x represents a sample image, y represents the category label corresponding to the sample image, y∈[1,…,C], C represents the number of categories, and N source domain image data constitute a training data set D i represents the training data subset corresponding to the i-th source domain image data;

[0022] Discriminative loss function It is expressed as:

[0023]

[0024] Among them, F θ (x) represents the feature extractor F in the image detection model θ Based on the features extracted from the input sample image x, is the Mahalanobis distance of feature z on the jth class.

[0025] In some possible implementations, the intra-class loss function It is expressed as:

[0026]

[0027] Here, e represents a natural constant.

[0028] In some possible implementations, the center loss function is It is expressed as:

[0029]

[0030] in, represents the cache average of the jth class, and ||·||2 represents the L2 norm.

[0031] In some possible implementations, the inter-class difference loss function is It is expressed as:

[0032]

[0033] in,

[0034] In some possible implementations, training the image detection model using the training data set includes:

[0035] The sample images of the training data in the training data set are taken as input, and the category labels corresponding to the input sample images are taken as output to train the image detection model.

[0036] In some possible implementations, taking a sample image of training data in a training data set as input and taking a category label corresponding to the input sample image as output, training an image detection model further includes the following steps:

[0037] Step S51, inputting sample images of multiple training data in the training data set into the image detection model in sequence, and obtaining features corresponding to the sample images output by the feature extractor in the image detection model;

[0038] Step S52, calculating the target detection loss function according to the category label corresponding to the sample image and the features output by the feature extractor;

[0039] Step S53, determine whether the preset training stop condition is reached. If so, use the current image detection model as the trained image detection model. If not, use the target detection loss function to update the parameters of the image detection model and return to step S51.

[0040] The main advantages of the technical solution of the present invention are as follows:

[0041] The multi-source generalization method based on the Mahalanobis classifier of the present invention can fully capture cross-domain class changes, avoid over-fitting the image data of the source domain, and significantly improve the classification accuracy of the image detection model on images in different source domains by modeling the conditional distribution of features as a multivariate Gaussian distribution and using a classifier based on the Mahalanobis distance to distinguish categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The drawings described herein are used to provide a further understanding of the embodiments of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0043] Figure 1 The present invention is a flowchart of a multi-source generalization method based on a Mahalanobis classifier according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0045] The technical solution provided by the embodiments of the present invention is described in detail below with reference to the accompanying drawings.

[0046] refer to Figure 1 An embodiment of the present invention provides a multi-source generalization method based on a Mahalanobis classifier, the method comprising the following steps S1-S6:

[0047] Step S1, obtaining image data of multiple source domains.

[0048] Specifically, according to actual conditions, image data is acquired from a source domain where data can be easily acquired.

[0049] In an embodiment of the present invention, for different source domains, image data including multiple images may be acquired through various approaches.

[0050] Step S2: acquiring a training data set based on image data from multiple source domains.

[0051] In one embodiment of the present invention, the training data in the training data set includes sample images and their corresponding category labels. Among them, one image in each source domain image data obtained is used as a sample image, and based on the image data obtained in multiple source domains, the category label of each image is determined and annotated to obtain the corresponding training data, thereby obtaining the training data set.

[0052] Step S3, constructing an image detection model, the image detection model includes a feature extractor and a generative classifier connected in sequence, the feature extractor adopts a convolutional neural network, and the generative classifier adopts a Mahalanobis classifier based on Mahalanobis distance.

[0053] In one embodiment of the present invention, an image detection model is trained on image data of multiple source domains, and then the trained image detection model is used to perform image detection, classification and recognition on a target source domain.

[0054] In one embodiment of the present invention, in order to improve the classification accuracy, the image detection model includes a feature extractor F connected in sequence θ :x→z and generate a classifier G μ,∑:z→y, the feature extractor adopts a convolutional neural network parameterized by θ, and the generated classifier adopts a Mahalanobis classifier based on Mahalanobis distance, which is parameterized by the class mean and covariance estimated from the training data. The feature extractor is used to extract and output the features in the input image, and the Mahalanobis classifier is used to calculate the Mahalanobis distance of the features on each class, and the class corresponding to the minimum Mahalanobis distance is output as the class of the input image.

[0055] Step S4, based on the definition of Mahalanobis distance, construct an object detection loss function for training the image detection model.

[0056] Specifically, it is assumed that the feature extractor extracts the feature from the input image as z, z∈R H , the category label corresponding to the input image is recorded as y, y∈[1,…,C], R H represents an H-dimensional real vector, C represents the number of categories, and C is a positive integer.

[0057] Assumption: Each class j (j is a positive integer) conforms to a multivariate Gaussian distribution The mean value is μ j , μ j ∈R H , the covariance is Σ j ,Σ j ∈R H×H , the conditional probability that feature z exists in the jth class is P(z|y=j;μ j ,∑ j ).

[0058] According to the Bayesian formula, the posterior probability that feature z will be classified as class j is:

[0059]

[0060] As a multivariate Gaussian distribution P(z|y=j;μ j ,∑ j ), we can get:

[0061]

[0062] Without changing monotonicity, we can obtain the following logarithmic posterior:

[0063]

[0064] Among them, P(y=j) represents the probability of the category being the jth category, P(z) represents the probability of the existence of feature z, the superscript T represents the transpose, and ζ is used to represent the items in the model that are not directly related to the classification of feature z.

[0065] According to the above formula, we can see that the logarithmic posterior of feature z for a given class y=j is calculated as input z and mean μ j , the precision matrix (inverse matrix of covariance) is The negative Mahalanobis distance between class j.

[0066] Furthermore, for the problem of multi-source image generalization, N source domain image data are set, each source domain image data includes multiple images, an image and its corresponding category label constitute a training data {x, y}, and multiple images in each source domain and their corresponding category labels constitute multiple training data, x represents a sample image for inputting the image detection model, x∈R H×W×K , H, W and K represent the height, width and number of channels of the sample image respectively, y represents the category label corresponding to the sample image, y∈[1,…,C], and N source domain image data constitute the training data set D i Represents the training data subset corresponding to the i-th source domain image data.

[0067] In one embodiment of the present invention, since the Mahalanobis classifier is parameterized by the class mean and covariance estimated from the training data; based on the above settings, for the image detection model, the following discriminant loss function is constructed based on the Mahalanobis distance:

[0068] Among them, F θ (x) represents the feature extractor F in the image detection model θ Based on the features extracted from the input sample image x, is the Mahalanobis distance of feature z on the jth class.

[0069] In one embodiment of the present invention, the Mahalanobis classifier is used in the model inference process. In the above-constructed discriminant loss function, the Mahalanobis distance needs to be calculated. The distance metric requires the mean and covariance of the multivariate Gaussian distribution of each class modeling. However, in the end-to-end iterative training of the model, since the training data set is usually large, it means that the cost of calculating the statistical moments of the features extracted from the entire training data set is very high, and during training, the feature statistics are dynamically evolving, making the mean and covariance difficult to capture.

[0070] To this end, in one embodiment of the present invention, an exponential moving average is used to cache the estimated mean and estimated covariance corresponding to each batch of training data during the training iteration. Specifically, the estimated mean and estimated covariance of the current training step are calculated based on the mean and covariance corresponding to the training data input in the current training step, and the estimated mean and estimated covariance calculated in the previous training step, and the estimated mean and estimated covariance of the current training step are used as the mean and covariance of the current training step.

[0071] Specifically, the estimated mean and estimated covariance are calculated using the following formula:

[0072]

[0073] in, represents the estimated mean of the jth class at the tth training step, represents the estimated mean of the jth class at the t-1th training step, represents the mean of the jth class obtained based on the input training data at the tth training step, α represents the hyperparameter of the exponential moving average, represents the estimated covariance of the jth class at the tth training step, represents the estimated covariance of the jth class at the t-1th training step, Represents the covariance of the jth class obtained based on the input training data at the tth training step.

[0074] Set the training data input for the tth training step to (x t ,y t ),but and Respectively expressed as:

[0075]

[0076]

[0077] Among them, Mean represents the mean, F θ (x t ) represents the feature extractor F θ Based on the input sample image x t The extracted features, Indicates that according to the extracted feature F θ (x t ) is the mean of the statistic, Cov represents the covariance, Indicates that according to the extracted feature F θ (x t ) statistic’s covariance.

[0078] In one embodiment of the present invention, at the tth training step, the obtained and As μ j and∑ j , to calculate the Mahalanobis distance.

[0079] Furthermore, in the multi-source image generalization problem, since image data of the same category contains information from different source domains, the intra-class variation is large, and large intra-class variation will damage the robustness of the model. In order to achieve domain invariance, it is necessary to minimize the intra-class variation of learned features.

[0080] In one embodiment of the present invention, since each class is modeled as a multivariate Gaussian distribution, the entropy of the multivariate Gaussian distribution can indicate its change, so the following intra-class loss function is established: Multivariate Gaussian distribution modeled by minimization Entropy of:

[0081]

[0082] Where e represents a natural constant. The intra-class loss function formula also matches the logarithmic determinant term in the above logarithmic posterior calculation formula.

[0083] Furthermore, since the center loss function can effectively minimize intra-class variations, in one embodiment of the present invention, the center loss is also used to optimize the model.

[0084] Specifically, the following center loss function is constructed

[0085]

[0086] in, represents the cache average of the jth class, and ||·||2 represents the L2 norm. Calculation method and estimated mean The calculation method is the same. Taking the tth training step as an example,

[0087] In one embodiment of the present invention, although the defined intra-class loss function and the center loss function have some overlapping effects on the optimization model, by combining the two, the intra-class compactness can be improved from different aspects, making the features of the model in the same class more concentrated, which is helpful for the convergence of the model and improving the final performance of the model.

[0088] Furthermore, in one embodiment of the present invention, since the source domain to which the data used to train the model belongs is different from the target source domain, in order to enhance the classification performance of the trained model in the target source domain, maximizing the inter-class difference is also considered.

[0089] Specifically, in one embodiment of the present invention, since each class is modeled as a multivariate Gaussian distribution, maximizing the inter-class difference between the i-th class and the j-th class can be achieved by maximizing their probability distribution and At the same time, based on this, the use of probability product kernel (PPK) can achieve better performance.

[0090] To this end, in one embodiment of the present invention, based on the above-defined inter-class differences, the PPK distance function is used to construct the following inter-class difference loss function:

[0091]

[0092] in, Indicates that the mean is μ i -μ j , the covariance is Σ i +Σ j Gaussian distribution.

[0093] Further, based on the above analysis process, in one embodiment of the present invention, the target detection loss function for training the image detection model is constructed as:

[0094]

[0095] Among them, θ represents the parameters of the feature extractor, that is, the parameters of the convolutional neural network, and λ1 represents The corresponding weight, λ2 represents The corresponding weight, λ3 represents The corresponding weight.

[0096] Step S5: Based on the target detection loss function, the image detection model is trained using the training data set.

[0097] Specifically, in one embodiment of the present invention, using a training data set to train an image detection model includes:

[0098] The sample images of the training data in the training data set are taken as input, and the category labels corresponding to the input sample images are taken as output to train the image detection model.

[0099] In one embodiment of the present invention, a sample image of training data in a training data set is used as input, and a category label corresponding to the input sample image is used as output to train an image detection model, further comprising the following steps:

[0100] Step S51, inputting sample images of multiple training data in the training data set into the image detection model in sequence, and obtaining features corresponding to the sample images output by the feature extractor in the image detection model.

[0101] In one embodiment of the present invention, a sample image of training data is input from the input layer of an image detection model, processed in sequence by each layer of a feature extractor in the image detection model, and output from the output layer of the feature extractor. The information output by the output layer of the feature extractor is the feature corresponding to the sample image.

[0102] Among them, the image detection model is the model to be trained, and the parameters of each layer of the model are initialization parameters. During the training process of the model, the parameters of each layer of the model are continuously updated.

[0103] Step S52, calculating the target detection loss function according to the category label corresponding to the sample image and the features output by the feature extractor.

[0104] In one embodiment of the present invention, based on the target detection loss function set above, the target detection loss function is calculated according to the category label corresponding to each input sample image and its corresponding feature output by the feature extractor in the image detection model.

[0105] Step S53, determine whether the preset training stop condition is reached. If so, use the current image detection model as the trained image detection model. If not, use the target detection loss function to update the parameters of the image detection model and return to step S51.

[0106] In one embodiment of the present invention, the training stop condition is specifically set according to the actual situation, for example, the number of training iterations reaches a set iteration number or the optimization index reaches a set threshold value, for example, the target detection loss function can be used as the optimization index. If the preset training stop condition is met, the current image detection model is used as the image detection model that has completed the training. If the preset training stop condition is not met, the parameters of the image detection model are updated using the target detection loss function, specifically the parameters of the feature extractor are updated, and the process returns to step S51.

[0107] In one embodiment of the present invention, a stochastic gradient descent method is used to train and update the parameters of the image detection model.

[0108] Specifically, the following formula is used to update the parameters of the image detection model:

[0109]

[0110] Among them, θ t+1 represents the parameters of the model at the t+1th iteration, θ t represents the parameters of the model at the tth iteration, Δ[] represents the optimizer, η represents the learning rate, L represents the target detection loss function, and θ represents the parameters of the model.

[0111] Step S6, obtaining the image to be classified, inputting the image to be classified into the trained image detection model, and obtaining and outputting the category corresponding to the image to be classified.

[0112] Specifically, according to the actual image recognition requirements, the image to be classified is obtained, and the image to be classified is input into the trained image detection model. The feature extractor in the image detection model outputs the features corresponding to the image to be classified. The Mahalanobis classifier in the image detection model is based on the features output by the feature extractor. The Mahalanobis distance of the feature on each class is calculated, and the class corresponding to the smallest Mahalanobis distance is output as the category of the image to be classified.

[0113] The multi-source generalization method based on the Mahalanobis classifier provided in one embodiment of the present invention can fully capture cross-domain class changes, avoid over-fitting of image data in the source domain, and significantly improve the classification accuracy of the image detection model on images in different source domains by modeling the conditional distribution of features as a multivariate Gaussian distribution and using a classifier based on the Mahalanobis distance to distinguish categories.

[0114] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In addition, "front", "back", "left", "right", "upper" and "lower" in this article are all referenced to the placement state shown in the accompanying drawings.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-source generalization method based on Mahalanobis classifier, characterized in that: include: Acquire image data from multiple source domains; According to the image data of multiple source domains, a training data set is obtained, wherein one image in the image data of each source domain is used as a sample image, and the training data in the training data set includes the sample image and its corresponding category label; Construct an image detection model, which includes a feature extractor and a generative classifier connected in sequence. The feature extractor uses a convolutional neural network, and the generative classifier uses a Mahalanobis classifier based on Mahalanobis distance. Based on the definition of Mahalanobis distance, an object detection loss function is constructed for training image detection models. Based on the target detection loss function, the image detection model is trained using the training data set; Obtain the image to be classified, input the image to be classified into the trained image detection model, and obtain and output the category corresponding to the image to be classified.

2. The multi-source generalization method based on Mahalanobis classifier according to claim 1, characterized in that: Setting: The feature extracted by the feature extractor from the image is z, the class label corresponding to the image is y, and each class conforms to a multivariate Gaussian distribution; The Mahalanobis distance is defined as: in, represents the Mahalanobis distance of feature z on the jth class, μ j Represents the multivariate Gaussian distribution that the jth class conforms to The mean value, Σ j Represents the multivariate Gaussian distribution that the jth class conforms to The covariance of , the superscript T indicates the transpose, Represents ∑ j The inverse matrix of .

3. The multi-source generalization method based on Mahalanobis classifier according to claim 2, characterized in that: The target detection loss function is: Among them, θ represents the parameters of the feature extractor, λ1 represents The corresponding weight, λ2 represents The corresponding weight, λ3 represents The corresponding weights, represents the discriminant loss function based on Mahalanobis distance, represents the intra-class loss function used to minimize the modeled multivariate Gaussian distribution, represents the center loss function, represents the inter-class difference loss function used to maximize the inter-class difference.

4. The multi-source generalization method based on Mahalanobis classifier according to claim 3, characterized in that: Suppose there are N source domain image data, each source domain image data includes multiple images, an image and its corresponding category label constitute a training data {x, y}, multiple images of each source domain and their corresponding category labels constitute multiple training data, x represents the sample image, y represents the category label corresponding to the sample image, y∈[1,…,C], C represents the number of categories, and N source domain image data constitute the training data set D i represents the training data subset corresponding to the i-th source domain image data; Discriminative loss function It is expressed as: Among them, F θ (x) represents the feature extractor F in the image detection model θ Based on the features extracted from the input sample image x, F θ (x) = z, is the Mahalanobis distance of feature z on the jth class.

5. The multi-source generalization method based on Mahalanobis classifier according to claim 4, characterized in that: Intra-class loss function It is expressed as: Here, e represents a natural constant.

6. The multi-source generalization method based on Mahalanobis classifier according to claim 5, characterized in that: Center loss function It is expressed as: in, represents the cache average of the jth class, and ||·||2 represents the L2 norm.

7. The multi-source generalization method based on Mahalanobis classifier according to claim 6, characterized in that: Inter-class difference loss function It is expressed as: in 8. The multi-source generalization method based on Mahalanobis classifier according to any one of claims 1 to 7, characterized in that: Training an image detection model using a training data set includes: The sample images of the training data in the training data set are taken as input, and the category labels corresponding to the input sample images are taken as output to train the image detection model.

9. The multi-source generalization method based on Mahalanobis classifier according to claim 8, characterized in that: Taking a sample image of training data in a training data set as input and taking a category label corresponding to the input sample image as output, training an image detection model further includes the following steps: Step S51, inputting sample images of multiple training data in the training data set into the image detection model in sequence, and obtaining features corresponding to the sample images output by the feature extractor in the image detection model; Step S52, calculating the target detection loss function according to the category label corresponding to the sample image and the features output by the feature extractor; Step S53, determine whether the preset training stop condition is reached. If so, use the current image detection model as the trained image detection model. If not, use the target detection loss function to update the parameters of the image detection model and return to step S51.

Images