A method for characterizing and correcting the conditional distribution deviation between two sets of data

Abstract

A method for characterizing and correcting the conditional distribution deviation between two sets of data. In view of the situation where there is a complex conditional distribution deviation between the two sets of data, the difficult-to-solve problem of complex distribution deviation in the original feature space is converted into the solution of simple distribution deviation measurement indicators in multiple subspaces. The method first divides the feature space where the two sets of data are located into multiple subspaces, and then uses the membership function of the data points in the feature space to each subspace, as well as the conditional distribution deviation measurement indicator of each subspace, to calculate the conditional distribution deviation function between the two sets of data. The main purpose of the present invention is to characterize the distribution deviation between two sets of data, and the characterization results can be used to correct distribution deviations, guide data sampling, guide transfer learning, etc.

Description

Technical Field

[0001] The present invention relates to a big data technology, in particular to a machine learning technology, specifically a method for characterizing and correcting the conditional distribution deviation between two groups of data. Background Art

[0002] In recent years, machine learning has been widely used in various fields, such as computer vision, natural language processing, fault diagnosis, system state prediction, etc. However, the basic assumption for the effective operation of machine learning methods is that the training data and test data must have the same data distribution. But in actual application scenarios, many data do not meet the assumption of the same distribution, especially there are conditional distribution deviations in the data of many regression problems, such as the distribution of house prices in different regions, the distribution of the remaining life of processing equipment under different working conditions, and the distribution of airport passenger flow in different seasons. When there is a conditional distribution deviation between the data, the traditional machine learning algorithm cannot run effectively, and the deviation between the two sets of data needs to be corrected. The premise of the correction is to effectively characterize the deviation.

[0003] Patent CN111626351A invented a method and system for obtaining the concept drift of data distribution. The method can give a single numerical index describing the distribution deviation between two sets of data. According to the size relationship between the index and a given threshold, it is used to determine whether the trained machine learning model is suitable for a new scene. The method can comprehensively express the distribution deviation between the two sets of data, which has guiding significance for the practical application of machine learning models. However, the conditional distribution deviation between the two sets of data often has a complex functional relationship with the data feature information. Using a single numerical index to describe it can only play a general summary role, and it is impossible to fully characterize the distribution deviation between the two sets of data. For example, in the task of predicting height based on age information, there are significant conditional distribution differences between Asians and Europeans, that is, the distribution of height under the same age condition is significantly different. However, this difference is constantly changing with the change of age (feature information). A single numerical index cannot describe the complex conditional distribution difference relationship. Therefore, a method for characterizing the conditional distribution deviation between the two sets of data is needed, which can fully characterize the relationship between the conditional distribution deviation and the feature information, and is used to correct the distribution deviation, guide data sampling, guide transfer learning, etc. Summary of the invention

[0004] The purpose of the present invention is to address the problem that a single numerical indicator cannot describe the complex conditional distribution difference relationship. A method for characterizing and correcting the conditional distribution deviation between two sets of data is invented to characterize the relationship between the conditional distribution deviation between the two sets of data and the change of feature information. The characterization result can be used to guide the correction of distribution deviation, data sampling, and transfer learning, etc.

[0005] One of the technical solutions of the present invention is:

[0006] A method for characterizing the conditional distribution deviation between two sets of data, characterized in that it comprises the following steps:

[0007] First, obtain the first set of sample data and the second set of sample data, where each sample in the two sets of data consists of a feature x and a label y;

[0008] Secondly, the common feature space where the features of the first group of sample data and the features of the second group of sample data are located is divided into multiple subspaces, and the membership function of the data points in the feature space to each subspace is obtained;

[0009] Third, calculate the conditional distribution deviation measure of the two sets of data in each subspace;

[0010] Finally, the conditional distribution deviation function between the two sets of data is calculated using the membership function of the data points in the feature space to each subspace and the conditional distribution deviation measure of the two sets of data in each subspace.

[0011] The feature x is feature information describing the sample data, and the information type is scalar data, vector data, tensor data, image data, text data or other user-specified data; the label y is label information describing the sample data, and the information type is scalar data, vector data, tensor data, image data, text data or other user-specified data. The common feature space acquisition method is one of the following:

[0012] Determine the common feature space according to the distribution range of the features of the two groups of sample data;

[0013] Determine the common feature space based on the task characteristics faced by the sample data;

[0014] At the same time, the common feature space is determined according to the distribution range of the sample data characteristics and the task characteristics faced by the sample data; the feature space division includes two types: hard division and soft division. The hard division method includes decision tree or K-means clustering, and the soft division method includes Gaussian mixture model or fuzzy C-means.

[0015] The membership function is used to represent the degree of membership of a data point in the feature space to each subspace, and the membership function calculation method is:

[0016] When the feature space is hard-divided into multiple subspaces, the feature point uniquely belongs to a subspace, and the data point x in the feature space i The membership function belonging to the kth subspace is expressed as:

[0017]

[0018] When the feature space is soft-divided into multiple subspaces, the feature point has a different degree of membership to each subspace, and the membership function is a Gaussian membership function, a triangular membership function, or a Gamma membership function.

[0019] The subspace conditional distribution deviation measurement index is used to approximately represent the conditional distribution deviation between the first sample data and the second sample data in each subspace, and the specific form of the subspace conditional distribution deviation measurement index includes a single numerical form or a probability density function form;

[0020] The calculation method of the subspace conditional distribution deviation measurement index includes: solving by kernel mean difference; or solving by KL divergence; or solving by Euclidean distance.

[0021] The specific method for calculating the conditional distribution deviation measurement index of two sets of data in each subspace includes:

[0022] Select one set of sample data as the reference sample set, and the other set of sample data as the auxiliary sample set. r To refer to the feature data set in the sample set, assume that the feature space is divided into K subspaces. First, define the fuzzy set for the kth subspace:

[0023]

[0024] Among them, x i is the characteristic information of the i-th data point in the reference sample set, is the residual of the ith data point for the auxiliary sample set, which is obtained by the auxiliary sample set at x i The estimated label value of the position is the same as the reference sample set in x i The actual label value of the location is obtained by subtracting; is the membership of the i-th data point to the k-th subspace, and the solution is:

[0025] When the feature space is hard-divided into multiple subspaces, the feature point uniquely belongs to a subspace, and the data point x in the feature space i The membership function belonging to the kth subspace is expressed as:

[0026]

[0027] When the feature space is soft-divided into multiple subspaces, the feature points have different membership degrees for each subspace, and the membership function is a Gaussian membership function, a triangular membership function, or a Gamma membership function.

[0028] The distribution deviation measure z of the kth subspace k It can be expressed as a pair of fuzzy sets B kDefuzzification is performed, and the specific calculation method is:

[0029]

[0030] Where n is the number of samples in the reference sample set.

[0031] The method for calculating the conditional distribution deviation function between two sets of data is one of the following methods:

[0032] Method 1: Using the membership function of the data points in the feature space to each subspace as the weight, the conditional distribution deviation measurement index of each subspace is weighted summed to obtain the conditional distribution deviation function between the two sets of data, expressed as:

[0033]

[0034] where μ k (x) is the membership function of the data point to the kth subspace, z k is the distribution deviation measure of the kth subspace, h(x) is the conditional distribution deviation function between two sets of data;

[0035] Method 2: Calculate the membership value of the data points in the feature space to each subspace, select the subspace with the largest membership value, and use the conditional distribution deviation measurement index of this subspace as the conditional distribution deviation value of the data point, thereby obtaining the conditional distribution deviation function between the two sets of data, which is expressed as:

[0036]

[0037] The second technical solution of the present invention is:

[0038] A method for correcting the conditional distribution deviation between two sets of data, characterized in that it includes the following steps:

[0039] The method for characterizing the conditional distribution deviation between two sets of data is executed, with one set of sample data sets as a reference and the other set of sample data sets as an auxiliary data set, to obtain a conditional distribution deviation function between the two sets of data, and to perform a specified operation on the data in the auxiliary data set in combination with the conditional distribution deviation function, thereby correcting the conditional distribution deviation between the two sets of data, so that the conditional distribution between the data in the auxiliary data set and the data in the reference data set after the operation is closer.

[0040] The specified operation on the auxiliary sample set data adopts one of the following methods:

[0041] Performing feature transformation correction on the data in the auxiliary sample set so that the conditional distribution between the data in the auxiliary sample set and the data in the reference sample set after the correction is closer, and the transformation matrix used to perform the feature transformation operation is determined by the conditional distribution deviation function between the two sets of data;

[0042] Performing weighted correction on the data in the auxiliary sample set so that the conditional distribution between the data in the auxiliary sample set and the data in the reference sample set after correction is closer, wherein the weight function used to perform the weighted correction is determined by the conditional distribution deviation function between the two sets of data;

[0043] Residual correction is performed on the data in the auxiliary sample set so that the conditional distribution between the data in the auxiliary sample set and the data in the reference sample set after correction is closer, and the residual function used to perform residual correction is determined by the conditional distribution deviation function between the two sets of data.

[0044] The beneficial effects of the present invention are:

[0045] The present invention aims at the situation where there is a complex conditional distribution deviation between two sets of data, and transforms the difficult-to-solve problem of complex distribution deviation in the original feature space into the solution of simple distribution deviation measurement indicators in multiple subspaces.

[0046] The characterization results of the present invention can be used to correct distribution deviations, guide data sampling, guide transfer learning, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 The figure shows a first set of sample data and a second set of sample data in a specific embodiment of the present invention.

[0048] Figure 2 4 is a flow chart of a method for correcting the distribution difference between two sets of data according to an embodiment of the present invention.

[0049] Figure 3 Schematic diagram of two sets of data after the distribution difference between the two sets of data is corrected according to an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0051] like Figure 1-3 shown.

[0052] A method for characterizing the conditional distribution deviation between two sets of data is used to characterize the relationship between the conditional distribution deviation between the two sets of data and the change of feature information. The characterization result can be used to guide the correction of distribution deviation, data sampling and transfer learning. The present invention is further described below with reference to the accompanying drawings and example parts:

[0053] The distribution difference characterization method comprises the following steps:

[0054] Step 1: Obtain the first set of sample data and the second set of sample data

[0055] Features X and labels Y can be of various data types such as scalar data, vector data, tensor data, image data, text data, or other user-specified data. Figure 1 As shown, for the convenience of visual expression, the feature X of each sample in this example is a one-dimensional continuous variable, that is, represented as the horizontal axis, and the label Y is also a one-dimensional continuous variable, that is, represented as the vertical axis. The first set of sample data is shown in the attached Figure 1 The ● symbol indicates that there are 15 samples in total. The second set of sample data is in the attached Figure 1 In the example, the number of samples is represented by ×, which means there are 7 samples in total. Figure 1 It can be seen that the conditional distribution probability P(Y|X) of the two sets of data is obviously biased. When users use machine learning algorithms to train feature-to-label prediction models, the two sets of data cannot be trained directly at the same time. Therefore, it is necessary to characterize the conditional distribution deviation between the two sets of data. The characterization results can be used to guide the correction of distribution deviations and the effective operation of machine learning methods.

[0056] The characteristics and label information of the first set of sample data are as follows:

[0057] Serial number 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 feature 0.00 0.07 0.14 0.21 0.28 0.35 0.42 0.50 0.57 0.64 0.71 0.78 0.85 0.92 1.00 Label 1.00 1.48 1.83 2.00 1.91 1.60 1.14 0.64 0.23 0.02 0.03 0.28 0.70 1.21 1.64

[0058] The characteristics and label information of the second set of sample data are as follows:

[0059] Serial number 1 2 3 4 5 6 7 feature 0.1 0.25 0.4 0.55 0.7 0.85 1 Label 0.16 0.34 0.23 -0.25 -0.58 -0.17 0.75

[0060] Step 2: Divide into multiple subspaces and obtain the membership function of the data points in the feature space to each subspace

[0061] The feature space can be determined based on the distribution range of the features of the two sets of sample data, or based on the characteristics of the task faced by the sample data. The feature space can be the entire Euclidean space or a finite set, such as Figure 1 , according to the feature distribution of the two groups of samples, a feature space X∈[0,1] expressed by a finite set can be given.

[0062] When faced with specific tasks, the determination of the feature space can take into account both the distribution of sample features and the characteristics of the task. For example, in a machine learning task of predicting height based on age, the feature is age (years) and the label is height (cm). In this case, the feature space can be determined based on common sense about the distribution of age. For example, the age space can be defined as a finite interval of age∈[0,100].

[0063] The result of the feature space division is expressed as the membership function of the data points in the feature space to each subspace. The feature space division can be hard division, that is, the data points in the feature space are determined to belong to a certain subspace, or it can be soft division, that is, the data points in the feature space belong to multiple subspaces at the same time with different probabilities. Hard division can use decision tree or K-means clustering methods; soft division can use fuzzy C-means (Fuzzy c-means) or Gaussian mixture model. In this example, the fuzzy C-means method is used for subspace soft division, and the feature space is divided into K = 5 subspaces. The kth subspace is determined by the center parameter c k and the width parameter σ k Common representation, data point x in feature space i The membership function belonging to the k-th subspace may be a Gaussian membership function, a triangular membership function, or a Gamma membership function. This embodiment adopts a Gaussian membership function, which is expressed as:

[0064]

[0065] The constant A in the formula is the normalization coefficient, which makes the sum of the membership of the data point to all subspaces equal to 1. The center parameter and width parameter of the subspace can be calculated by the fuzzy C-means. The solution results are shown in the following table:

[0066] Subspace number k 1 2 3 4 5 <![CDATA[Central parameter c k > 0.06 0.28 0.49 0.72 0.93 <![CDATA[Width parameter σ k > 0.020 0.019 0.019 0.019 0.021

[0067] Step 3: Calculate the conditional distribution deviation measure of the two sets of data in each subspace

[0068] The subspace conditional distribution deviation measure index can be expressed as a single numerical form or a probability density function form. The calculation method of the subspace conditional distribution deviation measure index includes: using kernel mean difference to solve; or using KL divergence to solve; or using Euclidean distance to solve; the above methods are mature methods in this field, so the implementation methods are not shown here. The subspace conditional distribution deviation measure index can also be solved by the following method, introducing the fuzzy set definition and defuzzification method in the field of fuzzy mathematics:

[0069] Select one set of sample data as the reference sample set, and the other set of sample data as the auxiliary sample set. Generally, the sample set with a smaller number of samples is used as the reference sample set. Here, the second sample set is used as the reference, and the first set of sample data is used as the auxiliary sample set. Let X r To refer to the feature data set in the sample set, assume that the feature space is divided into K subspaces. First, define the fuzzy set for the kth subspace:

[0070]

[0071] Among them, xi is the characteristic information of the i-th data point in the reference sample set, is the residual of the ith data point for the auxiliary sample set, which is obtained by the auxiliary sample set at x i The estimated label value of the position is the same as the reference sample set in x i The actual label value of the location is obtained by subtracting it. is the membership of the ith data point to the kth subspace, and the calculation method is: when the feature space is hard divided into multiple subspaces, the feature point uniquely belongs to a subspace, and the data point x in the feature space i The membership function belonging to the kth subspace is expressed as:

[0072]

[0073] When the feature space is soft-divided into multiple subspaces, the feature point has a different degree of membership to each subspace, and the membership function is a Gaussian membership function, a triangular membership function, or a Gamma membership function.

[0074] The distribution deviation measure z of the kth subspace k It can be expressed as a pair of fuzzy sets B k Defuzzification is performed, and the specific calculation method is:

[0075]

[0076] Where n is the number of samples in the reference sample set, that is, n = 7. After the defuzzification calculation method, the calculation results of the distribution deviation measurement index of K subspaces are shown in the following table:

[0077] Subspace number k 1 2 3 4 5 <![CDATA[Condition distribution deviation measurement index z k > -1.50 -1.44 -0.81 -0.68 -0.86

[0078] Step 4: Calculate the conditional distribution deviation function between the two sets of data

[0079] The method for calculating the conditional distribution deviation function between two sets of data is one of the following methods:

[0080] Method 1: Using the membership function of the data points in the feature space to each subspace as the weight, the conditional distribution deviation measurement index of each subspace is weighted summed to obtain the conditional distribution deviation function between the two sets of data.

[0081] Method 2: Calculate the membership value of the data points in the feature space to each subspace, select the subspace with the largest membership value, and use the conditional distribution deviation measurement index of this subspace as the conditional distribution deviation value of the data point, thereby obtaining the conditional distribution deviation function between the two groups of data.

[0082] Here, method 1 is used as an embodiment, and the conditional distribution deviation function h(x) between two sets of data can be expressed as:

[0083]

[0084] Where μ k (x) is the degree of membership of the data point to the kth subspace, z k is the kth subspace distribution deviation measurement index. At this point, the conditional distribution deviation function between the two sets of data described in the present invention is solved.

[0085] Another embodiment of the present invention is as shown in the attached Figure 2 As shown, firstly, a conditional distribution deviation function between two sets of data is obtained, and a specified operation is performed on the data in the auxiliary sample set in combination with the conditional distribution deviation function, so as to correct the conditional distribution deviation between the two sets of data, so that the conditional distribution between the data in the auxiliary sample set and the data in the reference sample set after the operation is closer. The specified operation on the data in the auxiliary sample set includes feature transformation, or weighting, or residual, and the matrix or function used to perform the operation is determined by the conditional distribution deviation function between the two sets of data.

[0086] In this example, the residual method is used to correct the conditional distribution deviation. The first sample set is corrected with the second sample set as a reference to reduce the conditional distribution deviation between the two sets of data. The specific implementation steps are:

[0087] Step 1: Obtain the conditional distribution deviation function h(x) of the two sets of data

[0088] Step 2: For the sample points (x i ,y i ), add the residual to the label value according to the conditional distribution deviation function, that is, the corrected sample point is expressed as (x i ,y i +h(x i ))

[0089] The two sets of data after correction are shown in the attached Figure 3 As shown in the figure, the conditional distribution probability P(Y|X) of the two sets of data is basically consistent, and the classic machine learning algorithm can be used directly to train the feature-to-label prediction model.

[0090] The parts not involved in the present invention are the same as the prior art and are implemented by using the prior art.

Claims

1. A method for characterizing and correcting the conditional distribution deviation between two sets of data for machine learning, characterized in that: It includes the following steps: First, obtain the first set of sample data and the second set of sample data, where each sample in the two sets of data consists of a feature x and a label y; Secondly, the common feature space where the features of the first group of sample data and the features of the second group of sample data are located is divided into multiple subspaces, and the membership function of the data points in the feature space to each subspace is obtained; Third, calculate the conditional distribution deviation measure of the two sets of data in each subspace; Finally, the conditional distribution deviation function between the two sets of data is calculated using the membership function of the data points in the feature space to each subspace and the conditional distribution deviation measure of the two sets of data in each subspace; The specific method for calculating the conditional distribution deviation measurement index of two sets of data in each subspace includes: Select one set of sample data as the reference sample set, and the other set of sample data as the auxiliary sample set; let X r To refer to the feature data set in the sample set, assume that the feature space is divided into K subspaces. First, define the fuzzy set for the kth subspace: Among them, x i is the characteristic information of the i-th data point in the reference sample set, is the residual of the ith data point for the auxiliary sample set, which is obtained by the auxiliary sample set at x i The estimated label value of the position is the same as the reference sample set in x i The actual label value of the position is obtained by subtracting; is the membership of the i-th data point to the k-th subspace; The distribution deviation measure z of the kth subspace k It can be expressed as a pair of fuzzy sets B k Defuzzification is performed, and the specific calculation method is: Where n is the number of samples in the reference sample set; The method for calculating the conditional distribution deviation function between two sets of data is one of the following methods: Method 1: Using the membership function of the data points in the feature space to each subspace as the weight, the conditional distribution deviation measurement index of each subspace is weighted summed to obtain the conditional distribution deviation function between the two sets of data, expressed as: where μ k (x) is the membership function of the data point to the kth subspace, z k is the distribution deviation measure of the kth subspace, h(x) is the conditional distribution deviation function between two sets of data; Method 2: Calculate the membership value of the data points in the feature space to each subspace, select the subspace with the largest membership value, and use the conditional distribution deviation measurement index of this subspace as the conditional distribution deviation value of the data point, thereby obtaining the conditional distribution deviation function between the two sets of data, which is expressed as: The information types of the feature x and the label y are both image data or text data.

2. The method according to claim 1, characterized in that The common feature space acquisition method is one of the following: determining the common feature space according to the distribution range of the features of the two groups of sample data; Determine the common feature space based on the task characteristics faced by the sample data; At the same time, the common feature space is determined according to the distribution range of the sample data characteristics and the characteristics of the tasks faced by the sample data.

3. The method according to claim 1, characterized in that: The feature space division includes two types: hard division and soft division. The hard division method includes decision tree or K-means clustering, and the soft division method includes Gaussian mixture model or fuzzy C-means.

4. The method according to claim 1, characterized in that The membership function is used to represent the degree of membership of a data point in the feature space to each subspace, and the membership function calculation method is: When the feature space is hard-divided into multiple subspaces, the feature point uniquely belongs to a subspace, and the data point x in the feature space i The membership function belonging to the kth subspace is expressed as: When the feature space is soft-divided into multiple subspaces, the feature point has a different degree of membership to each subspace, and the membership function is a Gaussian membership function, a triangular membership function, or a Gamma membership function.

5. The method according to claim 1, characterized in that The subspace conditional distribution deviation measurement index is used to approximately represent the conditional distribution deviation between the first sample data and the second sample data in each subspace, and the specific form of the subspace conditional distribution deviation measurement index includes a single numerical form or a probability density function form; The calculation method of the subspace conditional distribution deviation measurement index includes: solving by kernel mean difference; or solving by KL divergence; or solving by Euclidean distance.

6. The method according to claim 1, characterized in that: The correction method is: A method for characterizing the conditional distribution deviation between two sets of data is executed to obtain a conditional distribution deviation function between the two sets of data. The conditional distribution deviation function is used to perform a specified operation on the data in the auxiliary sample set, thereby correcting the conditional distribution deviation between the two sets of data, so that the conditional distribution between the data in the auxiliary sample set and the data in the reference sample set after the operation is closer.

7. The method according to claim 6, characterized in that The specified operation on the auxiliary sample set data adopts one of the following methods: Performing feature transformation correction on the data in the auxiliary sample set so that the conditional distribution between the data in the auxiliary sample set and the data in the reference sample set after the correction is closer, and the transformation matrix used to perform the feature transformation operation is determined by the conditional distribution deviation function between the two sets of data; Performing weighted correction on the data in the auxiliary sample set so that the conditional distribution between the data in the auxiliary sample set and the data in the reference sample set after correction is closer, wherein the weight function used to perform the weighted correction is determined by the conditional distribution deviation function between the two sets of data; Residual correction is performed on the data in the auxiliary sample set so that the conditional distribution between the data in the auxiliary sample set and the data in the reference sample set after correction is closer, and the residual function used to perform residual correction is determined by the conditional distribution deviation function between the two sets of data.

Images