Machine learning high-value data subset generation method based on self-information measure

Abstract

The invention provides a machine learning high-value data subset generation method based on self-information measurement, which comprises the following steps: establishing a probability model for each data point or event in a data set, selecting a parameter estimation method according to the determined model complexity and data volume, and generating a high-value data subset according to the parameter estimation method. Estimating the probability of occurrence of the data points by using the estimated model parameters; for each data point in the data set, according to the occurrence probability of the data point, calculating the self-information amount of the data point by using a self-information amount function; and performing value evaluation on the data through the obtained self-information amount to obtain the value of each data point, and taking a set formed by high-value data points as a high-value data subset for further machine learning model training. According to the method, the self-information principle in the information theory is effectively utilized to quantify and evaluate the data value, and the data subset capable of improving the model training efficiency is screened out by accurately measuring the uncertainty and scarcity of the data.

Description

Technical Field

[0001] The present invention belongs to the technical field of data value assessment, and in particular, relates to a method for generating high-value data subsets for machine learning based on self-information measurement. Background Art

[0002] With the rapid development of information technology, the value of data in all walks of life has become increasingly prominent. In this context, data is not only regarded as an important production factor, but also as an independent asset. Its magnitude and value have made a qualitative leap in the digital age. However, with the rise of big data and the popularity of machine learning models, how to accurately evaluate and fully utilize the value of data, especially the value of data after it has been converted or processed, remains a major challenge in this field.

[0003] At present, there are two main methods for traditional data value assessment: one is to regard data as a production factor, emphasizing its role in creating economic value; the other is to regard data as an independent asset, emphasizing its value as a resource and its potential as an asset. Although these two methods provide a framework for understanding the value of data in theory, in practical applications, they face the challenges of high complexity and dynamic changes in data, especially in the big data environment, where data types are diverse and come from a wide range of sources, often containing a large amount of redundant information, which greatly increases the complexity of the assessment work.

[0004] In machine learning, with the rapid growth of data volume and model complexity, the performance of the model has been significantly improved, but it has also increased computing costs and resource requirements. Therefore, in the process of training large machine learning models, it is urgent to screen out high-value subsets from huge data sets, especially when resources are limited, to ensure that the data sets used for model training have the highest value density, reduce the required computing resources and improve the efficiency of model training. To solve these problems, this study proposes a method for generating high-value subsets for machine learning training models based on self-information measurement. This method uses the principle of self-information to quantify and evaluate the value of data, and extracts high-value data subsets based on the value of the data. These subsets will be used to train machine learning models to optimize model performance and reduce training costs.

[0005] The key to self-information measurement is that it can quantify the amount of information contained in data points. It not only focuses on the uncertainty and scarcity of data, but also can effectively deal with the complexity of data. By quantifying the amount of information of each data point, we can screen out data points with high value, thereby forming a high-value data subset. The advantage of this method is that it does not depend on the specific content of the data, but evaluates its value based on the probability distribution of the data, making the evaluation method both scientific and reasonable and highly adaptable. In addition, this high-value subset generation method based on self-information also takes into account the dynamics and timeliness of the data, and can adapt to the characteristics of data value changing over time, usage environment and purpose. This not only improves the utilization efficiency of data resources, but also provides a new methodology for the training of large-scale machine learning models, reducing the high computing costs caused by large amounts of data, making it an ideal choice for evaluating the value of data in large-scale data sets. Summary of the invention

[0006] In view of this, the present invention aims to overcome the shortcomings of the above-mentioned problems in the prior art and proposes a method for generating high-value data subsets for machine learning based on self-information measurement.

[0007] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0008] The first aspect of the present invention provides a method for generating a high-value data subset for machine learning based on self-information measurement, comprising the following steps:

[0009] Step 1: Establish a probability model for each data point or event in the data set. According to the determined model complexity and data size, select a parameter estimation method and use the estimated model parameters to estimate the probability of occurrence of these data points.

[0010] Step 2: For each data point in the data set, use the self-information function to calculate its self-information according to its probability of occurrence;

[0011] Step 3: Evaluate the data using the self-information obtained to obtain the value of each data point. The set of high-value data points is used as a high-value data subset for further machine learning model training.

[0012] Furthermore, step 1 also includes preprocessing the data set, specifically including:

[0013] Missing value processing: Select a filling strategy to fill in data based on the type of data features and the distribution of missing data;

[0014] Data filtering: Identify and filter data by type, remove non-numeric data, evaluate the correlation between attributes in the data set, and identify and remove highly correlated attributes;

[0015] Feature encoding: converting non-numerical features into numerical data that machine learning models can process;

[0016] Distribution property analysis: Analyze the distribution properties of each feature in the dataset, including examining the mean, median, variance, standard deviation, skewness, and kurtosis of each feature to identify any potential deviations or outliers.

[0017] Furthermore, in step 1, establishing a probability model for each data point or event in the data set includes:

[0018] After data preprocessing, the probability model is selected according to the type and distribution characteristics of the data and the set criteria, including binomial distribution model, Poisson distribution model, Markov model, normal distribution (Gaussian distribution) model, lognormal distribution model, exponential distribution model, and gamma distribution model.

[0019] Furthermore, in step 1, according to the determined model complexity and data size, selecting a parameter estimation method includes:

[0020] For binomial distribution model and Poisson distribution model, the maximum likelihood estimation method was selected to estimate model parameters;

[0021] For the normal distribution model, the moment estimation method is applied to estimate the distribution parameters based on the sample moments;

[0022] For the parameter estimation of the lognormal distribution model, it is done by taking the logarithm of the data and then applying the parameter estimation method of the normal distribution;

[0023] For the exponential distribution model, the maximum likelihood estimation method was chosen to observe the mean of the reciprocal of the data interval;

[0024] For the gamma distribution model, the maximum likelihood estimation (MLE) method is used to find the parameters that maximize the probability of occurrence of the given sample data;

[0025] For Markov models, use the EM algorithm or Bayesian estimation.

[0026] Furthermore, in step 1, estimating the probability of occurrence of these data points using the estimated model parameters includes:

[0027] The selected parameter estimation method was used to evaluate the validity and goodness of fit of the model. The goodness of fit and complexity of different models were compared by calculating the Akaike Information Criterion and the Bayesian Information Criterion values, and the most appropriate model was selected.

[0028] Using the estimated model parameters, the probability mass function or probability density function of the selected probability model is determined to calculate the probability of each data point.

[0029] Furthermore, the step 2 specifically includes:

[0030] For each data point in the data set, its self-information is calculated based on its probability of occurrence. The self-information measures the uncertainty of a single event, and this uncertainty is closely related to the probability of the event. The self-information of a specific event can be calculated by the prior probability of the event, and its mathematical expression is

[0031]

[0032] Where P(a i ) is event a i The prior probability of occurrence, I(a i ) represents event a i The amount of information contained in the occurrence is called self-information. If the logarithmic base is 2, the unit of self-information is bit.

[0033]

[0034] If the logarithm base is e, the resulting self-information is in nights.

[0035]

[0036] Furthermore, the step 3 specifically includes:

[0037] The self-information obtained by calculation is used to sort the self-information of each data point using a merge sort algorithm. The data points and their self-information are sequentially filled into the database table according to the sorting results. The self-information data is extracted from the database, and the mean, standard deviation, skewness, and kurtosis are calculated. The dynamic threshold is determined based on the calculated statistical indicators, data characteristics, and specific application requirements.

[0038] If the data is significantly skewed, adjust the threshold to include more informative but less frequent data points, or dynamically adjust the threshold in combination with real-time data updates and model feedback to ensure that the data subset always reflects the latest data characteristics and model requirements. If the performance feedback of the model shows that the data subset under the current threshold is no longer valid, further adjust the threshold based on the model error or accuracy loss.

[0039] Using the above threshold, we filter out data points that exceed the threshold from the entire data set and mark them as high-value data points. The set of these data points serves as the required high-value subset. We use the high-value subset to train the machine learning model, observe the performance of the model on specific tasks, and feed the model training results back to the data subset generation process to further optimize the selection of data points and threshold setting.

[0040] A second aspect of the present invention provides a device for generating a high-value data subset for machine learning based on self-information measurement, comprising:

[0041] Data processing unit: used to establish a probability model for each data point or event in the data set, select a parameter estimation method based on the determined model complexity and data volume, and use the estimated model parameters to estimate the probability of occurrence of these data points;

[0042] A self-information calculation unit is used to calculate the self-information of each data point in the data set according to its probability of occurrence using a self-information function;

[0043] The data value assessment unit is used to assess the value of the data through the obtained self-information to obtain the value of each data point. The set of high-value data points is used as a high-value data subset for further machine learning model training.

[0044] A third aspect of the present invention provides an electronic device, comprising a processor and a memory communicatively connected to the processor and used to store executable instructions of the processor, wherein the processor is used to execute the above-mentioned method for generating a high-value data subset for machine learning based on self-information measurement.

[0045] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned method for generating a high-value data subset for machine learning based on self-information measurement.

[0046] Compared with the prior art, the method for generating high-value data subsets for machine learning based on self-information measurement described in the present invention has the following advantages:

[0047] The method of the present invention effectively utilizes the self-information principle in information theory to quantify and evaluate the value of data, generates the required number of high-value data subsets specifically for the needs of machine learning, highlights the potential value of data after being converted into information, and screens out the data subset that can best increase the efficiency of model training by accurately measuring the uncertainty and scarcity of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0049] Figure 1 The figure is a flow chart of a method for generating a high-value data subset for machine learning based on self-information measurement according to the present invention. DETAILED DESCRIPTION

[0050] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0051] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and the like are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", and the like may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0052] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood by specific circumstances.

[0053] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0054] Embodiment 1:

[0055] The present invention proposes a method for generating high-value data subsets for a machine learning training model based on self-information measurement. The method utilizes the self-information principle in information theory to quantify and evaluate data value, and generates the required number of high-value data subsets based on the evaluation results.

[0056] In the process of data value assessment, this method copes with the high complexity of data by quantifying the amount of self-information of each data point. The calculation of self-information reflects the uniqueness of each data point and the amount of information it adds to the overall data set. Since self-information is a probability-based measurement, it can naturally adapt to a variety of data types and sources. It does not depend on the specific content of the data, but on the distribution and probabilistic characteristics of the data. This feature enables this method to effectively solve the problem of data complexity and identify and filter out redundant information. In addition, self-information provides a mathematically clear evaluation criterion that does not rely on prior knowledge or evaluation criteria in a specific field. This makes it a universal evaluation tool suitable for value assessment in different fields and data types.

[0057] Self-information measurement can also dynamically adapt to changes in data value because it is based on the probability of a data point occurring, which can change over time, usage environment, and purpose. By regularly recalculating the amount of self-information of data points, the evaluation method can reflect real-time changes in data value, provide dynamic data value evaluation, and effectively solve the problem of data value changing based on dynamics and timeliness.

[0058] To achieve the above objectives, the present invention proposes a new technical solution for evaluating the value of self-information measurement data in machine learning training models and generating high-value data subsets. The overall process is as follows: Figure 1 As shown, it mainly includes the following steps:

[0059] A method for generating a high-value data subset for a machine learning training model based on self-information measurement comprises the following steps:

[0060] Step 1: Establish a probability model for each data point or event in the data set. According to the determined model complexity and data size, select a parameter estimation method and use the estimated model parameters to estimate the probability of occurrence of these data points.

[0061] Step 2: For each data point in the data set, use the self-information function to calculate its self-information according to its probability of occurrence. The level of self-information directly affects the possibility of the data point being selected as a high-value subset;

[0062] Step 3: Use the obtained self-information to evaluate the value of the data and obtain the value of each data point. The set of high-value data points is the high-value data subset, which is used for further machine learning model training.

[0063] Specifically, for step 1,

[0064] 1. First, preprocess the collected data. Dataset preprocessing is a key step in machine learning and is essential to ensure the effectiveness of model training. At this stage, processing missing values ​​first is a key step to maintain data integrity. According to the type of data features and the distribution of missing data, choose an appropriate filling strategy, such as using the mean, median, or mode for filling.

[0065] 2. Identify and filter data by type, and remove non-numeric data to ensure consistency and accuracy of subsequent analysis. Next, evaluate the correlation between attributes in the data set, identify and remove highly correlated attributes. This step helps reduce redundancy between data features and improve the efficiency and accuracy of model training. For categorical features, use appropriate encoding strategies, such as one-hot encoding or label encoding, to convert non-numeric features into numerical data that can be processed by machine learning models.

[0066] 3. After encoding the features, analyze the distribution characteristics of each feature in the dataset, including checking the mean, median, variance, standard deviation, skewness, and kurtosis of each feature to identify any potential biases or outliers.

[0067] 4. After preprocessing the data, the probability model is selected according to the type and distribution characteristics of the data and the following criteria:

[0068] 4.1 Binomial Distribution Model

[0069] The binomial distribution model can be used for analysis and calculation if there are only two possible results for each observation and the probabilities of these two results are fixed and constant, and the total number of results is fixed and the probability of a single result is known. The formula is:

[0070]

[0071] Where P(X=k) represents the probability of data k appearing in n sets of data. p is the probability of data k, and (1-p) is the probability of data not being k.

[0072] 4.2 Poisson Distribution Model

[0073] If the data generation times are independent and have a fixed average generation rate within a specific time interval or spatial range, the Poisson distribution model can be used for analysis. The formula is:

[0074]

[0075] Wherein, P(X=k) represents the probability that data is generated exactly k times in a given time interval, and λ is the average number of data generation per unit time.

[0076] 4.3 Markov Model

[0077] When predicting the probability of each data point in a series of data, and the probability of the next state of these data points depends only on their current state and has nothing to do with the previous state (showing memorylessness or Markov property), the Markov model can be used. t is the state at time t, the state space S = {s1, s2, s3, ..., s n}, n is the total number of states, then the state transition probability is

[0078] P ij =P(X t+1 =s j |X t =s i )(1≤i,j≤n) (3)

[0079] 4.4 Normal distribution (Gaussian distribution) model

[0080] For continuous data, when the data is symmetrically distributed around the central value, the normal distribution (Gaussian distribution) model can be used for analysis. The formula is

[0081]

[0082] Where μ is the mean and σ is the standard deviation.

[0083] 4.5 Lognormal Distribution Model

[0084] For continuous data, when all data points are positive and the distribution is obviously right-skewed (i.e., has a long tail), the lognormal distribution model can be used. This model is applicable when the natural logarithm of a random variable follows a normal distribution. Its formula is

[0085]

[0086] Among them, is the mean and σ is the standard deviation.

[0087] 4.6 Exponential Distribution Model

[0088] For continuous data, when the data points occur randomly and independently in continuous time or space, and the probability of a data point occurring in a certain period of time in the future is independent of the time it has waited or occurred so far (showing no memory), the exponential distribution model can be used for analysis. Its formula is

[0089] f(X)=λe -λx (6)

[0090] Where λ is the average occurrence rate of the data.

[0091] 4.7 Gamma Distribution Model

[0092] For continuous data, the gamma distribution model can be used to analyze and describe the total waiting time or cumulative amount of multiple independent and similar events that occur in a fixed time period or spatial interval. This model is suitable for cumulative event occurrences, especially when the focus is on the sum of the number of event occurrences or the sum of event waiting times. Its formula is

[0093]

[0094] where α is the shape parameter of the distribution, β is the scale parameter that affects the width or spread of the distribution, and Γ(α) is the gamma function. For integer α,

[0095] Γ(α)=(α-1)! (8)

[0096] 5. Select a parameter estimation method based on the determined model complexity and data size.

[0097] For binomial and Poisson distribution models, select Maximum Likelihood Estimation (MLE). MLE estimates model parameters by maximizing the likelihood function of the observed data.

[0098]

[0099] Where x is the number of times the data is observed and n is the total number of data.

[0100]

[0101] where x i The number of times data is generated in each unit, n is the total number of data.

[0102] For the normal distribution model, the moment estimation method is used to estimate the distribution parameters based on the sample moments, which includes two parameters, among which the estimate of the mean μ is

[0103]

[0104] Variance σ 2 The estimate is

[0105]

[0106] where x i is the sample data point, and n is the total number of sample data points.

[0107] For the parameter estimation of the lognormal distribution model, the parameter estimation method of the normal distribution is applied after taking the logarithm of the data. The estimate of the mean μ after logarithmic transformation is

[0108]

[0109] Logarithmic variance σ 2 The estimate is

[0110]

[0111] where x i is the sample data point, and n is the total number of sample data points.

[0112] For the exponential distribution model, choose the maximum likelihood estimation (MLE) method to observe the mean of the reciprocal of the data interval

[0113]

[0114] in is the sample mean;

[0115] For the gamma distribution model, the maximum likelihood estimation (MLE) method is used to find the parameters α and β that maximize the probability of occurrence of a given sample data. For a given data sample, the log-likelihood function of the gamma distribution is

[0116]

[0117] The partial derivatives of l(α, β) with respect to α and β are taken to obtain a set of equations, which are solved using numerical optimization methods. Starting from the initial estimates, the estimates of α and β are iteratively optimized until they converge to the maximum likelihood estimate.

[0118] For Markov models and more complex models with latent variables, use the EM algorithm or Bayesian estimation.

[0119] The EM algorithm iteratively estimates model parameters by alternating between the expectation step (E step) and the maximization step (M step). It is used to find parameter estimates that maximize the likelihood function in the presence of hidden variables. First, select the initial estimate of the parameter.

[0120] Step E (Expectation): Based on the current parameter estimates, calculate the conditional probability distribution of the hidden state, that is, the probability of each hidden state occurring given the observed data and the current parameter estimates.

[0121] M-step (Maximization): Update the parameter estimates to maximize the likelihood function using the probability distribution of the hidden state calculated in step E. This usually involves finding the maximum value of the likelihood function with respect to the parameters.

[0122] Repeat steps E and M until the parameter estimation converges or a certain termination condition is met, and finally obtain the parameter value that maximizes the likelihood function.

[0123] Bayesian estimation estimates parameters by combining prior knowledge and observed data. First, for the state transition probability P of the Markov model ij Define a prior distribution, and use the likelihood function L(θ|X) (Formula (1)(2)(4)) to express the probability of observing data X given the parameter θ. Then apply Bayes' theorem to combine the prior distribution and the likelihood function to provide the posterior distribution of the parameter. Calculate the posterior distribution

[0124] P(θ|X)=(L(θ|X)·P(θ)) / P(X) (17)

[0125] Where P(θ|X) is the posterior distribution of parameter θ, P(θ) is the prior distribution, L(θ|X) is the likelihood function, and P(X) is the marginal probability of the observed data, which serves as a normalizing constant.

[0126] 6. Then apply the selected parameter estimation method to evaluate the model's effectiveness and goodness of fit.

[0127] Through the evaluation results under different prediction models, the most suitable model is obtained for fitting.

[0128] By Akaike Information Criterion (AIC)

[0129]

[0130] where k is the number of model parameters, is the maximum likelihood value of the model.

[0131] By Bayesian Information Criterion (BIC)

[0132]

[0133] Where n is the sample size, k is the number of model parameters, is the maximum likelihood value of the model.

[0134] Akaike information criterion (AIC) and Bayesian information criterion (BIC) values ​​were calculated to compare the goodness of fit and complexity of different models and select the most appropriate model.

[0135] 7. Finally, use the estimated model parameters to determine the probability mass function (PMF) (Formula (1) (2) (3)) or probability density function (PDF) (Formula (4) (5) (6) (7)) of the selected probability model to calculate the probability of each data point.

[0136] Specifically, in step 2, for each data point in the data set, its self-information is calculated based on the probability of its occurrence. Self-information measures the uncertainty of the occurrence of a single event, and this uncertainty is closely related to the probability of the event. Specifically, the lower the probability of an event, the greater the amount of information obtained from the occurrence of the event, and the higher its uncertainty; conversely, the higher the probability of an event, the less information it provides. For an event that is bound to occur (i.e., an event with a probability of 1), its uncertainty is zero, so the amount of information obtained from the event is also zero. According to Shannon's definition, the self-information of a specific event can be calculated by the prior probability of the event, and its mathematical expression is:

[0137]

[0138] Where P(a i ) is event a i The prior probability of occurrence, I(a i ) represents event a i The amount of information contained in the occurrence is the self-information. If the logarithmic base is 2, the unit of the self-information is bits, that is,

[0139]

[0140] If the logarithmic base is e, the resulting self-information is in nights, that is,

[0141]

[0142] Then the calculated self-information is sorted according to the size of the self-information.

[0143] Specifically, in step 3, the self-information of each data point is first sorted using an efficient merge sort algorithm to ensure that high-value data is selected first. Then, the data points and their self-information are filled into the database table in turn according to the sorting results. Then, the self-information data is extracted from the database, and the following statistical indicators are calculated to define the generation strategy of the high-value subset:

[0144] Mean: It is the average value of the self-information of the data points, and the calculation formula is

[0145]

[0146] Standard Deviation: It is a statistical indicator to measure the degree of dispersion of the self-information distribution. The calculation formula is

[0147]

[0148] Among them I(a i) represents the self-information of each data point, and n is the total number of data points.

[0149] Skewness: describes the skewness direction and degree of data distribution, indicating the asymmetry of the self-information distribution. The calculation formula is

[0150]

[0151] Kurtosis (Kurtosi s): measures the sharpness of the distribution of self-information, and helps determine the concentration or dispersion of data points. The calculation formula is

[0152]

[0153] The dynamic threshold is selected and determined based on the calculated statistical indicators, data characteristics and specific application requirements.

[0154] The threshold is determined by using the mean double the standard deviation, which is calculated as:

[0155] Threshold=Mean+k·Standard Deviation (27)

[0156] Here, k is an adjustable coefficient, usually 1, 1.5 or 2, determined according to the distribution of the data and the rarity of the data points to be identified.

[0157] Or use percentile-based thresholding to determine the threshold based on the quantile of the data. Choose a specific percentile so that the data points above the threshold represent the most valuable part of the entire data set.

[0158] If the data is significantly skewed, adjust the threshold to include more informative but less frequent data points, calculated as:

[0159] Adjusted Threshold=Threshold±Skewness Factor (28)

[0160] Skewness Factor=α·Skewness (29)

[0161] The Skewness Factor can be calculated based on the skewness, and α is a scaling factor that depends on the positive or negative skewness and the characteristics of the data set.

[0162] Or, combine real-time data updates and model feedback to dynamically adjust the threshold to ensure that the data subset always reflects the latest data characteristics and model requirements. First, define the initial threshold, and its formula is as follows:

[0163] Threshold initial =Mean+k·Standard Deviation (30)

[0164] where k is a coefficient adjusted according to the data distribution.

[0165] When new data arrives or the model is iteratively updated, the threshold should be adjusted based on the new data characteristics and model feedback. This can be achieved through a weighted average formula, where the new threshold is the weight of the previous threshold and the new recommended threshold calculated based on the new data, and the formula is as follows:

[0166] Threshold new =α new Threshold old +(1-α new )·(Mean new +k new ·StandardDeviation new ) (31)

[0167] where α new is a decay factor between 0 and 1, used to balance the weight of the old threshold and the new threshold; Mean new and Standard Deviation new are the mean and standard deviation updated based on the new data, k new It can be adjusted according to the distribution characteristics of new data.

[0168] If the performance feedback of the model shows that the data subset under the current threshold is no longer valid, the threshold can be further adjusted according to the model error or accuracy loss, and the formula is as follows:

[0169] Threshold final =Threshold new +β new ΔPerformance (32)

[0170] where β new is the adjustment coefficient used to control the impact of model performance changes on threshold adjustment; ΔPerformance is the performance change of the model since the last evaluation, which can be an increase or decrease in error or a change in accuracy.

[0171] Using the above threshold, data points that exceed the threshold are filtered out from the entire data set and marked as high-value data points. The collection of these data points is the required high-value subset, which can be used for training machine learning models to improve the prediction accuracy and efficiency of the model. Through cross-validation or real-time performance monitoring, the actual impact of high-value subsets on model performance is evaluated to ensure the quality and effectiveness of the selected subsets. Use high-value subsets to train machine learning models and observe the performance of the models on specific tasks, such as accuracy, F1 score, mean square error (MSE), etc. At the same time, taking into account the dynamic changes of data, the model training results are fed back to the data subset generation process to further optimize the selection of data points and threshold setting. Let the current evaluation index of the model be M. Evaluate the quality of the subset by comparing the performance of the model trained using the full data set and the model trained only using the high-value subset. Let the index of the full data set model be M full and M subset The initial threshold is set to T, and based on the feedback of the evaluation indicators, the threshold is adjusted to optimize the model performance. The threshold adjustment can be expressed by the following formula:

[0172] T new =T+γ new ·(M subset -M full ) (33)

[0173] Among them, γ new is the learning rate parameter, which controls the speed and magnitude of the threshold adjustment.

[0174] Based on the feedback from the model performance, adjust the criteria for selecting data points. subset Significantly lower than M full , you may need to increase the number of data points in the subset or adjust the selection criteria of the data points. The formula can be expressed as:

[0175] S new =f(S,M subset -M full ) (34)

[0176] Here, S represents the current data point selection criteria, and f is the adjustment function, which may include increasing the diversity of data points or modifying the way the information content of data points is calculated.

[0177] Set the new threshold T new and data point selection criteria S new Applied to the next round of data subset generation and model training, the optimization process is continuously iterated.

[0178] Embodiment 2:

[0179] A device for generating high-value data subsets for machine learning based on self-information measurement, comprising:

[0180] Data processing unit: used to establish a probability model for each data point or event in the data set, select a parameter estimation method based on the determined model complexity and data volume, and use the estimated model parameters to estimate the probability of occurrence of these data points;

[0181] A self-information calculation unit is used to calculate the self-information of each data point in the data set according to its probability of occurrence using a self-information function;

[0182] The data value assessment unit is used to assess the value of the data through the obtained self-information to obtain the value of each data point. The set of high-value data points is used as a high-value data subset for further machine learning model training.

[0183] Embodiment three:

[0184] An electronic device comprises a processor and a memory which is communicatively connected to the processor and is used to store instructions executable by the processor, wherein the processor is used to execute the above-mentioned method for generating a high-value data subset for machine learning based on self-information measurement.

[0185] Embodiment 4:

[0186] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned method for generating a high-value data subset for machine learning based on self-information measurement.

[0187] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating high-value data subsets for machine learning based on self-information measurement, characterized in that: The steps include: Step 1: Establish a probability model for each data point or event in the data set. According to the determined model complexity and data size, select a parameter estimation method and use the estimated model parameters to estimate the probability of occurrence of these data points. Step 2: For each data point in the data set, use the self-information function to calculate its self-information according to its probability of occurrence; Step 3: Evaluate the data using the self-information obtained to obtain the value of each data point. The set of high-value data points is used as a high-value data subset for further machine learning model training.

2. The method for generating a high-value data subset for machine learning based on self-information measurement according to claim 1, characterized in that: The step 1 also includes preprocessing the data set, specifically including: Missing value processing: Select a filling strategy to fill in data based on the type of data features and the distribution of missing data; Data filtering: Identify and filter data by type, remove non-numeric data, evaluate the correlation between attributes in the data set, and identify and remove highly correlated attributes; Feature encoding: converting non-numerical features into numerical data that machine learning models can process; Distribution property analysis: Analyze the distribution properties of each feature in the dataset, including examining the mean, median, variance, standard deviation, skewness, and kurtosis of each feature to identify any potential deviations or outliers.

3. The method for generating a high-value data subset for machine learning based on self-information measurement according to claim 1, characterized in that: In step 1, establishing a probability model for each data point or event in the data set includes: After data preprocessing, the probability model is selected according to the type and distribution characteristics of the data and the set criteria, including binomial distribution model, Poisson distribution model, Markov model, normal distribution (Gaussian distribution) model, lognormal distribution model, exponential distribution model, and gamma distribution model.

4. The method for generating a high-value data subset for machine learning based on self-information measurement according to claim 3, characterized in that: In step 1, according to the determined model complexity and data size, selecting a parameter estimation method includes: For binomial distribution model and Poisson distribution model, the maximum likelihood estimation method was selected to estimate model parameters; For the normal distribution model, the moment estimation method is applied to estimate the distribution parameters based on the sample moments; For the parameter estimation of the lognormal distribution model, it is done by taking the logarithm of the data and then applying the parameter estimation method of the normal distribution; For the exponential distribution model, the maximum likelihood estimation method was chosen to observe the mean of the reciprocal of the data interval; For the gamma distribution model, the maximum likelihood estimation (MLE) method is used to find the parameters that maximize the probability of occurrence of the given sample data; For Markov models, use the EM algorithm or Bayesian estimation.

5. The method for generating a high-value data subset for machine learning based on self-information measurement according to claim 1, characterized in that: In step 1, estimating the probability of occurrence of these data points using the estimated model parameters includes: The selected parameter estimation method was used to evaluate the validity and goodness of fit of the model. The goodness of fit and complexity of different models were compared by calculating the Akaike Information Criterion and the Bayesian Information Criterion values, and the most appropriate model was selected. Using the estimated model parameters, the probability mass function or probability density function of the selected probability model is determined to calculate the probability of each data point.

6. The method for generating a high-value data subset for machine learning based on self-information measurement according to claim 1, characterized in that: The step 2 specifically includes: For each data point in the data set, its self-information is calculated based on its probability of occurrence. The self-information measures the uncertainty of a single event, and this uncertainty is closely related to the probability of the event. The self-information of a specific event can be calculated by the prior probability of the event, and its mathematical expression is Where P(a i ) is event a i The prior probability of occurrence, I(a i ) represents event a i The amount of information contained in the occurrence is called self-information. If the logarithmic base is 2, the unit of self-information is bit. If the logarithm base is e, the resulting self-information is in nights.

7. The method for generating a high-value data subset for machine learning based on self-information measurement according to claim 1, characterized in that: The step 3 specifically includes: The self-information obtained by calculation is used to sort the self-information of each data point using a merge sort algorithm. The data points and their self-information are sequentially filled into the database table according to the sorting results. The self-information data is extracted from the database, and the mean, standard deviation, skewness, and kurtosis are calculated. The dynamic threshold is determined based on the calculated statistical indicators, data characteristics, and specific application requirements. If the data is significantly skewed, adjust the threshold to include more informative but less frequent data points, or dynamically adjust the threshold in combination with real-time data updates and model feedback to ensure that the data subset always reflects the latest data characteristics and model requirements. If the performance feedback of the model shows that the data subset under the current threshold is no longer valid, further adjust the threshold based on the model error or accuracy loss. Using the above threshold, we filter out data points that exceed the threshold from the entire data set and mark them as high-value data points. The set of these data points serves as the required high-value subset. We use the high-value subset to train the machine learning model, observe the performance of the model on specific tasks, and feed the model training results back to the data subset generation process to further optimize the selection of data points and threshold setting.

8. A device for generating high-value data subsets for machine learning based on self-information measurement, characterized in that: include Data processing unit: used to establish a probability model for each data point or event in the data set, select a parameter estimation method based on the determined model complexity and data volume, and use the estimated model parameters to estimate the probability of occurrence of these data points; A self-information calculation unit is used to calculate the self-information of each data point in the data set according to its probability of occurrence using a self-information function; The data value assessment unit is used to assess the value of the data through the obtained self-information to obtain the value of each data point. The set of high-value data points is used as a high-value data subset for further machine learning model training.

9. An electronic device, comprising a processor and a memory connected to the processor for storing instructions executable by the processor, characterized in that: The processor is used to execute a method for generating a high-value data subset for machine learning based on self-information measurement as described in any one of claims 1-7 above.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements a method for generating a high-value data subset for machine learning based on self-information measurement as described in any one of claims 1 to 7.

Images