A data value evaluation method and system based on information entropy measure

Abstract

The application provides a data value evaluation method and system based on information entropy measurement, the method comprises the following steps: obtaining data sets to be evaluated; establishing a probability model for each data set; selecting a parameter estimation method according to the complexity of the determined model and the size of the data quantity; using the estimated model parameters to estimate the probability of the data points; for each data set, using an information quantity function to calculate the information entropy of the data set; using the calculated information entropy to evaluate the average information quantity or uncertainty of the entire data set to obtain the value of each data set. The method does not directly depend on the specific content of the data, but depends on the overall distribution and probability characteristics of the data, so that the problem of high data complexity can be effectively solved, and redundant information can be identified and filtered out.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data processing, and particularly relates to a data value evaluation method and system based on information entropy measurement. BACKGROUND

[0002] With the rapid development of information technology, data has become an important resource in various industries. However, how to accurately evaluate the value of data, especially after data is converted or processed, is still a challenge in this field. Currently, the main direction of measuring data value mainly includes two aspects:

[0003] First, data is converted into traditional production factors. In the context of digital economy, data has become a key resource that is on par with traditional production factors, and its value and circulation have a profound impact on economic growth and innovation. Treating data as a production factor highlights its core role in creating new economic value, driving business model innovation, and promoting social progress. Data, through participation in the production process, creates value together with capital, labor, and other factors. However, in actual operation, how to accurately evaluate the value of data is still a challenge. Existing evaluation methods often rely on superficial data analysis, ignoring the potential value and complexity behind the data. Secondly, the circulation and value realization of data are restricted by multiple factors such as technology, policy, and law, and the existing framework and technical support are insufficient to cope with these challenges, resulting in the effective release of data value being hindered. In addition, the problem of data security and privacy protection is also an important aspect that existing methods cannot fully solve, and the existence of these problems seriously affects the value evaluation of data after factorization.

[0004] Second, data is regarded as an independent asset, emphasizing its value as a resource and potential as an asset. In the context of the digital era, the magnitude and value of data have undergone a qualitative leap, and data has become an asset that can be quantified, traded, and circulated. The management of data assets covers two stages of data resourceization and data assetization. Data resourceization is the basis for assessing data value, focusing on improving data quality and ensuring data security, including data governance, data model management, data standard management, and other aspects. Data assetization is the process of converting data resources into assets that can generate specific economic benefits for enterprises or organizations, including data trading and circulation, data asset operation, and data value assessment. The goal of data assetization is to release the potential value of data resources, expand the application range of data assets, and form a virtuous feedback loop between data supply and demand. Currently, the value assessment of data as an asset faces multiple challenges, including the improvement of data quality, the guarantee of data security, and the selection of data assetization paths. As the role of data in the economic and social spheres becomes increasingly prominent, how to efficiently and securely manage and utilize these data assets has become a key factor in promoting the development of the digital economy. However, there is currently no scientific and reasonable data asset value assessment system.

[0005] The two mainstream data value assessment methods mentioned above have encountered several major problems in practice. First, data is highly complex, with diverse types, extensive sources, and a large amount of redundant information, which greatly increases the complexity of the assessment work. Second, there is a lack of a universal assessment standard. Different fields have different understandings and assessment standards for data value, and there is currently no universally accepted evaluation system. Third, the dynamic and time-sensitive nature of data value is a challenge, as data value changes over time, usage environment, and purpose, requiring assessment methods to dynamically adapt to these changes. Finally, existing assessment methods are often single and rely heavily on specific scenarios or data types, lacking universality and flexibility. Therefore, there is an urgent need to find a data value assessment method that can solve some of the above problems. SUMMARY

[0006] In view of the above problems in the prior art, the present application proposes a data value assessment method and system based on information entropy measure.

[0007] To achieve the above-mentioned purposes, the technical solution of the present application is as follows:

[0008] The first aspect of the present application proposes a data value assessment method based on information entropy measure, comprising the following steps:

[0009] Step 1: Obtain the data set to be evaluated, establish a probability model for each data set, select a parameter estimation method according to the determined model complexity and the size of the data amount, and use the estimated model parameters to estimate the probability of occurrence of these data points;

[0010] Step 2: For each data set, use the information function to calculate the information entropy of the data set;

[0011] Step 3: Through the calculated information entropy, the average information amount or uncertainty of the entire data set is evaluated, and the value of each data set is obtained.

[0012] Further, the step 1 specifically comprises:

[0013] Step 1.1: Preprocess each data set obtained to ensure data quality, and then identify the type of each data set to determine whether the data set is a continuous data set or a discrete data set;

[0014] Step 1.2: Analyze the data type and distribution characteristics of each data set after preprocessing, and select a probability model according to the following characteristics:

[0015] For the processing method of discrete data sets, including but not limited to:

[0016] The binomial distribution model is used for the case where there are only two possible results for each observation, where the probability of each result is fixed and unchanged, with the characteristics of fixed total number and known single result probability, and its probability calculation formula is

[0017]

[0018] P(X=k) represents the probability of data k appearing in n groups of data, and p is the single observation probability of the event;

[0019] The Poisson distribution model is used for the condition that the number of events in a certain time interval or space range is independent and has a fixed average generation rate, and its probability calculation formula is

[0020]

[0021] Where λ p is the average occurrence rate of data, indicating the average frequency of event occurrence per unit time or unit space;

[0022] The Markov model is used to predict the probability of each data point in a series of data, and the probability of the next state of these data points only depends on its current state, showing the property of no memory or Markov property, X t is the state at time t, the state space S={s1, s2, s3,..., s M}, M is the total number of states, and the state transition probability is

[0023] P ij = P(X t+1 = s j |X t = s i )(1≤i,j≤M) (3);

[0024] For the processing mode of continuous data set, including but not limited to:

[0025] Normal distribution (Gaussian distribution) model, used for the characteristic that data is symmetrically distributed around the center value, and its probability function is

[0026]

[0027] Where μ is the mean, and σ is the standard deviation;

[0028] Lognormal distribution model, used for the case that all data points are positive and the distribution is obviously right-biased, when the natural logarithm of the random variable follows the normal distribution, and its probability function is

[0029]

[0030] Where μ is the mean, and σ is the standard deviation;

[0031] Exponential distribution model, used for the case that events occur randomly and independently in continuous time or space, and the probability of an event occurring in a certain time period in the future is independent of the time it has been waiting or has occurred so far, and its probability function is

[0032]

[0033] Where λ e is the average occurrence rate of data, indicating the frequency of time occurrence between events, i.e. the average number of events occurring per unit time;

[0034] Gamma distribution model, suitable for analyzing and describing the condition of the total waiting time or cumulative amount of multiple independent and similar events occurring in a fixed time period or spatial interval, and its probability function is

[0035]

[0036] Γ(α)=(α-1)! (8)

[0037] Where α is the shape parameter of the distribution, β is the scale parameter affecting the width or dispersion of the distribution, and Γ(α) is the gamma function;

[0038] Step 1.3: Select the parameter estimation method according to the following characteristics:

[0039] For the binomial distribution model and the Poisson distribution model of discrete data, the maximum likelihood estimation method (MLE) is used for parameter estimation. The parameter estimation of the binomial distribution model is

[0040]

[0041] where x is the number of times the observed data appears, and n is the total number of data;

[0042] The parameter estimation of the Poisson distribution model

[0043]

[0044] where x i is the number of times data is generated within each unit, and n is the total number of data;

[0045] For the normal distribution model of continuous data, the moment estimation method is applied, and the distribution parameters are estimated based on the sample moments, where the estimate of the mean μ is

[0046]

[0047] The estimate of the variance σ 2 is

[0048]

[0049] where x i is the sample data point, and n is the total number of sample data points;

[0050] For the lognormal distribution model of continuous data, the parameter estimation method of the normal distribution is applied after taking the logarithm of the data, and the estimate of the mean μ after logarithmic transformation is

[0051]

[0052] The estimate of the logarithmic variance σ 2 is

[0053]

[0054] where x i is the sample data point, and n is the total number of sample data points;

[0055] For the exponential distribution model and the gamma distribution model of continuous data, the maximum likelihood estimation method (MLE) is selected, and the average of the reciprocals of the observed data intervals

[0056]

[0057] where is the sample mean;

[0058] and the parameters a and b that maximize the probability of the given sample data, the log-likelihood function of the gamma distribution is

[0059]

[0060] For Markov models, the EM algorithm or Bayesian estimation is used, where the EM algorithm iteratively estimates the model parameters by alternately performing an expectation step (E-step) and a maximization step (M-step), the E-step computes the conditional probability distribution of the hidden states given the observed data and the current parameter estimates, i.e., the probability of each hidden state given the observed data and the current parameter estimates, the M-step updates the parameter estimates using the probability distribution of the hidden states computed in the E-step to maximize the likelihood function, the E-step and the M-step are repeated until the parameter estimates converge or a certain termination condition is met, and the final parameter values that maximize the likelihood function are obtained;

[0061] Bayesian estimation, on the other hand, estimates the parameters by combining prior knowledge and observed data, providing the posterior distribution of the parameters, which is

[0062] P(θ|X) = (L(θ|X) · P(θ)) / P(X) (17)

[0063] where P(θ|X) is the posterior distribution of the parameters θ, P(θ) is the prior distribution, L(θ|X) is the likelihood function, and P(X) is the marginal probability of the observed data;

[0064] Step 1.4: Apply the selected parameter estimation method: evaluate the effectiveness and fitting degree of the model according to the following characteristics:

[0065] Through the evaluation results under different prediction models, the most suitable model is obtained for fitting;

[0066] Through the Akaike information criterion (AIC)

[0067]

[0068] where m is the number of model parameters, is the maximum likelihood value of the model;

[0069] Through the Bayesian information criterion (BIC)

[0070]

[0071] where n is the sample size, m is the number of model parameters, is the maximum likelihood value of the model;

[0072] Step 1.5: Using the estimated model parameters, further determine the probability of the selected probability model according to its characteristics; for discrete data sets, use formulas (1), (2), (3) to calculate the probability of a specific result; for continuous data sets, use formulas (4), (5), (6), (7) to calculate the probability density of a data point around a certain value.

[0073] Further, in step 1.1, preprocessing includes handling missing values, outliers and noisy data.

[0074] Further, the step 2 includes:

[0075] Step 2.1: For each data point in the data set, calculate its self-information according to its occurrence probability; according to Shannon's definition, the self-information of a specific event can be calculated by the prior probability of the occurrence of the event, and its mathematical expression is

[0076]

[0077] where (Pa i ) is the prior probability of the occurrence of event a i , I(a i ) represents the amount of information contained in the occurrence of event a i ;

[0078] Step 2.2: Calculate the information entropy of the entire data set by taking the expected value of the self-information of all possible events in the data set, and its mathematical expression is

[0079] H(X) = E(I(a i )) (21)

[0080] where H(X) is the information entropy of the data set, and E represents the overall expectation function;

[0081] In addition, for a data set, the general form is

[0082]

[0083] where the data set X i has r rows of data, each row of data has d attributes, the i-th row of data is denoted as i = (x i1 , x i2 ,..., x id ), the j-th column of data is denoted as j = (x 1j , x 2j ,..., x rj ), and for a single attribute a j , there are v values, then the information entropy calculation formula of this form of data set is

[0084]

[0085] where P(a ji ) is the probability of the jth attribute in the data set taking the ith value, and its formula is

[0086]

[0087] Further, the step 3 includes:

[0088] By calculating the information entropy, using the merge sort algorithm to sort the information entropy of each data set, according to the sorting result, the information entropy of each data set and their corresponding information entropy are filled into the database table, the information entropy data is extracted from the database, and then the following statistical indicators are calculated:

[0089] The mean (Mean) calculation formula is

[0090]

[0091] The standard deviation (Standard Deviation) calculation formula is

[0092]

[0093] Where H(X i ) represents the information entropy of the ith sub-data set, and n is the total number of sub-data sets;

[0094] The mean plus double standard deviation is used to determine the threshold, and the calculation formula is:

[0095] Threshold=Mean+z×Standad Deviatiom (28)

[0097] Where z is an adjustable coefficient;

[0098] Or use the percentile method to determine the threshold according to the quantile of the data, select a specific percentile, so that the sub-data set above the threshold represents the most valuable part of the entire data set;

[0099] Using the determined threshold to query and mark all high-value sub-data sets in the database.

[0100] The second aspect of the present application provides a data value evaluation system based on information entropy measure, comprising

[0101] The data acquisition unit is used to acquire the data set to be evaluated, to establish a probability model for each data set, to select a parameter estimation method according to the determined model complexity and the size of the data amount, and to estimate the probability of the occurrence of these data points using the estimated model parameters.

[0102] a data computing unit configured to calculate, for each data set, an information entropy of the data set using an information quantity function;

[0103] a data evaluating unit configured to evaluate a value of an average information quantity or uncertainty of the whole data set by the calculated information entropy, and obtain a value of each data set.

[0104] The third aspect of the present application provides an electronic device, comprising a processor and a memory connected with the processor and used for storing executable instructions of the processor, and the processor is used for executing the data value evaluation method based on the information entropy measure.

[0105] The fourth aspect of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the data value evaluation method based on the information entropy measure.

[0106] Compared with the prior art, the data value evaluation method and system based on the information entropy measure has the following advantages:

[0107] The method of the present application does not directly depend on the specific content of the data, but depends on the overall distribution and probability characteristics of the data, so it can effectively solve the problem of high data complexity, and identify and filter out redundant information. BRIEF DESCRIPTION OF DRAWINGS

[0108] The accompanying drawings, which form a part of the present application, are intended to provide further understanding of the present application and serve to explain the illustrative embodiments of the present application together with their descriptions. The present application is not limited by the illustrative embodiments in the accompanying drawings.

[0109] Figure 1 FIG. 1 is a flow chart of a data value evaluation method based on an information entropy measure according to the present application. DETAILED DESCRIPTION

[0110] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0111] In the description of the present application, it needs to be understood that the terms "center", "longitudinal", "transverse", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship shown in the drawings, which are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second" and the like are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined with "first", "second" and the like can be explicitly or implicitly included one or more. In the description of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0112] In the description of the present application, it needs to be understood that unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connecting" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium, or it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood through specific circumstances.

[0113] The present application will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0114] Example one:

[0115] In the data value evaluation process, the information entropy principle method measures the value of the data set by quantifying the average amount of information of the entire data set. The information entropy of the entire data set reflects its overall uniqueness and increased information, which is closely related to the probability distribution of the events in the data set. As a measure based on probability distribution, information entropy naturally adapts to various types and sources of data sets. This method does not directly depend on the specific content of the data, but depends on the overall distribution and probability characteristics of the data, so it can effectively solve the problem of high data complexity, and at the same time identify and filter out redundant information.

[0116] Information entropy provides a well-defined evaluation standard that does not depend on any prior knowledge or evaluation standard in a specific field. This makes information entropy a universal evaluation tool that can be applied to different fields and various data types. Although the information entropy principle method itself may need to be fine-tuned according to the specific circumstances of different fields, its basic principles and calculation methods provide a common foundation for data value evaluation, effectively solving the problem of lack of universal evaluation standard and overcoming the limitations of existing evaluation methods.

[0117] The information entropy principle can dynamically adapt to the change of data value, which is based on the probability distribution of event occurrence in the entire data set, and these probabilities can change over time, usage environment and purpose. By periodically recalculating the information entropy of the entire data set, the evaluation method can reflect the real-time change of data value, providing a dynamic data value evaluation. This effectively solves the problem of data value changing based on its dynamic and timeliness. To achieve the above purpose, the present application proposes a data value evaluation method based on information entropy measure, which is used for the evaluation of data value based on information entropy principle, and the overall process is as shown in Figure 1

[0118] A data value evaluation method based on information entropy principle, comprising the following steps:

[0119] Step 1: Establish a probability model for each data set, select a parameter estimation method according to the determined model complexity and the size of the data amount, and use the estimated model parameters to estimate the probability of occurrence of these data points;

[0120] Step 2: For each data set, use the information function to calculate the information entropy of the data set;

[0121] Step 3: Through the calculated information entropy, the average information amount or uncertainty of the entire data set is evaluated, and the value of each data set is obtained.

[0122] Further, for step 1,

[0123] 1. First, pre-process each data set collected, clean the collected data set, including processing missing values, abnormal values and noise data, to ensure data quality. Further, identify the type of each data set, determine whether the data set is a continuous data set or a discrete data set. The key of this step is to identify the suitable analysis model for the data. Next, we check the distribution characteristics of the data, such as central tendency (mean, median), dispersion (variance, standard deviation), skewness (symmetry of data distribution) and kurtosis (sharpness of data distribution), etc., to help determine the type of distribution that the data may follow.

[0124] 2. Based on the pre-processing of the data, according to the type and distribution characteristics of the data, select the probability model according to the following criteria:

[0125] For data sets, discrete data sets (discrete database) and continuous data sets (continuous database) are two types used to describe different data set attributes. Discrete data set refers to data in the data set that can only take specific, finite or countably infinite values. Continuous data set refers to data in the data set that can take any value within a continuous range.​

[0126] 2.1 Discrete Data Set

[0127] 2.1.1 Binomial Distribution Model

[0128] For a discrete data set that satisfies the condition that each observation has only two possible outcomes, and the probabilities of these two outcomes are fixed, a binomial distribution model can be used for analysis and calculation. The formula is

[0129]

[0130] where P(X=k) represents the probability of data k appearing in n groups of data. p is the probability of data k, and (1-p) is the probability of data not being k.

[0131] 2.1.2 Poisson Distribution Model

[0132] For a discrete data set that satisfies the condition that the number of data generated within a specific time interval or spatial range is independent and has a fixed average generation rate, a Poisson distribution model can be used for analysis. The formula is

[0133]

[0134] where P(X=k) represents the probability of data generating exactly k times within a given time interval, λ p is the average occurrence rate of data, indicating the average frequency of events occurring per unit time or unit space.

[0135] 2.1.3 Markov Model

[0136] For a discrete data set, when predicting the probability of each data point in a series of data, and the probability of the next state of these data points only depends on its current state, and has nothing to do with the previous state (showing the property of no memory or Markov property), Markov model can be used. Where X t is the state at time t, the state space S = {s1, s2, s3, …, s M}M is the total number of states, and the state transition probability is

[0137] P ij = P(X t+1 = s j |X t = s i )(1≤i, j≤M) (3)

[0138] 2.2 Continuous Data Set

[0139] 2.1.1 Normal Distribution (Gaussian Distribution) Model

[0140] For continuous data sets that satisfy the characteristic of symmetric distribution around the central value, the normal distribution (Gaussian distribution) model can be used for analysis. Its formula is

[0141]

[0142] where μ is the mean and σ is the standard deviation.

[0143] 2.1.2 Lognormal distribution model

[0144] For continuous data sets that satisfy the condition that all data points are positive and the distribution is obviously right-skewed (i.e., with long tail characteristics), the lognormal distribution model can be used. This model is suitable for cases where the natural logarithm of a random variable follows a normal distribution. Its formula is

[0145]

[0146] where μ is the mean and σ is the standard deviation.

[0147] 2.1.3 Exponential distribution model

[0148] For continuous data sets that satisfy the condition that data points occur randomly and independently over continuous time or space, and the probability of a data point occurring within a certain time period in the future is independent of the time it has already waited or occurred so far (showing no memory), the exponential distribution model can be used for analysis. Its formula is

[0149]

[0150] where λ e is the average occurrence rate of data, indicating the frequency of time between events, i.e., the average number of events per unit time.

[0151] 2.1.4 Gamma distribution model

[0152] For continuous data sets that satisfy the condition of analyzing and describing the total waiting time or cumulative amount of multiple independent and similar events occurring within a fixed time period or spatial interval, the gamma distribution model can be used. This model is suitable for cumulative event occurrence situations, especially when the focus is on the total number of events or the total waiting time of events. Its formula is

[0153]

[0154] where α is the shape parameter of the distribution, β is the scale parameter affecting the width or spread of the distribution, and Γ(α) is the gamma function, for integer α,

[0155] T(α) = (α-1)! (8)

[0156] 3. Select the parameter estimation method according to the determined model complexity and the size of the data set.

[0157] 3.1 Parameter estimation for discrete data set

[0158] For binomial distribution model and Poisson distribution model, maximum likelihood estimation (MLE) is selected. MLE estimates the model parameters by maximizing the likelihood function of the observed data

[0159]

[0160] where x is the number of times the data is observed, n is the total number of data.

[0161]

[0162] where x i is the number of times the data is generated within each unit, i.e. the observed data points, n is the total number of data.

[0163] 3.2 Parameter estimation for continuous data set

[0164] For normal distribution model of continuous data set, moment estimation method is applied to estimate the distribution parameters based on sample moments, which contains two parameters, where the estimation of mean μ is

[0165]

[0166] The estimation of variance σ 2 is

[0167]

[0168] where x i is the sample data points, n is the total number of sample data points.

[0169] For parameter estimation of lognormal distribution model of continuous data set, the parameter estimation method of normal distribution is applied after taking the logarithm of the data, and the estimation of mean μ of the logarithmic transformation is

[0170]

[0171] The estimation of log variance σ 2 is

[0172]

[0173] where x i is the sample data points, n is the total number of sample data points.

[0174] For the exponential distribution model of continuous data sets, the maximum likelihood estimation method (MLE) is chosen to observe the average of the reciprocals of the data intervals

[0175]

[0176] where is the sample mean;

[0177] For the gamma distribution model of continuous data sets, the maximum likelihood estimation method (MLE) is chosen to find the parameters a and β that maximize the probability of the given sample data. The log-likelihood function of the gamma distribution for a given data sample is

[0178]

[0179] Take the partial derivative of l(a, β) with respect to a and β to get a set of equations. Use numerical optimization methods to solve the equations, starting from initial estimates, and iteratively optimize the estimates of a and β until convergence to the maximum likelihood estimates.

[0180] For Markov models and more complex models with hidden variables, the EM algorithm or Bayesian estimation is used.

[0181] The EM algorithm iteratively estimates model parameters by alternating between the expectation step (E-step) and the maximization step (M-step). It is used to find the parameter estimates that maximize the likelihood function in the presence of hidden variables. First, choose an initial estimate of the parameters,

[0182] E-step (Expectation): Based on the current parameter estimates, calculate the conditional probability distribution of the hidden states, i.e., the probability of each hidden state given the observed data and the current parameter estimates.

[0183] M-step (Maximization): Using the probability distribution of the hidden states calculated in the E-step, update the parameter estimates to maximize the likelihood function. This usually involves finding the maximum of the likelihood function with respect to the parameters.

[0184] Repeat the E-step and M-step until the parameter estimates converge or a certain termination condition is met, ultimately obtaining the parameter values that maximize the likelihood function.

[0185] Bayesian estimation estimates parameters by combining prior knowledge and observed data. First, for the state transition probabilities P ij Define a prior distribution, which represents the probability of observing the data X given the parameters θ through the likelihood function L(θ|X) (equations (1)(2)(4)). Then apply Bayes' theorem to combine the prior distribution and the likelihood function to provide the posterior distribution of the parameters, which is calculated

[0186] P(0|X) = (L(0|X) - P(0)) / P(X) (17)

[0187] where P(0|X) is the posterior distribution of parameter 0, P(0) is the prior distribution, L(0|X) is the likelihood function, P(X) is the marginal probability of the observed data, as a normalization constant.

[0188] 4. Then apply the selected parameter estimation method to evaluate the effectiveness and fitting degree of the model.

[0189] Through the evaluation results under different prediction models, the most suitable model is obtained for fitting.

[0190] Through the Akaike information criterion (AIC)

[0191]

[0192] where m is the number of model parameters, is the maximum likelihood value of the model.

[0193] Through the Bayesian information criterion (BIC)

[0194]

[0195] where n is the sample size, m is the number of model parameters, is the maximum likelihood value of the model.

[0196] Calculate the Akaike information criterion (AIC) and Bayesian information criterion (BIC) values to compare the fitting degree and complexity of different models, and select the most suitable model.

[0197] 5. Finally, using the estimated model parameters, determine the probability mass function (PMF) or probability density function (PDF) of the selected probability model to calculate the probability of each data point.

[0198] For discrete data sets, use PMF (as shown in equations (1), (2), (3)) to calculate the probability of a specific result.

[0199] For continuous data sets, use PDF (as shown in equations (4), (5), (6), (7)) to calculate the probability density of a data point around a certain value.

[0200] Specifically, in step 2, first, for each data point in the data set, calculate its self-information according to its occurrence probability. According to Shannon's definition, the self-information of a specific event can be calculated by the prior probability of the event occurring, and its mathematical expression is

[0201]

[0202] where P(a i ) is the prior probability of event a i occurrence, I(a i ) represents the amount of information contained in the occurrence of event a i , i.e. the self-information.

[0203] Then the information entropy of the whole data set is obtained by calculating the expected value of the self-information of all possible events in the data set, and its mathematical expression is

[0204] H(X) = E(I(a i )) (21)

[0205] where H(X) is the information entropy of the data set, and E represents the overall expectation function.

[0206] In addition, for the data set, the general form is

[0207]

[0208] where the data set X i has r rows of data, each row of data has d attributes, the i-th row of data is denoted as i = (x i1 , x i2 ,..., x id ), the j-th column of data is denoted as j = (x 1j , x 2j ,..., x rj ), and for the value of single attribute a j , there are v values, then the information entropy calculation formula of the data set of this form is

[0209]

[0210] where p(a ji ) is the probability of the j-th attribute taking the i-th value in the data set, and its formula is

[0211]

[0212] Specifically, in step 3, the information entropy obtained by calculation not only reflects the average uncertainty of the whole data set, but also provides a method for measuring the intrinsic value of the data set. First, use the efficient merge sort algorithm to sort the information entropy of each data set. Then according to the sorting result, fill each data set and their corresponding information entropy into the database table in turn. Then extract the information entropy data from the database, and then calculate the following statistical indicators:

[0213] Mean (Mean): is the average value of the information entropy of the data set, and the calculation formula is

[0214]

[0215] Standard Deviation: is a statistical measure of the dispersion of information entropy distribution, the formula is

[0216]

[0217] Where H(X i ) represents the information entropy of the i-th sub-data set, and n is the total number of sub-data sets.

[0218] According to the calculated statistical indicators, the characteristics of the data and the specific application requirements to select the dynamic threshold.

[0219] Using the mean plus double standard deviation to determine the threshold, the formula is:

[0220] Threshold = Mean + z x Standard Deviation (28)

[0221] Where z is an adjustable coefficient, usually selected as 1, 1.5 or 2, according to the distribution of data and the degree of rarity of the data points to be identified.

[0222] Or using the percentile method (Percentile-based Thresholding), according to the quantile of the data to determine the threshold. Select a specific percentile, so that the sub-data set above the threshold represents the most valuable part of the entire data set.

[0223] Finally, using the determined threshold to query and mark all high-value sub-data sets in the database. The list of sub-data sets marked as high-value can be used for further analysis, decision support or as the basis for feature selection. The high or low of information entropy can be used as an indicator to evaluate the value of the data set. We can get a measure representing the average uncertainty of the entire data set, and then evaluate the value of the data set.

[0224] Example two:

[0225] A data value evaluation system based on information entropy measure, comprising

[0226] A data acquisition unit for acquiring data sets to be evaluated, establishing a probability model for each data set, selecting a parameter estimation method according to the determined model complexity and the size of the data amount, and using the estimated model parameters to estimate the probability of the occurrence of these data points;

[0227] A data calculation unit for calculating the information entropy of each data set using the information function;

[0228] a data evaluation unit configured to evaluate the value of each data set by calculating the average information amount or uncertainty of the whole data set through the calculated information entropy.

[0229] Embodiment three:

[0230] An electronic device, comprising a processor and a memory connected with the processor and used for storing executable instructions of the processor, wherein the processor is configured to execute the data value evaluation method based on the information entropy measure.

[0231] Embodiment four:

[0232] A computer readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement the data value evaluation method based on the information entropy measure.

[0233] The above is not exhaustive of the prior art.

[0234] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A data value evaluation method based on information entropy measure, characterized in that: The method comprises the following steps: Step 1: obtaining a data set to be evaluated, establishing a probability model for each sub-data set in the data set, selecting a parameter estimation method according to the determined model complexity and the size of the data amount, and estimating the probability of occurrence of data points in each sub-data set using the estimated model parameters; The obtaining of the data set to be evaluated and the establishment of the probability model for each sub-data set in the data set specifically comprise: Pretreating each obtained sub-data set to ensure data quality, and then identifying the type of each sub-data set to determine whether the data set is a continuous data set or a discrete data set; Analyzing the data type and distribution characteristics of each pretreated sub-data set, and selecting a probability model according to the following characteristics: for the processing mode of a discrete data set, including but not limited to: a binomial distribution model, a Poisson distribution model, and a Markov model; for the processing mode of a continuous data set, including but not limited to: a normal distribution model, a lognormal distribution model, an exponential distribution model, and a gamma distribution model; the selection of the parameter estimation method according to the determined model complexity and the size of the data amount specifically comprises: For the binomial distribution model and the Poisson distribution model of discrete data, a maximum likelihood estimation method (MLE) is used for parameter estimation; for the Markov model, an EM algorithm or a Bayesian estimation is used, For the normal distribution model of continuous data, a moment estimation method is applied to estimate the distribution parameters based on sample moments; For the lognormal distribution model of continuous data, the parameter estimation method of the normal distribution is applied after taking the logarithm of the data; For the exponential distribution model and the gamma distribution model of continuous data, a maximum likelihood estimation method is selected; Step 2: for each sub-data set, an information entropy of each sub-data set is calculated using an information amount function, including calculating the self-information amount of each data point in each sub-data set according to the probability of occurrence thereof, and calculating the information entropy of each sub-data set by taking the expected value of the self-information amount of all possible events in each sub-data set; Step 3: the average information amount or uncertainty of the entire data set is evaluated in value based on the calculated information entropy, and the value of each sub-data set is obtained; The step 3 comprises: The information entropy of each sub-data set is sorted using a merge sort algorithm based on the calculated information entropy, each sub-data set and its corresponding information entropy are filled into a database table in sequence according to the sorting result, the information entropy data is extracted from the database, and then the mean value and the standard deviation are calculated; the threshold value is determined using the mean value plus twice the standard deviation, and all high-value sub-data sets are queried and marked in the database using the determined threshold value. 2.The data value evaluation method based on information entropy measure of claim 1, characterized in that: In the step 1: The binomial distribution model is used for the case where there are only two possible results for each observation, and the probability of each result is fixed and unchanged, with the characteristics of a fixed total number of times and a known single result probability, and the probability calculation formula is （1） P( = k) represents the probability of occurrence of data k in n sets of data, p is the single observation probability of the data; is a binomial distributed random variable; The Poisson distribution model is used for the condition that the number of events in a specific time interval or space range is independent and has a fixed average generation rate, and the probability calculation formula is （2）； where λ is the average occurrence rate of the data, indicating the average frequency of events occurring within a unit of time or space; denotes the probability weight of the expected occurrence of k data within a unit of time or space, k = 0, 1, 2,...; denotes a random variable of the Poisson distribution; Markov model, which is used to predict the probability of each data point in a series of data, and the probability of the next state of these data points only depends on its current state, showing the memoryless or Markov property, is the state at time t, the state space S = {s1, s2, s3,..., s M}, M is the total number of states, and the state transition probability is (3)； wherein is the transition probability of data from data point to data point , is the Markov chain state variable; For the processing mode of a continuous data set, including but not limited to: The normal distribution model is used for the characteristic that the data is symmetrically distributed around the center value, and the probability function is （4） where μ is the mean, σ is the standard deviation, and e is the natural constant; Lognormal distribution model, used for all data points are positive and the distribution is obviously right skewed, when the natural logarithm of the random variable follows a normal distribution, its probability function is （5）； a random variable representing a normal distribution; Exponential distribution model, used for events to occur randomly and independently in continuous time or space, and the probability of an event occurring in a certain time period in the future is independent of the time it has been waiting or has occurred so far, its probability function is （6） a random variable representing an exponential distribution; Gamma distribution model, suitable for analyzing and describing the conditions of the total waiting time or cumulative amount of multiple independent and similar events occurring within a fixed time period or spatial interval, its probability function is Γ(α) = (α - 1)! (8) where a is a shape parameter of the distribution, is a scale parameter that influences the width or spread of the distribution, and Γ(a) is the gamma function; Parameter estimation of binomial distribution model （9） wherein is the number of times the data is observed to occur, n is the total number of data; Parameter estimation of Poisson distribution model （10） wherein is the total number of data; wherein i is the observation interval, i = 1, 2, …, wherein denotes the number of times of data generation within the i-th unit; For the method of moment estimation, the estimation of the mean μ is （11） Variance σ 2 is estimated as （12） wherein is a sample data point, is the total number of sample data points; For the lognormal distribution model of continuous data, the parameter estimation method of the normal distribution is applied after taking the logarithm of the data, and the estimation of the mean μ of the logarithmic transformation is （13） Logarithmic variance σ 2 is estimated as （14） For the exponential distribution model and the gamma distribution model of continuous data, the maximum likelihood estimation method was selected, and the average value of the reciprocal of the observation data interval was calculated wherein is the sample mean; The log-likelihood function of the gamma distribution is （16） For the Markov model, the EM algorithm or Bayesian estimation is used, where the EM algorithm iteratively estimates the model parameters by alternately performing the expectation step E step and the maximization step M step, the E step calculates the conditional probability distribution of the hidden state based on the current parameter estimation, the M step updates the parameter estimation to maximize the likelihood function using the probability distribution of the hidden state calculated in the E step, and the E step and the M step are repeated until the parameter estimation converges or meets a certain termination condition, and finally the parameter value that maximizes the likelihood function is obtained; Bayesian estimation estimates the parameters by combining prior knowledge and observed data, and provides the posterior distribution of the parameters, which is P(θ|X) = (L(θ|X)·P(θ)) / P(X) (17) where P(θ|X) is the posterior distribution of the parameter θ, P(θ) is the prior distribution, L(θ|X) is the likelihood function, and P(X) is the marginal probability of the observed data; The step 1 further includes Applying the selected parameter estimation method: evaluating the effectiveness and fitting degree of the model according to the following characteristics: Through the evaluation results under different prediction models, the most suitable model is obtained for fitting; Through the Akaike information criterion AIC where m is the number of model parameters, is the maximum likelihood value of the model; Through the Bayesian information criterion BIC where n is the sample size and m is the number of model parameters, is the maximum likelihood value of the model. Using the estimated model parameters, the probability of the selected probability model is determined according to its characteristics; for discrete data sets, formulas (1), (2), and (3) are used to calculate the probability of a specific result; for continuous data sets, formulas (4), (5), (6), and (7) are used to calculate the probability density of the data points around a certain value. 3.The data value evaluation method based on information entropy measure of claim 1, characterized in that: In step 1, preprocessing includes handling missing values, outliers, and noise data.

4. The data value evaluation method based on information entropy measure according to claim 1, characterized in that: In step 2: Calculating the self-information of each data point in each sub-data set according to its occurrence probability includes: according to Shannon's definition, the self-information of a specific event is calculated by the prior probability of the event, and its mathematical expression is For the lognormal distribution model of continuous data, the parameter estimation method of the normal distribution is applied after taking the logarithm of the data, and the estimation of the mean μ of the logarithmic transformation is The log-likelihood function of the gamma distribution is For the Markov model, the EM algorithm or Bayesian estimation is used, where the EM algorithm iteratively estimates the model parameters by alternately performing the expectation step E step and the maximization step M step, the E step calculates the conditional probability distribution of the hidden state based on the current parameter estimation, the M step updates the parameter estimation to maximize the likelihood function using the probability distribution of the hidden state calculated in the E step, and the E step and the M step are repeated until the parameter estimation converges or meets a certain termination condition, and finally the parameter value that maximizes the likelihood function is obtained; Bayesian estimation estimates the parameters by combining prior knowledge and observed data, and provides the posterior distribution of the parameters, which is P(θ|X) = (L(θ|X)·P(θ)) / P(X) (17) where P(θ|X) is the posterior distribution of the parameter θ, P(θ) is the prior distribution, L(θ|X) is the likelihood function, and P(X) is the marginal probability of the observed data; The step 1 further includes Applying the selected parameter estimation method: evaluating the effectiveness and fitting degree of the model according to the following characteristics: Through the evaluation results under different prediction models, the most suitable model is obtained for fitting; Through the Akaike information criterion AIC Through the Bayesian information criterion BIC Using the estimated model parameters, the probability of the selected probability model is determined according to its characteristics; for discrete data sets, formulas (1), (2), and (3) are used to calculate the probability of a specific result; for continuous data sets, formulas (4), (5), (6), and (7) are used to calculate the probability density of the data points around a certain value. In step 1, preprocessing includes handling missing values, outliers, and noise data. In step 2: Calculating the self-information of each data point in each sub-data set according to its occurrence probability includes: according to Shannon's definition, the self-information of a specific event is calculated by the prior probability of the event, and its mathematical expression is where P(a i ) is the prior probability of event a i occurring, and I(a i ) represents the amount of information contained in the occurrence of event a i . The information entropy of each sub-data set is obtained by calculating the expected value of the self-information of all possible events in each sub-data set, and the mathematical expression is H(X) = E(I(a i )) (21) Where H(X) is the information entropy of the sub-data set, and E represents the overall expected function. In addition, for the data set, the form is Wherein, the data set X has r rows of data, each row of data has d attributes, the i-th row of data is denoted as X i = (x i1 , x i2 ,..., x id ), the j-th column of data is denoted as X j = (x 1j , x 2j ,..., x rj ), for the value of a single attribute A j v {a1, a2,..., av}, then the information entropy of the j-th attribute column of the form data set is The calculation formula is （23） P (a ji ) is the probability of the jth attribute in the data set taking the ith value, and its formula is （24） （25）； is the hth value in the form data set, is the A j value is probability that is the data value in the ith row and jth column, is the indicator function that takes the value 1 when the data value in the ith row and jth column is equal to and 0 otherwise.

5. The data value evaluation method based on information entropy measure according to claim 1, characterized in that: The mean Mean of step 3 is calculated by the formula The standard deviation Standard Deviation is calculated by the formula The threshold value is determined by using the mean plus double standard deviation, and the calculation formula is: where H(X i ) represents the information entropy of the ith sub-data set, and n is the total number of sub-data sets. Where z is an adjustable coefficient. (28) Or using the percentile method, according to the quantile of the data, a specific percentile is selected, so that the sub-data set above the threshold value represents the most valuable part of the whole data set. It includes 6. A data value evaluation system based on information entropy measure, characterized in that: The data acquisition unit is used to acquire the data set to be evaluated, establish a probability model for each sub-data set in the data set, select a parameter estimation method according to the determined model complexity and the size of the data amount, and estimate the probability of each data point in each sub-data set using the estimated model parameters. The data set to be evaluated is acquired, and a probability model is established for each sub-data set in the data set, which specifically includes: The acquired each sub-data set is preprocessed to ensure data quality, and then the type of each sub-data set is identified to determine whether the data set is a continuous data set or a discrete data set; The data type and distribution characteristics of each preprocessed sub-data set are analyzed, and a probability model is selected according to the following characteristics: for the processing method of discrete data set, including but not limited to: binomial distribution model, Poisson distribution model, Markov model; for the processing method of continuous data set, including but not limited to: normal distribution model, lognormal distribution model, exponential distribution model, gamma distribution model; the selection of parameter estimation method according to the determined model complexity and the size of the data amount specifically includes: For the binomial distribution model and Poisson distribution model of discrete data, maximum likelihood estimation MLE is used for parameter estimation; for the Markov model, EM algorithm or Bayesian estimation is used, For the normal distribution model of continuous data, the moment estimation method is applied to estimate the distribution parameters based on the sample moments; For the lognormal distribution model of continuous data, the parameter estimation method of normal distribution is applied after taking the logarithm of the data; For the exponential distribution model and gamma distribution model of continuous data, maximum likelihood estimation is selected; The data calculation unit is used to calculate the information entropy of each sub-data set using the information function, including calculating the self-information of each data point in each sub-data set according to its probability, and obtaining the information entropy of each sub-data set by calculating the expected value of the self-information of all possible events in each sub-data set. The data evaluation unit is used to sort the information entropy of each sub-data set using the merge sort algorithm according to the calculated information entropy, fill each sub-data set and its corresponding information entropy into the database table according to the sorting result, extract the information entropy data from the database, and then calculate the mean and standard deviation; the threshold value is determined by using the mean plus double standard deviation, and the determined threshold value is used to query and mark all high-value sub-data sets in the database. ​ 7. An electronic device, comprising a processor and a memory connected to the processor in communication, and configured to store executable instructions of the processor, characterized in that: The processor is configured to execute the data value evaluation method based on the information entropy measure according to any one of claims 1-5.

8. A computer readable storage medium storing a computer program, characterized in that: The computer program is configured to implement the data value evaluation method based on the information entropy measure according to any one of claims 1-5 when executed by the processor.

Images