Data value evaluation method and system based on information entropy measurement

Through the information entropy measurement method, a probability model is established and the information entropy of the data set is calculated, which solves the problems of data complexity and lack of evaluation standards in the existing technology, and realizes the accuracy, flexibility and security assessment of data value.

CN118939934AActive Publication Date: 2024-11-12TIANJIN UNIV

Patent Information

Application Number
CN202411051629.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-01
Publication Date
2024-11-12
Estimated Expiration
2044-08-01

AI Technical Summary

Technical Problem

Existing data value assessment methods are difficult to effectively solve the problems of data complexity, lack of general evaluation standards, dynamics and timeliness, as well as data security and privacy protection, resulting in inaccurate and inflexible data value assessment.

Method used

Using a method based on information entropy measurement, we use a probability model to calculate the information entropy of the data set to evaluate its value, which is applicable to data of different types and sources, and dynamically adapts to the changes in data value.

Benefits of technology

It provides a general and dynamic data value evaluation standard that can effectively identify and filter redundant information, and is suitable for different fields and data types, ensuring the accuracy and flexibility of data value evaluation, while improving data security.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939934A_ABST
    Figure CN118939934A_ABST
Patent Text Reader

Abstract

The invention provides a data value evaluation method and system based on information entropy measure, and the method comprises the steps: obtaining to-be-evaluated data sets, building a probability model for each data set, selecting a parameter estimation method according to the determined model complexity and the size of the data size, and carrying out the evaluation of the data value. Estimating the probability of occurrence of the data points by using the estimated model parameters; for each data set, calculating the information entropy of the data set by using an information amount function; and carrying out value evaluation on the average information amount or uncertainty of the whole data set through the calculated information entropy to obtain the value of each data set. The method does not directly depend on the specific content of the data, but depends on the overall distribution and probability characteristics of the data, so that the problem of high data complexity can be effectively solved, and redundant information can be identified and filtered out at the same time.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and in particular relates to a data value assessment method and system based on information entropy measurement. Background Art

[0002] With the rapid development of information technology, the value of data in all walks of life has become increasingly prominent and has become an important resource. However, how to accurately evaluate the value of data, especially the value of data after it has been converted or processed, remains a challenge in this field. Currently, there are two main directions for measuring data value:

[0003] The first is to transform data into traditional production factors. In the context of the digital economy, data has become a key resource that goes hand in hand with traditional production factors, and its value and circulation have a profound impact on economic growth and innovation. Viewing data as a production factor highlights its core role in creating new economic value, driving business model innovation, and promoting social progress. By participating in the production process, data creates value together with factors such as capital and labor. However, in actual operations, how to accurately evaluate the value of data remains a challenge. Existing evaluation methods often rely on superficial data analysis, ignoring the potential value and complexity behind the data. Secondly, the circulation and value realization of data are restricted by multiple factors such as technology, policies, and laws. The existing framework and technical support are not sufficient to meet these challenges, resulting in the effective release of data value being hindered. In addition, the issues of data security and privacy protection are also important aspects that are difficult to fully address with existing methods. The existence of these problems seriously affects the value assessment of data after factorization.

[0004] Second, data is regarded as an independent asset, emphasizing its value as a resource and its potential as an asset. In the context of the digital age, the magnitude and value of data have made a qualitative leap, and data has become an asset that can be quantified, traded, and circulated. The management of data assets covers two stages: data resourceization and data assetization. Data resourceization is the basis for evaluating the value of data, focusing on improving the quality of data and ensuring data security, including data governance, data model management, data standard management, and other aspects. Data assetization is the process of transforming data resources into assets that can generate specific economic benefits for enterprises or organizations, including data transactions and circulation, data asset operations, and data value assessment. The goal of data assetization is to release the potential value of data resources, expand the scope of application of data assets, and form a benign feedback loop between data supply and demand. At present, the value assessment of data as an asset faces multiple challenges, including the improvement of data quality, the guarantee of data security, and the choice of data assetization path. As the role of data in the economy and society becomes increasingly prominent, how to efficiently and safely manage and utilize these data assets has become a key factor in promoting the development of the digital economy. Moreover, there is currently no scientific and reasonable data asset value assessment system.

[0005] The above two mainstream data value assessment methods have encountered several major problems in practice. First, the complexity of data is extremely high. Due to the diverse data types, wide sources, and a large amount of redundant information, this greatly increases the complexity of the assessment work. Secondly, there is a lack of a universal assessment standard. Different fields have different understandings and assessment standards for data value, and there is currently no universally accepted evaluation system. Furthermore, the dynamic and timeliness of data value is also a challenge. The value of data will change with time, usage environment and purpose, which requires the assessment method to be able to dynamically adapt to these changes. Finally, the existing assessment methods are often single, mostly dependent on specific scenarios or data types, and lack versatility and flexibility. Therefore, at this stage, it is urgent to find a data value assessment method that can solve some of the above problems. Summary of the invention

[0006] In view of this, the present invention aims to overcome the shortcomings of the above-mentioned problems in the prior art and proposes a data value assessment method and system based on information entropy measurement.

[0007] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0008] The first aspect of the present invention provides a data value assessment method based on information entropy measurement, comprising the following steps:

[0009] Step 1: Obtain the data set to be evaluated, establish a probability model for each data set, select a parameter estimation method based on the determined model complexity and data size, and use the estimated model parameters to estimate the probability of occurrence of these data points;

[0010] Step 2: For each data set, use the information function to calculate the information entropy of the data set;

[0011] Step 3: By calculating the information entropy, evaluate the average amount of information or uncertainty of the entire data set to obtain the value of each data set.

[0012] Furthermore, the step 1 specifically includes:

[0013] Step 1.1: Preprocess each acquired data set to ensure data quality, and then identify the type of each data set to determine whether the data set is a continuous data set or a discrete data set;

[0014] Step 1.2: Analyze the data type and distribution characteristics of each preprocessed data set, and select a probability model based on the following characteristics:

[0015] The processing methods for discrete data sets include but are not limited to:

[0016] The binomial distribution model is used when there are only two possible outcomes for each observation, where the probability of each outcome is fixed, with the characteristics of a fixed total number of times and a known probability of a single outcome. The probability calculation formula is:

[0017]

[0018] P(X=k) represents the probability of data k appearing in n sets of data, and p is the single observation probability of the event;

[0019] The Poisson distribution model is used for conditions where the number of events occurring within a specific time interval or spatial range is independent and has a fixed average generation rate. The probability calculation formula is

[0020]

[0021] where λ p It is the average occurrence rate of the data, indicating the average frequency of events occurring per unit time or unit space;

[0022] Markov model is used to predict the probability of each data point in a series of data, and the probability of the next state of these data points depends only on their current state, showing no memory or Markov property, X t is the state at time t, the state space S = {s1, s2, s3, ..., s M}, M is the total number of states, and its state transition probability is

[0023] P ij =P(X t+1 =s j |X t =s i )(1≤i,j≤M) (3);

[0024] The processing methods for continuous data sets include but are not limited to:

[0025] The normal distribution (Gaussian distribution) model is used for the characteristics of data symmetrically distributed around the central value, and its probability function is

[0026]

[0027] Where μ is the mean and σ is the standard deviation;

[0028] The lognormal distribution model is used when all data points are positive and the distribution is obviously right-skewed. When the natural logarithm of the random variable follows a normal distribution, its probability function is

[0029]

[0030] Where μ is the mean and σ is the standard deviation;

[0031] The exponential distribution model is used for events that occur randomly and independently in continuous time or space, and the probability of an event occurring in a certain period of time in the future is independent of the time it has waited or occurred so far. Its probability function is

[0032]

[0033] where λ e It is the average occurrence rate of the data, indicating the frequency of time between events, that is, the average number of times an event occurs per unit time;

[0034] The gamma distribution model is suitable for analyzing and describing the total waiting time or cumulative amount of multiple independent and similar events occurring in a fixed time period or spatial interval. Its probability function is

[0035]

[0036] Γ(α)=(α-1)! (8)

[0037] Where α is the shape parameter of the distribution, β is the scale parameter that affects the width or spread of the distribution, and Γ(α) is the gamma function;

[0038] Step 1.3: Select the parameter estimation method based on the following characteristics:

[0039] For the binomial distribution model and Poisson distribution model of discrete data, the maximum likelihood estimation (MLE) method is used for parameter estimation.

[0040]

[0041] Where x is the number of times the observed data occurs, and n is the total number of data;

[0042] Parameter Estimation of Poisson Distribution Model

[0043]

[0044] where x i It indicates the number of times data is generated in each unit, and n is the total number of data;

[0045] For the normal distribution model of continuous data, the moment estimation method is applied to estimate the distribution parameters based on the sample moments, where the mean μ is estimated as

[0046]

[0047] Variance σ 2 The estimate is

[0048]

[0049] where x i is the sample data point, n is the total number of sample data points;

[0050] For the lognormal distribution model of continuous data, the parameter estimation method of the normal distribution is completed by taking the logarithm of the data and applying it. The estimate of the mean μ after logarithmic transformation is

[0051]

[0052] Logarithmic variance σ 2 The estimate is

[0053]

[0054] where x i is the sample data point, n is the total number of sample data points;

[0055] For the exponential distribution model and the gamma distribution model of continuous data, the maximum likelihood estimation method (MLE) is selected, which is to observe the average value of the reciprocal of the data interval

[0056]

[0057] in is the sample mean;

[0058] The parameters α and β that make the given sample data appear with the highest probability are used for parameter estimation. The log-likelihood function of the gamma distribution is:

[0059]

[0060] For the Markov model, the EM algorithm or Bayesian estimation is used, where the EM algorithm iteratively estimates the model parameters by alternately executing the expectation step (E step) and the maximization step (M step). The E step calculates the conditional probability distribution of the hidden state based on the current parameter estimate, that is, the probability of each hidden state appearing given the observed data and the current parameter estimate. The M step uses the probability distribution of the hidden state calculated in the E step to update the parameter estimate to maximize the likelihood function. The E step and the M step are repeated until the parameter estimate converges or a certain termination condition is met, and finally the parameter value that maximizes the likelihood function is obtained;

[0061] Bayesian estimation estimates parameters by combining prior knowledge and observed data, providing the posterior distribution of the parameters, which is

[0062] P(θ|X)=(L(θ|X)·P(θ)) / P(X) (17)

[0063] Where P(θ|X) is the posterior distribution of parameter θ, P(θ) is the prior distribution, L(θ|X) is the likelihood function, and P(X) is the marginal probability of the observed data;

[0064] Step 1.4: Apply the selected parameter estimation method: Evaluate the model's validity and goodness of fit based on the following characteristics:

[0065] Through the evaluation results under different prediction models, the most suitable model is obtained for fitting;

[0066] By Akaike Information Criterion (AIC)

[0067]

[0068] where m is the number of model parameters, is the maximum likelihood value of the model;

[0069] By Bayesian Information Criterion (BIC)

[0070]

[0071] Where n is the sample size, m is the number of model parameters, is the maximum likelihood value of the model;

[0072] Step 1.5: Use the estimated model parameters to further determine the probability of the selected probability model based on its characteristics; for discrete data sets, use formulas (1), (2), and (3) to calculate the probability of a specific result; for continuous data sets, use formulas (4), (5), (6), and (7) to calculate the probability density of a data point around a certain value.

[0073] Furthermore, in step 1.1, the preprocessing includes processing missing values, outliers and noise data.

[0074] Furthermore, the step 2 comprises:

[0075] Step 2.1: For each data point in the data set, calculate its self-information based on its probability of occurrence; according to Shannon's definition, the self-information of a specific event can be calculated by the prior probability of the event, and its mathematical expression is

[0076]

[0077] Where (Pa i ) is event a i The prior probability of occurrence, I(a i ) represents event a i The amount of information contained in the occurrence;

[0078] Step 2.2: By calculating the expected value of the self-information of all possible events in the data set, the information entropy of the entire data set is obtained, and its mathematical expression is:

[0079] H(X)=E(I(a i )) (twenty one)

[0080] Where H(X) is the information entropy of the data set, and E represents the overall expectation function;

[0081] In addition, for data sets, the usual form is

[0082]

[0083] Among them, the data set X i There are r rows of data, each row of data has d attributes, and the i-th row of data is recorded as i=(x i1 , x i2 , ..., x id ), the data in the jth column is recorded as j=(x 1j , x 2j , ..., x rj ), for a single attribute a j The value of is v, then the information entropy calculation formula of this form of data set is

[0084]

[0085] Where P(a ji ) is the probability of the jth attribute in the data set taking the i-th value, and its formula is

[0086]

[0087] Furthermore, the step 3 comprises:

[0088] The information entropy obtained by calculation is used to sort the information entropy of each data set using the merge sort algorithm. According to the sorting results, each data set and their corresponding information entropy are filled into the database table in turn, the information entropy data is extracted from the database, and then the following statistical indicators are calculated:

[0089] The mean calculation formula is:

[0090]

[0091] The formula for calculating standard deviation is:

[0092]

[0093] Where H(X i ) represents the information entropy of the i-th sub-dataset, and n is the total number of sub-datasets;

[0094] The threshold is determined by using the mean double the standard deviation, which is calculated as:

[0095] Threshold=Mean+z×Standad Deviatiom (28)

[0097] Among them, z is an adjustable coefficient;

[0098] Alternatively, the percentile method is used to determine the threshold based on the quantile of the data, and a specific percentile is selected so that the sub-dataset above the threshold represents the most valuable part of the entire data set;

[0099] Use the determined threshold to query and mark all high-value sub-datasets in the database.

[0100] The second aspect of the present invention provides a data value assessment system based on information entropy measurement, comprising:

[0101] A data acquisition unit is used to acquire the data set to be evaluated, establish a probability model for each data set, select a parameter estimation method according to the determined model complexity and data volume, and use the estimated model parameters to estimate the probability of occurrence of these data points;

[0102] A data calculation unit, used for calculating the information entropy of each data set using an information quantity function;

[0103] The data evaluation unit is used to evaluate the average amount of information or uncertainty of the entire data set through the calculated information entropy to obtain the value of each data set.

[0104] A third aspect of the present invention provides an electronic device, comprising a processor and a memory that is communicatively connected to the processor and is used to store executable instructions of the processor, wherein the processor is used to execute the above-mentioned data value assessment method based on information entropy measurement.

[0105] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned data value assessment method based on information entropy measurement.

[0106] Compared with the prior art, the data value assessment method and system based on information entropy measurement described in the present invention have the following advantages:

[0107] The method of the present invention does not directly depend on the specific content of the data, but on the overall distribution and probability characteristics of the data, and can thus effectively solve the problem of high data complexity while identifying and filtering out redundant information. BRIEF DESCRIPTION OF THE DRAWINGS

[0108] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0109] Figure 1 This is a flow chart of a data value assessment method based on information entropy measurement of the present invention. DETAILED DESCRIPTION

[0110] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0111] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and the like are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", and the like may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0112] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood by specific circumstances.

[0113] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0114] Embodiment 1:

[0115] In the process of data value assessment, the information entropy principle method measures the value of a data set by quantifying the average amount of information in the entire data set. The information entropy of the entire data set reflects its overall uniqueness and the amount of information added, and is closely related to the probability distribution of each event in the data set. As a measurement based on probability distribution, information entropy naturally adapts to various types and sources of data sets. This method does not directly rely on the specific content of the data, but on the overall distribution and probabilistic characteristics of the data, so it can effectively solve the problem of high data complexity while identifying and filtering out redundant information.

[0116] Information entropy provides a well-defined evaluation criterion that does not rely on any prior knowledge or evaluation criteria in a specific field. This makes information entropy a universal evaluation tool that is applicable to different fields and various data types. Although the information entropy principle method itself may need to be fine-tuned according to the specific circumstances of different fields, its basic principles and calculation methods provide a common foundation for data value assessment, effectively solving the problem of lack of universal evaluation criteria and overcoming the limitations of existing evaluation methods.

[0117] The information entropy principle can dynamically adapt to changes in data value. It is based on the probability distribution of events in the entire data set, and these probabilities can change with time, usage environment, and purpose. By periodically recalculating the information entropy of the entire data set, the evaluation method can reflect real-time changes in data value and provide a dynamic data value assessment. This effectively solves the problem that data value changes based on its dynamics and timeliness. To achieve the above purpose, the present invention proposes a data value assessment method based on information entropy measurement, which is used for the evaluation of data value based on the information entropy principle. The overall process is as follows: Figure 1 As shown,

[0118] A data value assessment method based on the information entropy principle includes the following steps:

[0119] Step 1: Establish a probability model for each data set, select a parameter estimation method based on the determined model complexity and data size, and use the estimated model parameters to estimate the probability of occurrence of these data points;

[0120] Step 2: For each data set, use the information function to calculate the information entropy of the data set;

[0121] Step 3: By calculating the information entropy, evaluate the average amount of information or uncertainty of the entire data set to obtain the value of each data set.

[0122] Furthermore, for step 1,

[0123] 1. First, preprocess the collected data sets and clean them, including processing missing values, outliers and noise data to ensure data quality. Then identify the type of each data set to determine whether it is a continuous data set or a discrete data set. The key to this step is to identify the analysis model that is suitable for the data. Next, we check the distribution characteristics of the data, such as central tendency (mean, median), degree of dispersion (variance, standard deviation), skewness (symmetry of data distribution) and kurtosis (the degree of sharpness of data distribution), etc., to help determine the type of distribution that the data may follow.

[0124] 2. After preprocessing the data, the probability model is selected according to the type and distribution characteristics of the data and the following criteria:

[0125] For data sets, discrete data sets and continuous data sets are two types used to describe different data set attributes. Discrete data sets refer to data sets in which the data can only take specific, finite, or countably infinite values. Continuous data sets refer to data sets in which the data can take any value within a continuous range.

[0126] 2.1 Discrete Datasets

[0127] 2.1.1 Binomial Distribution Model

[0128] For discrete data sets that satisfy the conditions that there are only two possible results for each observation and the probabilities of these two results are fixed, with a fixed total number of times and known single result probabilities, the binomial distribution model can be used for analysis and calculation. The formula is

[0129]

[0130] Where P(X=k) represents the probability of data k appearing in n sets of data. p is the probability of data k, and (1-p) is the probability of data not being k.

[0131] 2.1.2 Poisson distribution model

[0132] For discrete data sets, if the data generation times are independent and have a fixed average generation rate within a specific time interval or spatial range, the Poisson distribution model can be used for analysis. The formula is:

[0133]

[0134] Where P(X=k) represents the probability that data is generated exactly k times in a given time interval, λ p It is the average occurrence rate of the data, indicating the average frequency of an event occurring per unit time or unit space.

[0135] 2.1.3 Markov Model

[0136] For discrete data sets, when predicting the probability of each data point in a series of data, and the probability of the next state of these data points depends only on their current state and is independent of the previous state (showing memorylessness or Markov properties), the Markov model can be used. t is the state at time t, the state space S = {s1,s2,s3,…,s M}M is the total number of states, and its state transition probability is

[0137] P ij =P(X t+1 =s j |X t =s i )(1≤i,j≤M) (3)

[0138] 2.2 Continuous Datasets

[0139] 2.1.1 Normal distribution (Gaussian distribution) model

[0140] For continuous data sets that satisfy the symmetrical distribution characteristics of the data around the central value, the normal distribution (Gaussian distribution) model can be used for analysis. Its formula is

[0141]

[0142] Where μ is the mean and σ is the standard deviation.

[0143] 2.1.2 Lognormal Distribution Model

[0144] For continuous data sets, when all data points are positive and the distribution is clearly right-skewed (i.e., has a long tail), the lognormal distribution model can be used. This model is applicable when the natural logarithm of a random variable follows a normal distribution. Its formula is

[0145]

[0146] Where μ is the mean and σ is the standard deviation.

[0147] 2.1.3 Exponential Distribution Model

[0148] For continuous data sets, when the data points occur randomly and independently in continuous time or space, and the probability of a data point occurring in a certain period of time in the future is independent of the time it has waited or occurred so far (showing no memory), the exponential distribution model can be used for analysis. Its formula is

[0149]

[0150] where λ e It is the average occurrence rate of the data, indicating the frequency of time between events, that is, the average number of times an event occurs per unit time.

[0151] 2.1.4 Gamma Distribution Model

[0152] For continuous data sets, the gamma distribution model can be used to analyze and describe the total waiting time or cumulative amount of multiple independent and similar events that occur within a fixed time period or spatial interval. This model is suitable for cumulative event occurrences, especially when the focus is on the sum of the number of event occurrences or the sum of event waiting times. Its formula is

[0153]

[0154] where α is the shape parameter of the distribution, β is the scale parameter that affects the width or spread of the distribution, and Γ(α) is the gamma function. For integer α,

[0155] T(α)=(α-1)! (8)

[0156] 3. Select a parameter estimation method based on the determined model complexity and data size.

[0157] 3.1 Parameter Estimation for Discrete Data Sets

[0158] For binomial and Poisson distribution models, select Maximum Likelihood Estimation (MLE). MLE estimates model parameters by maximizing the likelihood function of the observed data.

[0159]

[0160] Where x is the number of times the data is observed and n is the total number of data.

[0161]

[0162] where x i The number of times data is generated within each unit, that is, the observed data points, and n is the total number of data.

[0163] 3.2 Parameter Estimation for Continuous Data Sets

[0164] For the normal distribution model of continuous data sets, the moment estimation method is used to estimate the distribution parameters based on the sample moments, which contains two parameters, among which the estimate of the mean μ is

[0165]

[0166] Variance σ 2 The estimate is

[0167]

[0168] where x i is the sample data point, and n is the total number of sample data points.

[0169] For the parameter estimation of the lognormal distribution model of a continuous data set, the parameter estimation method of the normal distribution is applied after taking the logarithm of the data. The estimate of the mean μ after logarithmic transformation is

[0170]

[0171] Logarithmic variance σ 2 The estimate is

[0172]

[0173] where x i is the sample data point, and n is the total number of sample data points.

[0174] For the exponential distribution model of a continuous data set, choose the maximum likelihood estimation (MLE) method to observe the mean of the reciprocal of the data interval

[0175]

[0176] in is the sample mean;

[0177] For the gamma distribution model of a continuous data set, the maximum likelihood estimation method (MLE) is used to find the parameters α and β that maximize the probability of occurrence of a given sample data. For a given data sample, the log-likelihood function of the gamma distribution is

[0178]

[0179] The partial derivatives of l(α, β) with respect to α and β are taken to obtain a set of equations, which are solved using numerical optimization methods. Starting from the initial estimates, the estimates of α and β are iteratively optimized until they converge to the maximum likelihood estimate.

[0180] For Markov models and more complex models with latent variables, use the EM algorithm or Bayesian estimation.

[0181] The EM algorithm iteratively estimates model parameters by alternating between the expectation step (E step) and the maximization step (M step). It is used to find parameter estimates that maximize the likelihood function in the presence of hidden variables. First, select the initial estimate of the parameter.

[0182] Step E (Expectation): Based on the current parameter estimates, calculate the conditional probability distribution of the hidden state, that is, the probability of each hidden state occurring given the observed data and the current parameter estimates.

[0183] M-step (Maximization): Update the parameter estimates to maximize the likelihood function using the probability distribution of the hidden state calculated in step E. This usually involves finding the maximum value of the likelihood function with respect to the parameters.

[0184] Repeat steps E and M until the parameter estimation converges or a certain termination condition is met, and finally obtain the parameter value that maximizes the likelihood function.

[0185] Bayesian estimation estimates parameters by combining prior knowledge and observed data. First, for the state transition probability P of the Markov model ij Define a prior distribution, and use the likelihood function L(θ|X) (Formula (1)(2)(4)) to express the probability of observing data X given the parameter θ. Then apply Bayes' theorem to combine the prior distribution and the likelihood function to provide the posterior distribution of the parameter. Calculate the posterior distribution

[0186] P(θ|X)=(L(θ|X)·P(θ)) / P(X) (17)

[0187] Where P(θ|X) is the posterior distribution of the parameter θ, P(θ) is the prior distribution, L(θ|X) is the likelihood function, and P(X) is the marginal probability of the observed data, which serves as a normalizing constant.

[0188] 4. Then apply the selected parameter estimation method to evaluate the model's effectiveness and goodness of fit.

[0189] Through the evaluation results under different prediction models, the most suitable model is obtained for fitting.

[0190] By Akaike Information Criterion (AIC)

[0191]

[0192] where m is the number of model parameters, is the maximum likelihood value of the model.

[0193] By Bayesian Information Criterion (BIC)

[0194]

[0195] Where n is the sample size, m is the number of model parameters, is the maximum likelihood value of the model.

[0196] Akaike information criterion (AIC) and Bayesian information criterion (BIC) values ​​were calculated to compare the goodness of fit and complexity of different models and select the most appropriate model.

[0197] 5. Finally, the estimated model parameters are used to determine the probability mass function (PMF) or probability density function (PDF) of the selected probability model to calculate the probability of each data point.

[0198] For discrete data sets, PMF (as shown in formulas (1), (2), and (3)) is used to calculate the probability of a specific outcome.

[0199] For continuous data sets, PDF (as shown in formulas (4), (5), (6), (7)) is used to calculate the probability density of a data point around a certain value.

[0200] Specifically, in step 2, first, for each data point in the data set, its self-information is calculated according to its probability of occurrence. According to Shannon's definition, the self-information of a specific event can be calculated by the prior probability of the event, and its mathematical expression is:

[0201]

[0202] Where P(a i ) is event a i The prior probability of occurrence, I(a i ) represents event a i The amount of information contained in the occurrence is called self-information.

[0203] Then, by calculating the expected value of the self-information of all possible events in the data set, the information entropy of the entire data set is obtained, and its mathematical expression is:

[0204] H(X)=E(I(a i )) (twenty one)

[0205] Where H(X) is the information entropy of the data set and E represents the overall expectation function.

[0206] In addition, for data sets, the usual form is

[0207]

[0208] Among them, the data set X i There are r rows of data, each row of data has d attributes, and the i-th row of data is recorded as i=(x i1 ,x i2 ,...,x id ), the data in the jth column is recorded as j=(x 1j ,x 2j ,...,x rj ), for a single attribute a j The value of is v, then the information entropy calculation formula of this form of data set is

[0209]

[0210] Where p(a ji ) is the probability of the jth attribute in the data set taking the i-th value, and its formula is

[0211]

[0212] Specifically, in step 3, the information entropy calculated not only reflects the average uncertainty of the entire data set, but also provides a method to measure the intrinsic value of the data set. First, the information entropy of each data set is sorted using an efficient merge sort algorithm. Then, according to the sorting results, each data set and their corresponding information entropy are filled into the database table in turn. Then, the information entropy data is extracted from the database, and then the following statistical indicators are calculated:

[0213] Mean: It is the average value of the information entropy of the data set, and the calculation formula is

[0214]

[0215] Standard Deviation: It is a statistical indicator to measure the degree of dispersion of information entropy distribution. The calculation formula is:

[0216]

[0217] Where H(X i ) represents the information entropy of the ith sub-dataset, and n is the total number of sub-datasets.

[0218] The dynamic threshold is selected and determined based on the calculated statistical indicators, data characteristics and specific application requirements.

[0219] The threshold is determined by using the mean double the standard deviation, which is calculated as:

[0220] Threshold=Mean+z×Stamdard Deviation (28)

[0221] Here, z is an adjustable coefficient, usually 1, 1.5, or 2, depending on the distribution of the data and the rarity of the data points to be identified.

[0222] Or use the percentile method (Percentile-based Thresholding) to determine the threshold based on the quantile of the data. Select a specific percentile so that the sub-dataset above the threshold represents the most valuable part of the entire data set.

[0223] Finally, all high-value sub-datasets are queried and marked in the database using the determined threshold. The list of sub-datasets marked as high-value can be used for further analysis, decision support, or as a basis for feature selection. The level of information entropy can be used as an indicator to evaluate the value of a dataset. We can obtain a measure that represents the average uncertainty of the entire dataset, and then evaluate the value of the dataset.

[0224] Embodiment 2:

[0225] A data value assessment system based on information entropy measurement, including

[0226] A data acquisition unit is used to acquire the data set to be evaluated, establish a probability model for each data set, select a parameter estimation method according to the determined model complexity and data volume, and use the estimated model parameters to estimate the probability of occurrence of these data points;

[0227] A data calculation unit, used for calculating the information entropy of each data set using an information quantity function;

[0228] The data evaluation unit is used to evaluate the average amount of information or uncertainty of the entire data set through the calculated information entropy to obtain the value of each data set.

[0229] Embodiment three:

[0230] An electronic device includes a processor and a memory that is communicatively connected to the processor and is used to store executable instructions of the processor, wherein the processor is used to execute the above-mentioned data value assessment method based on information entropy measurement.

[0231] Embodiment 4:

[0232] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned data value assessment method based on information entropy measurement.

[0233] All matters not mentioned above are prior art.

[0234] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data value assessment method based on information entropy measurement, characterized by: The steps include: Step 1: Obtain the data set to be evaluated, establish a probability model for each data set, select a parameter estimation method based on the determined model complexity and data size, and use the estimated model parameters to estimate the probability of occurrence of these data points; Step 2: For each data set, use the information function to calculate the information entropy of the data set; Step 3: By calculating the information entropy, evaluate the average amount of information or uncertainty of the entire data set to obtain the value of each data set.

2. According to claim 1, a data value assessment method based on information entropy measurement is characterized by: The step 1 specifically includes: Step 1.1: Preprocess each acquired data set to ensure data quality, and then identify the type of each data set to determine whether the data set is a continuous data set or a discrete data set; Step 1.2: Analyze the data type and distribution characteristics of each preprocessed data set, and select a probability model based on the following characteristics: The processing methods for discrete data sets include but are not limited to: The binomial distribution model is used when there are only two possible outcomes for each observation, where the probability of each outcome is fixed, with the characteristics of a fixed total number of times and a known probability of a single outcome. The probability calculation formula is: P(X=k) represents the probability of data k appearing in n sets of data, and p is the single observation probability of the event; The Poisson distribution model is used for conditions where the number of events occurring within a specific time interval or spatial range is independent and has a fixed average generation rate. The probability calculation formula is where λ p It is the average occurrence rate of the data, indicating the average frequency of events occurring per unit time or unit space; Markov model is used to predict the probability of each data point in a series of data, and the probability of the next state of these data points depends only on their current state, showing no memory or Markov property, X t is the state at time t, the state space S = {s1, s2, s3, ..., s M }, M is the total number of states, and its state transition probability is P ij =P(X t+1 =s j |X t =s i )(1≤i,j≤M) (3); The processing methods for continuous data sets include but are not limited to: The normal distribution (Gaussian distribution) model is used for the characteristics of data symmetrically distributed around the central value, and its probability function is Where μ is the mean and σ is the standard deviation; The lognormal distribution model is used when all data points are positive and the distribution is obviously right-skewed. When the natural logarithm of the random variable follows a normal distribution, its probability function is Where μ is the mean and σ is the standard deviation; The exponential distribution model is used for events that occur randomly and independently in continuous time or space, and the probability of an event occurring in a certain period of time in the future is independent of the time it has waited or occurred so far. Its probability function is where λ e It is the average occurrence rate of the data, indicating the frequency of time between events, that is, the average number of times an event occurs per unit time; The gamma distribution model is suitable for analyzing and describing the total waiting time or cumulative amount of multiple independent and similar events occurring in a fixed time period or spatial interval. Its probability function is Γ(α)=(α-1)! (8)(8) Where α is the shape parameter of the distribution, μ is the scale parameter that affects the width or spread of the distribution, and Γ(α) is the gamma function; Step 1.3: Select the parameter estimation method based on the following characteristics: For the binomial distribution model and Poisson distribution model of discrete data, the maximum likelihood estimation (MLE) method is used for parameter estimation. Where x is the number of times the observed data occurs, and n is the total number of data; Parameter Estimation of Poisson Distribution Model where x i It indicates the number of times data is generated in each unit, and n is the total number of data; For the normal distribution model of continuous data, the moment estimation method is applied to estimate the distribution parameters based on the sample moments, where the mean μ is estimated as Variance σ 2 The estimate is where x i is the sample data point, n is the total number of sample data points; For the lognormal distribution model of continuous data, the parameter estimation method of the normal distribution is completed by taking the logarithm of the data and applying it. The estimate of the mean μ after logarithmic transformation is Logarithmic variance σ 2 The estimate is where x i is the sample data point, n is the total number of sample data points; For the exponential distribution model and the gamma distribution model of continuous data, the maximum likelihood estimation method (MLE) is selected, which is to observe the average value of the reciprocal of the data interval in is the sample mean; The parameters α and μ that maximize the probability of occurrence of given sample data are used for parameter estimation. The log-likelihood function of the gamma distribution is For the Markov model, the EM algorithm or Bayesian estimation is used, where the EM algorithm iteratively estimates the model parameters by alternately executing the expectation step (E step) and the maximization step (M step). The E step calculates the conditional probability distribution of the hidden state based on the current parameter estimate, that is, the probability of each hidden state appearing given the observed data and the current parameter estimate. The M step uses the probability distribution of the hidden state calculated in the E step to update the parameter estimate to maximize the likelihood function. The E step and the M step are repeated until the parameter estimate converges or a certain termination condition is met, and finally the parameter value that maximizes the likelihood function is obtained; Bayesian estimation estimates parameters by combining prior knowledge and observed data, providing the posterior distribution of the parameters, which is P(θ|X)=(L(θ|X)·P(θ)) / P(X) (17) Where P(θ|X) is the posterior distribution of parameter θ, P(θ) is the prior distribution, L(θ|X) is the likelihood function, and P(X) is the marginal probability of the observed data; Step 1.4: Apply the selected parameter estimation method: Evaluate the model's validity and goodness of fit based on the following characteristics: Through the evaluation results under different prediction models, the most suitable model is obtained for fitting; By Akaike Information Criterion (AIC) where m is the number of model parameters, is the maximum likelihood value of the model; By Bayesian Information Criterion (BIC) Where n is the sample size, m is the number of model parameters, is the maximum likelihood value of the model; Step 1.5: Use the estimated model parameters to further determine the probability of the selected probability model based on its characteristics; for discrete data sets, use formulas (1), (2), and (3) to calculate the probability of a specific result; for continuous data sets, use formulas (4), (5), (6), and (7) to calculate the probability density of a data point around a certain value.

3. The data value assessment method based on information entropy measurement according to claim 1 is characterized in that: In step 1.1, the preprocessing includes processing missing values, outliers and noise data.

4. The data value assessment method based on information entropy measurement according to claim 1 is characterized in that: The step 2 comprises: Step 2.1: For each data point in the data set, calculate its self-information based on its probability of occurrence; according to Shannon's definition, the self-information of a specific event can be calculated by the prior probability of the event, and its mathematical expression is Where P(a i ) is event a i The prior probability of occurrence, I(a i ) represents event a i The amount of information contained in the occurrence; Step 2.2: By calculating the expected value of the self-information of all possible events in the data set, the information entropy of the entire data set is obtained, and its mathematical expression is: H(X)=E(I(a i )) (21) Where H(X) is the information entropy of the data set, and E represents the overall expectation function; In addition, for data sets, the usual form is Among them, the data set X i There are r rows of data, each row of data has d attributes, and the i-th row of data is recorded as i=(x i1 , x i2 , ..., x id ), the data in the jth column is recorded as j=(x 1j , x 2j , ..., x rj ), for a single attribute a j The value of is v, then the information entropy calculation formula of this form of data set is Where P(a ji ) is the probability of the jth attribute in the data set taking the i-th value, and its formula is 5. The data value assessment method based on information entropy measurement according to claim 1 is characterized in that: The step 3 comprises: The information entropy obtained by calculation is used to sort the information entropy of each data set using the merge sort algorithm. According to the sorting results, each data set and their corresponding information entropy are filled into the database table in turn, the information entropy data is extracted from the database, and then the following statistical indicators are calculated: The mean calculation formula is: The formula for calculating standard deviation is: Where H(X i ) represents the information entropy of the i-th sub-dataset, and n is the total number of sub-datasets; The threshold is determined by using the mean double the standard deviation, which is calculated as: Threshold=Mean+z×Standard Deviation (28) Among them, z is an adjustable coefficient; Alternatively, the percentile method is used to determine the threshold based on the quantile of the data, and a specific percentile is selected so that the sub-dataset above the threshold represents the most valuable part of the entire data set; Use the determined threshold to query and mark all high-value sub-datasets in the database.

6. A data value assessment system based on information entropy measurement, characterized in that: include A data acquisition unit is used to acquire the data set to be evaluated, establish a probability model for each data set, select a parameter estimation method according to the determined model complexity and data volume, and use the estimated model parameters to estimate the probability of occurrence of these data points; A data calculation unit, used for calculating the information entropy of each data set using an information quantity function; The data evaluation unit is used to evaluate the average amount of information or uncertainty of the entire data set through the calculated information entropy to obtain the value of each data set.

7. An electronic device, comprising a processor and a memory connected to the processor for storing instructions executable by the processor, characterized in that: The processor is used to execute the data value assessment method based on information entropy measurement as described in any one of claims 1-5 above.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the data value assessment method based on information entropy measurement described in any one of claims 1 to 5 is implemented.

Citation Information

Patent Citations

  • Extreme hydrological event risk analysis method and system based on maximum entropy

    CN110598181A

  • Active learning method for optical network transmission quality regression estimation

    CN113627621A

  • Parameter probability uncertainty modeling method under data deletion

    CN114254516A

  • Information processing device, information processing method, and program

    US20130262013A1

Cited By

  • System for evaluating information amount of data set based on information entropy and weighting algorithm

    CN119719083A

  • An evaluation system for the information content of data sets based on information entropy and weighted algorithm

    CN119719083B