An evaluation system for the information content of data sets based on information entropy and weighted algorithm

Abstract

The present invention discloses a system for evaluating the information content of a dataset based on information entropy and a weighted algorithm. The system comprises a dataset selection unit, a dataset annotation unit, a dataset information processing unit, and an information entropy calculation model. By discretizing continuous variables in a dataset and combining information entropy with a weighted algorithm, the system enables the information entropy calculation model to evaluate not only discretized variables but also continuous variables. Furthermore, the system can evaluate not only single-field datasets but also multi-field datasets, broadening its applicability.

Description

Technical Field

[0001] The present invention relates to an evaluation system based on information entropy and a weighted algorithm in a process of quantifying the amount of information in a data set, and belongs to the field of data information technology. Background Art

[0002] Evaluating the information content of a dataset is a key task in data science and machine learning. It not only helps us quantify and understand the information content and uncertainty in a dataset, but also directly affects the performance and generalization ability of the model, helping to optimize data preprocessing and feature selection processes, thereby improving the accuracy and efficiency of the model.

[0003] Information entropy is a fundamental concept in information theory and an important metric for measuring the information content of a dataset. It assesses the amount of information contained in a dataset by calculating its uncertainty or degree of disorder. Specifically, a higher entropy indicates more disordered data and a greater amount of information; conversely, a lower entropy indicates more ordered data and a lower amount of information.

[0004] To quantify the amount of information in data, researchers have proposed a series of techniques and methods. However, information entropy can only be applied to a single field in a dataset and is mainly applicable to discretized variables, so it has certain limitations. Summary of the Invention

[0005] The purpose of the present invention is to provide an evaluation method based on information entropy and weighted algorithm to quantify the information content of a data set, expand the scope of application of information entropy, and be closer to actual needs, thereby providing a more intuitive and demand-oriented evaluation of the information content of a data set.

[0006] In order to achieve the above-mentioned object, the technical solution of the present invention is to disclose a system for evaluating the information content of a data set based on information entropy and a weighted algorithm, which is characterized by comprising:

[0007] Dataset selection unit: used to select the data set or related fields from the platform database that need to calculate the amount of information to construct the data set;

[0008] Dataset annotation unit: used to annotate the fields of the dataset in the dataset selection unit with field type, field attention level, and field uniform expectation;

[0009] Dataset information processing unit: used to perform data preprocessing and data discretization on the information data in the dataset selection unit, including:

[0010] Data preprocessing, which is used to clean the data set, extract valid values, and normalize the data;

[0011] Data discretization is used to discretize the continuous fields marked in the dataset annotation unit into discrete fields with equal width;

[0012] Information entropy calculation model: used to calculate the value Score of the information content of a dataset to determine the amount of information contained in the dataset. It further includes a frequency calculation unit, a field information entropy calculation unit, a weight distribution calculation unit, and a weighted algorithm calculation unit, where:

[0013] The frequency calculation unit counts the total number N of all possible events for each field in the statistical data set, as well as the number m of each event i , for each event x i , calculate the probability of its occurrence p(x i );

[0014] The field information entropy calculation unit calculates the probability p(x i ) is substituted into the Shannon information entropy calculation formula to obtain the information entropy value, and then normalized to unify the information entropy calculated for each field; according to the field uniformity expectation of the data set annotation unit, if the expected field data distribution is uniform, the normalized information entropy value is directly used; if the expected field data distribution is uneven, the complementary number of the normalized information entropy value is used as the information value score of the field;

[0015] The weight distribution calculation unit first calculates the weight of the information value of each field, and then adjusts the weight according to the field attention level of the dataset annotation unit to obtain a new weight value;

[0016] The weighted algorithm calculation unit performs a weighted summation on the information value and weight value of all fields to obtain the information value Score of the data set.

[0017] Preferably, the data set selection unit selects a data set to be calculated, or selects a field in a data set to be calculated, and saves it in the form of a database table.

[0018] Preferably, in the data set annotation unit, the field type annotation is divided into: serial number type, continuous type and discrete type, the field attention level annotation is divided into: low, medium and high, and the field uniform expectation annotation is divided into: uniform and uneven.

[0019] Preferably, in the data set information processing unit, the data preprocessing includes the following steps:

[0020] Clean invalid characters from text fields. After the initial cleaning, if only invalid characters remain, convert them to null. Standardize uppercase and lowercase English characters. Standardize half-width and full-width symbols.

[0021] Unify the data format for numeric fields;

[0022] Normalize and map dictionary data based on national standards, industry standards, or specific standards for dictionary fields;

[0023] Unify the format of date types;

[0024] Remove duplicate redundant data.

[0025] Preferably, in the data set information processing unit, the data discretization includes the following steps:

[0026] Filter out continuous fields based on the dataset annotation unit;

[0027] Standardize the continuous field data and convert the data into a distribution with a mean of 0 and a standard deviation of 1. The formula is as follows:

[0028]

[0029] Where y is the new value after the original value is standardized, x is the original value, μ is the mean of the column, and σ is the standard deviation of the column;

[0030] Determine the number of intervals n to be divided and perform equal-width discretization.

[0031] Preferably, the frequency calculation unit counts the total number N of all possible events in a certain field in the statistical data set, and the number m of each event i , calculate the probability of each event, for each event x i , calculate the probability of its occurrence

[0032] Preferably, the field information entropy calculation unit calculates each event x according to the frequency calculation unit. i The probability of occurrence p(x i ), and then calculate the information entropy value H(X) of the field according to the Shannon information entropy formula. The formula is as follows:

[0033]

[0034] Where n is the number of event categories.

[0035] Preferably, in the field information entropy calculation unit:

[0036] Information entropy normalization is used to unify the information entropy values ​​calculated for each field in the data set. The value range of information entropy is [0, log2n], where n is the number of categories of the indicator. The formula is as follows:

[0037]

[0038] Among them, H(J) is the information entropy value, and the normalized information entropy value E(J) ranges from [0,1];

[0039] According to the uniform expected value a of the field of the dataset annotation unit, if a=0, the user expects the field value to be unevenly distributed, then the field information value D(J)=1-E(J); if a=1, the user expects the field value to be evenly distributed, then the field information value D(J)=E(J).

[0040] Preferably, the weight distribution calculation unit calculates the size D of each information entropy value D(J) after the data set is calculated. i , calculate its weight w i , the formula is as follows:

[0041]

[0042] Then, the weight is adjusted according to the degree of attention of the field in the dataset annotation unit. The weight adjustment formula is as follows:

[0043]

[0044] Where b is the attention level value of the field.

[0045] Preferably, the weighted algorithm calculation unit is based on the information value D of each field in the data set. i size, and weight W i Size, calculate the information value Score of the data set, Get the information value of the data set whose calculation result range is between [0,1].

[0046] Compared with the existing technical solutions, the present invention has the following beneficial effects:

[0047] 1) Discretize the continuous variables in the data set and combine the information entropy with the weighting algorithm. This allows the information entropy calculation model to evaluate not only discrete variables but also continuous variables. At the same time, it can evaluate not only single-field data sets but also multi-field data sets, making its scope of application more extensive.

[0048] 2) Adding field attention level annotations can help us assign weights based on actual user needs;

[0049] 3) We designed a field uniform expectation annotation, which can adjust the information entropy value based on the user's understanding of the uniform distribution of the dataset, thereby making the application of information entropy closer to actual needs and reflecting the uncertainty or chaos of the dataset, helping us provide a richer perspective for data analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 The data processing flow of the system disclosed in the present invention is illustrated. DETAILED DESCRIPTION

[0051] Below in conjunction with specific embodiment, further set forth the present invention.Should be understood that these embodiments are only used to illustrate the present invention and are not used in limiting the scope of the present invention.In addition, should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall equally within the scope limited by the appended claims of the application.

[0052] In this embodiment of the present invention, a medical dataset is used as an example. The data of the medical institution involving patient consultation information is used as the original data source. The system for evaluating the information content of the dataset based on information entropy and weighted algorithm provided by the present invention is implemented. Specifically, the system includes the following parts:

[0053] Dataset selection unit: This unit is used to select fields from the platform database that need to calculate information volume and construct a data table. During the sample collection process, the number of samples obtained was 16,071,817. The collected fields included patient identity information (name, ID number, ID type, gender, date of birth), medical information (medical card, medical card type, outpatient / inpatient sign, medical date, diagnosis ICD code, diagnosis ICD name), medical order / prescription information (prescription time, prescription code, prescription name, quantity of medicine dispensed), and cost information (outpatient / inpatient medical expenses, outpatient / inpatient drug expenses), totaling 17 variables.

[0054] Dataset annotation unit: used to annotate the fields of the data table in the dataset selection unit with field type, field attention level, and uniform expectation. Annotation examples are shown in Table 1:

[0055] The field type annotations are mainly divided into three categories: serial type 0, discrete type 1 and continuous type 2;

[0056] (a) Serial number type: Serial number type mainly refers to relatively independent fields such as the primary key, serial number, and record time of this data table. These fields do not participate in information volume statistics (the information volume of these fields is often 0, which does not affect the final result, but can reduce model calculation time);

[0057] (b) Discrete: Discrete variables primarily refer to variables that can only take on specific values, typically integers or enumerable values. Discrete variables are not continuous but distinct. These variables are typically used for counting or classification and can be directly used in model calculations.

[0058] (c) Continuous: A continuous variable is one that can take on any real value within a range. Its value is continuous and, in theory, can take on an infinite number of values. Continuous variables are used for measurement or metric purposes, such as length, weight, temperature, and money.

[0059] Field attention level annotation (not labeled for serial number type) is mainly divided into three categories: low 1, medium 2 and high 3; uniform expectation annotation (not labeled for serial number type) is divided into two categories: expected uneven 0 and expected uniform 1.

[0060] Table 1 Field annotation examples

[0061]

[0062] 3. Dataset Information Processing Unit: This unit performs data preprocessing and discretization on the data table constructed in the Dataset Selection Unit. Data preprocessing primarily involves cleaning, extracting valid values, and normalizing the data in the data table. Data discretization discretizes the continuous fields annotated in the Dataset Annotation Unit into discrete fields with equal widths.

[0063] (1) Data preprocessing

[0064] I. Clean text fields of invalid characters, such as special symbols, spaces, and invisible characters. After the initial cleaning, if only invalid characters remain, convert them to null. Standardize uppercase and lowercase English characters, and standardize half-width and full-width characters.

[0065] II. Unify the data format of numeric fields. For example, convert the number of people to BIGINT, the cost to DECIMAL(28,4), and convert null to 0.

[0066] III. Normalize and map dictionary data for dictionary fields based on national, industry, or specific standards. For example, for gender fields, convert and retain valid results based on the national standard for human gender code (GB / T 2261.1-2003).

[0067] IV. Standardize the date format to yyyy-MM-dd and the time format to yyyy-MM-dd hh:mm:ss.

[0068] V. Remove duplicate redundant data;

[0069] After data preprocessing, the total number of samples is 15,845,378.

[0070] (2) Data discretization

[0071] (a) First, filter out continuous fields, such as medical expenses and drug expenses, based on the field type label.

[0072] (b) Standardize each field value one by one and transform the data into a distribution with a mean of 0 and a standard deviation of 1. The formula is as follows:

[0073]

[0074] Where y is the new value after the original value is standardized, x is the original value, μ is the mean of the column, and σ is the standard deviation of the column;

[0075] (c) Then determine the number of intervals to be divided (n=10) and perform equal-width discretization, so as to retain the original distribution pattern of the data as much as possible during the discretization process.

[0076] 4. Information entropy calculation model: Construct an information entropy calculation model, including frequency calculation unit, field information entropy calculation unit, weight distribution calculation unit, and weighted algorithm calculation unit.

[0077] (1) Frequency calculation unit: First, the total number N of all possible events in a field in the statistical data set is counted. N is the total number of samples 15845378, and the number of each event m i , calculate the probability of each event, for each event x i , calculate the probability of its occurrence, that is, the frequency Taking the field certificate type as an example, its frequency calculation results are shown in Table 2:

[0078] Table 2 Frequency statistics of certificate types

[0079]

[0080] (2) Field information entropy calculation unit, first calculates the probability of occurrence of each event p(x i ), and then calculate the information entropy value of the field according to the Shannon information entropy formula, the formula is as follows:

[0081]

[0082] Where n is the number of event categories.

[0083] Next, the field information entropy value is normalized. It is necessary to unify the information entropy values ​​calculated for each field in the data set so that each value can be compared and analyzed on the same scale, thereby accurately reflecting the amount of information in each field. Because the value range of information entropy is [0, log2n], n is the number of categories in the field, so its formula is as follows:

[0084]

[0085] Among them, H(J) is the information entropy value, and the normalized information entropy value E(J) ranges from [0,1].

[0086] Finally, according to the uniform expected value a of the field of the labeled unit of the dataset, if a=0, the user expects the field value to be unevenly distributed, and the information value of the field D(J)=1-E(J); if a=1, the user expects the field value to be evenly distributed, and the information value of the field D(J)=E(J); the calculation example is shown in Table 3:

[0087] Table 3 Field information entropy calculation example

[0088]

[0089]

[0090] (3) The weight distribution calculation unit distributes the calculated information value D of each field by weight. First, a preliminary weight calculation is performed. The formula is as follows:

[0091]

[0092] Then, according to the degree of attention paid to the fields in the dataset annotation unit, the weight is adjusted to obtain a new weight value. The weight adjustment formula is as follows:

[0093]

[0094] Where b is the field attention level value of the dataset annotation unit. The calculation example is shown in Table 4:

[0095] Table 4 Weight distribution calculation example

[0096]

[0097] (4) The weighted algorithm calculation unit calculates the information value D of each field in the data set. i size, and weight W i Size, calculate the information value Score of the data set, the result range is [0,1], the calculation formula is:

Claims

1. A system for evaluating the information content of a data set based on information entropy and weighted algorithm, characterized in that: include: Dataset selection unit: used to select the data set or related fields from the platform database that need to calculate the amount of information to construct the data set; Dataset annotation unit: used to annotate the fields of the dataset in the dataset selection unit with field type, field attention level, and field uniform expectation; Dataset information processing unit: used to perform data preprocessing and data discretization on the information data in the dataset selection unit, including: Data preprocessing, which is used to clean the data set, extract valid values, and normalize the data; Data discretization is used to discretize the continuous fields marked in the dataset annotation unit into discrete fields with equal width; Information entropy calculation model: used to calculate the value Score of the information content of a dataset to determine the amount of information contained in the dataset. It further includes a frequency calculation unit, a field information entropy calculation unit, a weight distribution calculation unit, and a weighted algorithm calculation unit, where: The frequency calculation unit counts the total number N of all possible events for each field in the statistical data set, as well as the number m of each event i , for each event x i , calculate the probability of its occurrence p(x i ); The field information entropy calculation unit calculates the probability p(x i ) is substituted into the Shannon information entropy calculation formula to obtain the information entropy value, and then normalized to unify the information entropy calculated for each field; according to the field uniformity expectation of the data set annotation unit, if the expected field data distribution is uniform, the normalized information entropy value is directly used; if the expected field data distribution is uneven, the complementary number of the normalized information entropy value is used as the information value score of the field; The weight distribution calculation unit first calculates the weight of the information value of each field, and then adjusts the weight according to the field attention level of the dataset annotation unit to obtain a new weight value; The weighted algorithm calculation unit performs a weighted summation on the information value and weight value of all fields to obtain the information value Score of the data set.

2. The system for evaluating the information content of a data set based on information entropy and weighted algorithm according to claim 1, characterized in that: The data set selection unit selects the data set to be calculated, or selects the fields to be calculated in the data set, and saves them in the form of a database table.

3. The system for evaluating the information content of a data set based on information entropy and weighted algorithm according to claim 1, characterized in that: In the dataset annotation unit, the field type annotation is divided into: serial number type, continuous type and discrete type, the field attention level annotation is divided into: low, medium and high, and the field uniform expectation annotation is divided into: uniform and uneven.

4. The system for evaluating the information content of a data set based on information entropy and weighted algorithm according to claim 1, wherein: In the data set information processing unit, the data preprocessing includes the following steps: Clean invalid characters from text fields. After the initial cleaning, if only invalid characters remain, convert them to null. Standardize uppercase and lowercase English characters. Standardize half-width and full-width symbols. Unify the data format for numeric fields; Normalize and map dictionary data based on national standards, industry standards, or specific standards for dictionary fields; Unify the format of date types; De-duplicate redundant data.

5. The system for evaluating the information content of a data set based on information entropy and weighted algorithm according to claim 1, wherein: In the data set information processing unit, the data discretization includes the following steps: Filter out continuous fields based on the dataset annotation unit; Standardize the continuous field data and convert the data into a distribution with a mean of 0 and a standard deviation of 1. The formula is as follows: Where y is the new value after standardization of the original value, x is the original value, μ is the column mean, and σ is the column standard deviation; Determine the number of intervals to be divided and perform equal-width discretization.

6. The system for evaluating the information content of a data set based on information entropy and weighted algorithm according to claim 1, wherein: The frequency calculation unit counts the total number N of all possible events in a field in the statistical data set, as well as the number m of each event i , calculate the probability of each event, for each event x i , calculate the probability of its occurrence 7. The system for evaluating the information content of a data set based on information entropy and weighted algorithm according to claim 1, wherein: The field information entropy calculation unit calculates each event x according to the frequency calculation unit. i The probability of occurrence p(x i ), and then calculate the information entropy value H(X) of the field according to the Shannon information entropy formula. The formula is as follows: Where n is the number of event categories.

8. The system for evaluating the information content of a data set based on information entropy and weighted algorithm according to claim 7, characterized in that: In the field information entropy calculation unit: Information entropy normalization is used to unify the information entropy values ​​calculated for each field in the data set. The value range of information entropy is [0, log2n]. The formula is as follows: Among them, H(J) is the information entropy value, and the normalized information entropy value E(J) ranges from [0,1]; According to the uniform expected value a of the field of the dataset annotation unit, if a=0, the user expects the field value to be unevenly distributed, then the field information value D(J)=1-E(J); if a=1, the user expects the field value to be evenly distributed, then the field information value D(J)=E(J).

9. The system for evaluating the information content of a data set based on information entropy and weighted algorithm according to claim 8, characterized in that: The weight distribution calculation unit calculates the size D of each information entropy value D(J) after the data set is calculated. i , calculate its weight w i , the formula is as follows: Then, the weight is adjusted according to the degree of attention of the field in the dataset annotation unit. The weight adjustment formula is as follows: Where b is the attention level value of the field.

10. The system for evaluating the information content of a data set based on information entropy and weighted algorithm according to claim 9, characterized in that: The weighted algorithm calculation unit calculates the information value D of each field in the data set according to the value D of each field in the data set. i size, and weight W i Size, calculate the information value Score of the data set, Get the information value of the data set whose calculation result range is between [0,1].

Images