System for evaluating information amount of data set based on information entropy and weighting algorithm

Abstract

The invention discloses a system for evaluating the information amount of a data set based on information entropy and a weighting algorithm, and the system is characterized in that the system comprises a data set selection unit; a data set labeling unit; a data set information processing unit; and an information entropy calculation model. According to the method, the continuous variables in the data set are discretized, and the information entropy is combined with the weighting algorithm, so that the information entropy calculation model not only can evaluate the discretized variables, but also can evaluate the continuous variables, and meanwhile, not only can evaluate the single-field data set, but also can evaluate the multi-field data set; and the application range becomes wider.

Description

Technical Field

[0001] The invention relates to an evaluation system based on information entropy and a weighted algorithm in a process of quantifying the information amount of a data set, and belongs to the technical field of data information. Background Art

[0002] Evaluating the information content of a dataset is a key task in the field of data science and machine learning. It not only helps us quantify and understand the information content and uncertainty in a dataset, but also directly affects the performance and generalization ability of the model, and helps optimize the data preprocessing and feature selection process, thereby improving the accuracy and efficiency of the model.

[0003] At the same time, information entropy is a basic concept in information theory and an important indicator for measuring the amount of information in a data set. It evaluates the amount of information by calculating the uncertainty or degree of disorder of a data set. Specifically, the larger the information entropy, the more disordered the data is and the greater the amount of information; conversely, the smaller the order, the smaller the amount of information.

[0004] In order to quantify the amount of information in data, researchers have proposed a series of techniques and methods. However, information entropy can only act on a single field in a data set and is mainly applicable to discretized variables, so it has certain limitations. Summary of the invention

[0005] The purpose of the present invention is to provide an evaluation method based on information entropy and weighted algorithm to quantify the amount of information of a data set, expand the scope of application of information entropy, and be closer to actual needs, so as to have a more intuitive and demand-oriented evaluation of the amount of information of a data set.

[0006] In order to achieve the above object, the technical solution of the present invention is to disclose a system for evaluating the information volume of a data set based on information entropy and a weighted algorithm, which is characterized by comprising:

[0007] Dataset selection unit: used to select the data set or related fields from the platform database that need to calculate the amount of information to construct the data set;

[0008] Dataset annotation unit: used to annotate the fields of the dataset in the dataset selection unit with the field type, field attention level, and field uniform expectation;

[0009] Data set information processing unit: used to perform data preprocessing and data discretization on the information data in the data set selection unit, where:

[0010] Data preprocessing, which is used to clean the data set, extract valid values, and normalize the data;

[0011] Data discretization is used to discretize the continuous fields marked in the dataset annotation unit into discrete fields with equal width;

[0012] Information entropy calculation model: used to calculate the value Score of the amount of information in a data set to determine the amount of information contained in the data set. It further includes a frequency calculation unit, a field information entropy calculation unit, a weight distribution calculation unit, and a weighted algorithm calculation unit, where:

[0013] The frequency calculation unit counts the total number N of all possible events for each field in the statistical data set, as well as the number m of each event i , for each event x i , calculate the probability of its occurrence p(x i );

[0014] The field information entropy calculation unit calculates the probability p(x i ) is substituted into the Shannon information entropy calculation formula to obtain the information entropy value, and then normalized to unify the information entropy calculated for each field; according to the uniform expectation of the field of the data set annotation unit, if the expected field data is evenly distributed, the normalized information entropy value is directly used; if the expected field data is unevenly distributed, the complementary number of the normalized information entropy value is used as the information value score of the field;

[0015] The weight distribution calculation unit first calculates the weight of the information value of each field, and then adjusts the weight according to the field attention level of the data set annotation unit to obtain a new weight value;

[0016] The weighted algorithm calculation unit performs a weighted summation on the information value and weight value of all fields to obtain the information value Score of the data set.

[0017] Preferably, the data set selection unit selects the data set to be calculated, or selects the field to be calculated in the data set, and saves it in the form of a database table.

[0018] Preferably, in the data set annotation unit, the field type annotation is divided into: serial number type, continuous type and discrete type, the field attention level annotation is divided into: low, medium and high, and the field uniform expectation annotation is divided into: uniform and uneven.

[0019] Preferably, in the data set information processing unit, the data preprocessing comprises the following steps:

[0020] Clean invalid characters from text fields. If only invalid characters remain after the initial cleaning, convert them to null. Unify uppercase and lowercase English characters. Unify half-width and full-width symbols.

[0021] Unify the data format for numeric fields;

[0022] Normalize and map dictionary data for dictionary fields based on national standards, industry standards or specific standards;

[0023] Process the date types in a unified format;

[0024] Remove duplicate redundant data.

[0025] Preferably, in the data set information processing unit, the data discretization comprises the following steps:

[0026] Filter out continuous fields based on the dataset annotation units;

[0027] Standardize the continuous field data and transform the data into a distribution with a mean of 0 and a standard deviation of 1. The formula is as follows:

[0028]

[0029] In the formula, y is the new value after the original value is standardized, x is the original value, μ is the mean of the column, and σ is the standard deviation of the column;

[0030] Determine the number of divided intervals n and perform equal-width discretization.

[0031] Preferably, the frequency calculation unit counts the total number N of all possible events in a field in the statistical data set, and the number m of each event i , calculate the probability of each event, for each event x i , calculate the probability of its occurrence

[0032] Preferably, the field information entropy calculation unit calculates each event x according to the frequency calculation unit. i The probability of occurrence p(x i ), and then calculate the information entropy value H(X) of the field according to the Shannon information entropy formula. The formula is as follows:

[0033]

[0034] Where n is the number of event categories.

[0035] Preferably, in the field information entropy calculation unit:

[0036] The information entropy value is normalized to unify the information entropy values ​​calculated for each field of the data set. The value range of information entropy is [0, log2n], where n is the number of categories of the indicator. The formula is as follows:

[0037]

[0038] Among them, H(J) is the information entropy value, and the normalized information entropy value E(J) ranges from [0,1];

[0039] According to the uniform expected value a of the field of the annotation unit of the data set, if a=0, the user expects that the field value is unevenly distributed, then the field information value D(J)=1-E(J); if a=1, the user expects that the field value is evenly distributed, then the field information value D(J)=E(J).

[0040] Preferably, the weight distribution calculation unit calculates the size D of each information entropy value D(J) after the data set is calculated. i , calculate its weight w i , the formula is as follows:

[0041]

[0042] Then, according to the degree of attention paid to the fields in the dataset annotation unit, the weight adjustment is performed. The weight adjustment formula is as follows:

[0043]

[0044] Wherein, b is the attention level value of the field.

[0045] Preferably, the weighted algorithm calculation unit calculates the information value D of each field in the data set according to i size, and weight W i Size, calculate the information value Score of the data set, Get the information value of the data set whose calculation result range is between [0,1].

[0046] Compared with the existing technical solutions, the present invention has the following beneficial effects:

[0047] 1) Discretize the continuous variables in the data set and combine the information entropy with the weighted algorithm, so that the information entropy calculation model can evaluate not only the discrete variables but also the continuous variables. At the same time, it can evaluate not only the single-field data set but also the multi-field data set, and its application scope becomes wider;

[0048] 2) Adding field attention level annotation can help us assign weights based on actual user needs;

[0049] 3) A field uniform expectation annotation is designed, which can adjust the information entropy value according to the user's understanding of the uniform distribution of the data set, so as to make the application of information entropy closer to actual needs and reflect the uncertainty or confusion of the data set, helping us to analyze the data with a richer perspective. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 The data processing flow of the system disclosed in the present invention is illustrated. DETAILED DESCRIPTION

[0051] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0052] In the embodiment of the present invention, a medical data set is taken as an example, and the data of the medical institution involving the patient's medical information is used as the original data source to implement an evaluation system for the information volume of the data set based on information entropy and weighted algorithm provided by the present invention, which specifically includes the following parts:

[0053] Data set selection unit: used to select fields that need to calculate information from the platform database to build a data table. During the sample collection process, the number of samples obtained was 16,071,817. The collection fields included patient identity information fields (name, ID number, ID type, gender, date of birth), medical information (medical card, medical card type, outpatient / inpatient sign, date of visit, diagnosis ICD code, diagnosis ICD name), doctor's order / prescription information (prescription time, prescription code, prescription name, number of drugs dispensed), and cost information (outpatient / inpatient medical expenses, outpatient / inpatient drug expenses), totaling 17 variables;

[0054] Dataset annotation unit: used to annotate the fields of the data table in the dataset selection unit with field type, field attention level, and uniform expectation. Annotation examples are shown in Table 1:

[0055] The field type annotations are mainly divided into three categories: serial type 0, discrete type 1 and continuous type 2;

[0056] (a) Serial number type: Serial number type mainly refers to relatively independent fields such as the primary key, serial number, and record time of this data table. Such fields do not participate in information statistics (the information volume of such fields is often 0, which does not affect the final result, but can reduce the model calculation time);

[0057] (b) Discrete: Discrete variables mainly refer to those variables that can only take specific values, which are usually integers or can be listed one by one. The values ​​of discrete variables are not continuous, but can be clearly distinguished. This type of variable is usually used for counting or classification. It can be directly put into the model calculation.

[0058] (c) Continuous: A continuous variable is a variable that can take any real value within a range. Its value is continuous and can theoretically take an infinite number of values. Continuous variables are used for measurement or measurement, such as length, weight, temperature, amount, etc.

[0059] Field attention level annotation (not labeled for serial number type) is mainly divided into three categories: low 1, medium 2 and high 3; average expectation annotation (not labeled for serial number type) is divided into two categories: expected unevenness 0 and expected evenness 1.

[0060] Table 1 Field annotation example

[0061]

[0062] 3. Dataset information processing unit: preprocess and discretize the data table constructed in the data set selection unit. Data preprocessing mainly involves data cleaning, valid value extraction, and data normalization of the data table; data discretization is used to discretize the continuous fields marked in the data set marking unit into equal-width fields to form discrete fields.

[0063] (1) Data preprocessing

[0064] I. Clean invalid characters in text fields, such as special symbols, spaces, invisible characters, etc. After the initial cleaning, only invalid characters are left, which are converted to null; unify English uppercase and lowercase letters; unify half-width and full-width symbols;

[0065] II. Unify the data format of numeric fields, for example, convert the number of people to BIGINT, convert the cost to DECIMAL(28,4), and convert null to 0;

[0066] III. Perform dictionary data normalization mapping for dictionary fields based on national standards, industry standards or specific standards. For example, for gender fields, convert and retain valid results based on the national standard human gender code GB / T 2261.1-2003;

[0067] IV. Unify the date format to yyyy-MM-dd and the time format to yyyy-MM-dd hh:mm:ss;

[0068] V. Remove duplicate redundant data;

[0069] After data preprocessing, the total number of samples is 15,845,378.

[0070] (2) Data discretization

[0071] (a) First, filter out continuous fields, such as medical expenses and drug expenses, based on field type annotation;

[0072] (b) Standardize the field values ​​one by one and transform the data into a distribution with a mean of 0 and a standard deviation of 1. The formula is as follows:

[0073]

[0074] In the formula, y is the new value after the original value is standardized, x is the original value, μ is the mean of the column, and σ is the standard deviation of the column;

[0075] (c) Then determine the number of intervals to be divided (n=10) and perform equal-width discretization, so that the original distribution pattern of the data can be retained as much as possible during the discretization process.

[0076] 4. Information entropy calculation model: Construct an information entropy calculation model, including frequency calculation unit, field information entropy calculation unit, weight distribution calculation unit, and weighted algorithm calculation unit.

[0077] (1) Frequency calculation unit: First, the total number of all possible events in a field in the statistical data set is N, where N is the total number of samples 15845378 and the number of each event m i , calculate the probability of each event, for each event x i , calculate the probability of its occurrence, that is, the frequency Taking the field certificate type as an example, the frequency calculation results are shown in Table 2:

[0078] Table 2 Frequency statistics of certificate types

[0079]

[0080] (2) The field information entropy calculation unit first calculates the probability of occurrence of each event p(x i ), and then calculate the information entropy value of the field according to the Shannon information entropy formula, the formula is as follows:

[0081]

[0082] Where n is the number of event categories.

[0083] Next, the field information entropy value is normalized. The information entropy values ​​calculated for each field of the data set need to be unified in dimension so that each value can be compared and analyzed on the same scale, thereby accurately reflecting the amount of information in each field. Because the value range of information entropy is [0, log2n], n is the number of categories in the field, so the formula is as follows:

[0084]

[0085] Among them, H(J) is the information entropy value, and the normalized information entropy value E(J) ranges from [0,1].

[0086] Finally, according to the uniform expected value a of the field of the data set annotation unit, if a=0, the user expects that the field value is unevenly distributed, then the field information value D(J)=1-E(J); if a=1, the user expects that the field value is evenly distributed, then the field information value D(J)=E(J); the calculation example is shown in Table 3:

[0087] Table 3 Field information entropy calculation example

[0088]

[0089]

[0090] (3) The weight distribution calculation unit distributes the calculated information value D of each field by weight, and first performs a preliminary weight calculation, and the formula is as follows:

[0091]

[0092] Then, according to the degree of attention paid to the fields in the dataset annotation unit, the weight is adjusted to obtain a new weight value. The weight adjustment formula is as follows:

[0093]

[0094] Where b is the field attention value of the dataset annotation unit. The calculation example is shown in Table 4:

[0095] Table 4 Weight distribution calculation example

[0096]

[0097] (4) The weighted algorithm calculation unit calculates the information value D of each field in the data set. i size, and weight W i Size, calculate the information value Score of the data set, the result range is [0,1], and the calculation formula is:

[0098]

Claims

1. A system for evaluating the information content of a data set based on information entropy and weighted algorithm, characterized in that: include: Dataset selection unit: used to select the data set or related fields from the platform database that need to calculate the amount of information to construct the data set; Dataset annotation unit: used to annotate the fields of the dataset in the dataset selection unit with the field type, field attention level, and field uniform expectation; Data set information processing unit: used to perform data preprocessing and data discretization on the information data in the data set selection unit, where: Data preprocessing, which is used to clean the data set, extract valid values, and normalize the data; Data discretization is used to discretize the continuous fields marked in the dataset annotation unit into discrete fields with equal width; Information entropy calculation model: used to calculate the value Score of the amount of information in a data set to determine the amount of information contained in the data set. It further includes a frequency calculation unit, a field information entropy calculation unit, a weight distribution calculation unit, and a weighted algorithm calculation unit, where: The frequency calculation unit counts the total number N of all possible events for each field in the statistical data set, as well as the number m of each event i , for each event x i , calculate the probability of its occurrence p(x i ); The field information entropy calculation unit calculates the probability p(x i ) is substituted into the Shannon information entropy calculation formula to obtain the information entropy value, and then normalized to unify the information entropy calculated for each field; according to the uniform expectation of the field of the data set annotation unit, if the expected field data is evenly distributed, the normalized information entropy value is directly used; if the expected field data is unevenly distributed, the complementary number of the normalized information entropy value is used as the information value score of the field; The weight distribution calculation unit first calculates the weight of the information value of each field, and then adjusts the weight according to the field attention level of the data set annotation unit to obtain a new weight value; The weighted algorithm calculation unit performs a weighted summation on the information value and weight value of all fields to obtain the information value Score of the data set.

2. The system for evaluating the amount of information in a data set based on information entropy and weighted algorithm as claimed in claim 1, characterized in that: The data set selection unit selects the data set to be calculated, or selects the field to be calculated in the data set, and saves it in the form of a database table.

3. The system for evaluating the amount of information in a data set based on information entropy and weighted algorithm as claimed in claim 1, characterized in that: In the data set annotation unit, the field type annotation is divided into: serial number type, continuous type and discrete type, the field attention level annotation is divided into: low, medium and high, and the field uniform expectation annotation is divided into: uniform and uneven.

4. The system for evaluating the amount of information in a data set based on information entropy and weighted algorithm as claimed in claim 1, characterized in that: In the data set information processing unit, the data preprocessing includes the following steps: Clean invalid characters from text fields. If only invalid characters remain after the initial cleaning, convert them to null. Unify uppercase and lowercase letters in English. Unify half-width and full-width symbols. Unify the data format for numeric fields; Normalize and map dictionary data for dictionary fields based on national standards, industry standards or specific standards; Process the date types in a unified format; Remove duplicate redundant data.

5. The system for evaluating the amount of information in a data set based on information entropy and weighted algorithm as claimed in claim 1, characterized in that: In the data set information processing unit, the data discretization includes the following steps: Filter out continuous fields based on the dataset annotation units; Standardize the continuous field data and transform the data into a distribution with a mean of 0 and a standard deviation of 1. The formula is as follows: In the formula, y is the new value after the original value is standardized, x is the original value, μ is the mean of the column, and σ is the standard deviation of the column; Determine the number of divided intervals n and perform equal-width discretization.

6. The system for evaluating the amount of information in a data set based on information entropy and weighted algorithm as claimed in claim 1, characterized in that: The frequency calculation unit counts the total number N of all possible events in a field in the statistical data set, as well as the number m of each event i , calculate the probability of each event, for each event x i , calculate the probability of its occurrence 7. The system for evaluating the amount of information in a data set based on information entropy and weighted algorithm as claimed in claim 1, characterized in that: The field information entropy calculation unit calculates each event x according to the frequency calculation unit. i The probability of occurrence p(x i ), and then calculate the information entropy value H(X) of the field according to the Shannon information entropy formula. The formula is as follows: Where n is the number of event categories.

8. The system for evaluating the amount of information in a data set based on information entropy and weighted algorithm as claimed in claim 1, characterized in that: In the field information entropy calculation unit: The information entropy value is normalized to unify the information entropy values ​​calculated for each field of the data set. The value range of information entropy is [0, log2n], where n is the number of categories of the indicator. The formula is as follows: Among them, H(J) is the information entropy value, and the normalized information entropy value E(J) ranges from [0,1]; According to the uniform expected value a of the field of the annotation unit of the data set, if a=0, the user expects that the field value is unevenly distributed, then the field information value D(J)=1-E(J); if a=1, the user expects that the field value is evenly distributed, then the field information value D(J)=E(J).

9. The system for evaluating the amount of information in a data set based on information entropy and weighted algorithm as claimed in claim 1, characterized in that: The weight distribution calculation unit calculates the size D of each information entropy value D(J) after the data set is calculated. i , calculate its weight w i , the formula is as follows: Then, according to the degree of attention paid to the fields in the dataset annotation unit, the weight adjustment is performed. The weight adjustment formula is as follows: Wherein, b is the attention level value of the field.

10. The system for evaluating the information volume of a data set based on information entropy and weighted algorithm according to claim 1, characterized in that: The weighted algorithm calculation unit calculates the information value D of each field in the data set according to the information value D i size, and weight W i Size, calculate the information value Score of the data set, Get the information value of the data set whose calculation result range is between [0,1].

Images