Dangerous chemical production data quality evaluation method and system

Abstract

The embodiment of the invention provides a dangerous chemical production data quality evaluation method and system, and belongs to the technical field of data processing. The method comprises the steps of collecting operation data of a to-be-evaluated hazardous chemical production enterprise; traversing each last-stage index in a pre-constructed multi-layer evaluation model, and calculating an evaluation value of each last-stage index based on the operation data; weighting each evaluation index based on an improved hybrid weighting model to obtain each weighted evaluation index; and performing step-by-step operation on the multi-layer evaluation model based on the evaluation value of each final-stage index and each weighted evaluation index to obtain a comprehensive evaluation value of the operation data of the corresponding hazardous chemical production enterprise. According to the scheme, the problems that a traditional data quality inspection means is difficult to support dangerous chemical production business requirements and does not have comprehensive quantitative evaluation are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method for evaluating the quality of hazardous chemical production data and a system for evaluating the quality of hazardous chemical production data. Background Art

[0002] Hazardous chemical production is the production of hazardous chemicals. Because its products are extremely threatening, it is necessary to achieve accurate monitoring during the production process and in the subsequent storage, transportation and management to avoid accidents that may cause major safety accidents. Especially in the production process, the production raw materials, intermediates and final products are concentrated. In the production environment that may be accompanied by high temperature and high pressure, the risk of major safety accidents due to operating errors or equipment problems increases.

[0003] Based on this, it is imperative to conduct precise supervision of hazardous chemical production processes. At present, the most effective solution is often to comprehensively collect production process data, and then identify abnormal data based on the production process data to identify suspected faulty production steps. In this way, the requirements for data quality are very high, because the accuracy of the data directly affects the accuracy of subsequent risk supervision. However, the existing hazardous chemical production data involves multiple types such as process, equipment monitoring, leakage alarm, etc., and there will be problems such as inconsistent data structure, a lot of dirty data in the data center, a lack of traditional data quality audit methods, difficulty in supporting hazardous chemical production business needs, and no comprehensive quantitative evaluation methods. In response to these problems, it is necessary to propose a new hazardous chemical production data quality evaluation scheme. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a method and system for hazardous chemical production data quality evaluation, so as to at least solve the problems of existing hazardous chemical production data quality evaluation schemes being difficult to support hazardous chemical production business needs and having no comprehensive quantitative evaluation method.

[0005] In order to achieve the above-mentioned purpose, the first aspect of the present invention provides a method for evaluating the quality of hazardous chemical production data, the method comprising: collecting the operating data of the hazardous chemical production enterprise to be evaluated; traversing each terminal indicator in a pre-constructed multi-layer evaluation model, and calculating the evaluation value of each terminal indicator based on the operating data; weighting each evaluation indicator based on an improved hybrid weighting model to obtain each weighted evaluation indicator; based on the evaluation value of each terminal indicator and each weighted evaluation indicator, performing step-by-step calculations on the multi-layer evaluation model to obtain a comprehensive evaluation value of the operating data of the corresponding hazardous chemical production enterprise.

[0006] Optionally, the method also includes constructing a multi-layer evaluation model, including: pushing preset data quality evaluation indicators and collecting user confirmation information; identifying the user's sensitive information based on the user's confirmation information, and determining the data quality evaluation indicators that need to be evaluated based on the sensitive information; wherein the data quality evaluation indicators include: data standardization indicators, data integrity indicators, data accuracy indicators, data consistency indicators, data timeliness indicators and data accessibility indicators; constructing a multi-layer evaluation model based on the determined data quality evaluation indicators that need to be evaluated.

[0007] Optionally, the construction of a multi-layer evaluation model based on the determined data quality evaluation indicators that need to be evaluated includes: based on the determined data quality evaluation indicators that need to be evaluated, decomposing the associated factors to obtain the associated factors under each data quality evaluation indicator; based on each associated factor, generating subordinate indicators under the corresponding data quality evaluation indicator; taking all subordinate indicators as the final indicators, taking each data quality evaluation indicator as a secondary indicator, and taking the evaluation quantitative indicators of the operating data of the hazardous chemical production enterprise to be evaluated as the top-level indicators, to form a corresponding multi-layer evaluation model.

[0008] Optionally, the subordinate indicators of the data standardization indicator include: data type accuracy, which indicates the proportion of data that conforms to the specified data type in the total data; constraint information accuracy, which indicates the proportion of data that conforms to the specified constraint conditions in the total data; verification rule accuracy, which indicates the proportion of data that conforms to the specified verification rules in the total data; default value accuracy, which indicates the proportion of data that conforms to the default value in the total data; data skewness coefficient, which indicates the bias of the data set characteristics.

[0009] Optionally, subordinate indicators of the data integrity indicator include: foreign key coverage rate, which indicates the proportion of data that meets the preset requirements of foreign key constraints in the total data; trigger compliance rate, which indicates the proportion of data that meets the preset requirements of triggers in the total data; primary key constraint compliance rate, which indicates the proportion of data that meets the preset requirements of unique primary key values ​​in the total data; and the number of unique constraints, which indicates the number of unique constraints that meet the preset requirements.

[0010] Optionally, the subordinate indicators of the data accuracy indicator include: content correctness rate, used to indicate the accuracy of data content; data duplication rate, used to indicate the data duplication values ​​of each data column; data uniqueness compliance rate, used to indicate the proportion of data containing data duplication values ​​in the total data; dirty data occurrence rate, used to indicate the proportion of data containing data outliers in the total data.

[0011] Optionally, subordinate indicators of the data consistency indicator include: data content consistency, which is used to indicate the consistency of data content in each data table of the operating data; data relationship consistency, which is used to indicate the consistency of data key values ​​in each data table of the operating data.

[0012] Optionally, the subordinate indicators of the data timeliness index include: time period accuracy rate, which is used to indicate the proportion of data that meets the specified time difference in the total data; time point timeliness rate, which is used to indicate the proportion of data that meets the time point timeliness in the total data; time series compliance rate, which is used to indicate the proportion of data that meets the time series in the total data.

[0013] Optionally, subordinate indicators of the data accessibility indicator include: accessibility, which is used to indicate the difficulty of obtaining data; usability, which is used to indicate the usability of data within its data life cycle.

[0014] Optionally, the method of weighting each evaluation indicator based on the improved hybrid weighting model includes: constructing a judgment matrix for each evaluation indicator based on the 1-9 scaling method; determining the subjective weight of each indicator based on the judgment matrix; determining the objective weight of each indicator based on the entropy weight method; and combining the subjective weight and the objective weight based on the game theory method to obtain a combined weight.

[0015] Optionally, the combining of the subjective weight and the objective weight based on the game theory method to obtain the combined weight includes: constructing an optimization problem with the goal of minimizing the sum of the combined weight and the deviation between the subjective weight and the objective weight; and adjusting the linear combination coefficients in the optimization problem step by step until an iteration termination condition is met, thereby obtaining an optimal combined weight, the expression of which is:

[0016]

[0017] Among them, W′ is the optimal combination weight; α′1 and α′2 are linear combination coefficients; W1 is the subjective weight; and W2 is the objective weight.

[0018] Optionally, the multi-layer evaluation model is operated step by step based on the evaluation value of each final-level indicator and each weighted evaluation indicator to obtain a comprehensive evaluation value of the operation data of the corresponding hazardous chemical production enterprise, including: in a preset grey clustering evaluation model, based on the value of each evaluation indicator and the preset value classification rule, the center point of each grey class is determined, and based on the center point of each grey class, the possibility function of each evaluation indicator and each grey class is obtained; based on the possibility function, the grey fusion coefficient of each evaluation indicator with respect to each grey class is calculated respectively, expressed as:

[0019]

[0020] Among them, j is the jth evaluation index; k is the kth gray class; is the possibility function of evaluation index j and gray class k; is the weight of the evaluation index j in the gray class k; based on the maximum value criterion, the gray class k corresponding to the object when the clustering coefficient is the largest is selected as the comprehensive evaluation value of the operating data of the corresponding hazardous chemical production enterprise.

[0021] The second aspect of the present invention provides a hazardous chemical production data quality evaluation system, the system comprising: a collection unit, used to collect operating data of a hazardous chemical production enterprise to be evaluated; a processing unit, used to traverse each terminal indicator in a pre-constructed multi-layer evaluation model, and calculate the evaluation value of each terminal indicator based on the preprocessed operating data; a weighting unit, used to weight each evaluation indicator based on an improved hybrid weighting model to obtain each weighted evaluation indicator; an evaluation unit, used to perform step-by-step calculations on the multi-layer evaluation model based on the evaluation value of each terminal indicator and each weighted evaluation indicator to obtain a comprehensive evaluation value of the operating data of the corresponding hazardous chemical production enterprise.

[0022] On the other hand, the present invention provides a computer-readable storage medium having instructions stored thereon, which, when executed on a computer, enables the computer to execute the above-mentioned method for evaluating the quality of hazardous chemical production data.

[0023] Through the above technical solution, the scheme of the present invention comprehensively collects the operating data of hazardous chemical production, then splits and classifies the collected data, and calculates the value of each final indicator in parallel to obtain the value score of each basic data. In view of the evaluation needs of hazardous chemical production, an applicable multi-layer evaluation model is proposed. Based on the model and the value score of each basic data, after the improved weighting model is weighted, the quality of hazardous chemical production data is comprehensively evaluated, and quantitative evaluation is achieved, which solves the problems of the lack of traditional data quality audit means, the difficulty in supporting the business needs of hazardous chemical production, and the lack of comprehensive quantitative evaluation.

[0024] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the accompanying drawings:

[0026] Figure 1 It is a flowchart of the steps of a method for evaluating the quality of hazardous chemical production data provided by an embodiment of the present invention;

[0027] Figure 2It is a flow chart of calculating the evaluation values ​​of each final-level indicator provided by one embodiment of the present invention;

[0028] Figure 3 is a combination weight determination flow chart provided by an embodiment of the present invention;

[0029] Figure 4 It is a system structure diagram of a hazardous chemical production data quality evaluation system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.

[0031] Hazardous chemical production is the production of hazardous chemicals. Because its products are extremely threatening, it is necessary to achieve accurate monitoring during the production process and in the subsequent storage, transportation and management to avoid accidents that may cause major safety accidents. Especially in the production process, the production raw materials, intermediates and final products are concentrated. In the production environment that may be accompanied by high temperature and high pressure, the risk of major safety accidents due to operating errors or equipment problems increases.

[0032] Based on this, it is imperative to conduct precise supervision of hazardous chemical production processes. At present, the most effective solution is often to comprehensively collect production process data, and then identify abnormal data based on the production process data to identify suspected faulty production steps. In this way, the data quality requirements are very high, because the accuracy of the data directly affects the accuracy of subsequent risk supervision.

[0033] However, existing hazardous chemical production data involves multiple types such as process, equipment monitoring, leakage alarm, etc., and there will be problems such as inconsistent data structure, a lot of dirty data in the data center, lack of traditional data quality auditing means, difficulty in supporting hazardous chemical production business needs, and no comprehensive quantitative evaluation methods. In response to these problems, the present invention proposes a data quality evaluation method for hazardous chemical production. The present invention scheme comprehensively collects the operating data of hazardous chemical production, and then splits and classifies the collected data, and calculates the value of each terminal indicator in parallel to obtain the value score of each basic data. In response to the evaluation needs of hazardous chemical production, a suitable multi-layer evaluation model is proposed. Based on this model and the value scores of each basic data, after weighting by the improved weighting model, the quality of hazardous chemical production data is comprehensively evaluated, and quantitative evaluation is achieved, which solves the problems of lack of traditional data quality auditing means, difficulty in supporting hazardous chemical production business needs, and no comprehensive quantitative evaluation.

[0034] Figure 1This is a method flow chart of a method for evaluating the quality of hazardous chemical production data provided by an embodiment of the present invention. Figure 1 As shown, an embodiment of the present invention provides a method for evaluating the quality of hazardous chemical production data, the method comprising:

[0035] Step S10: Collect the operating data of the hazardous chemical production enterprise to be evaluated.

[0036] Specifically, as explained above, existing hazardous chemical production data involves multiple types such as process, equipment monitoring, and leakage alarm, and there will be problems such as inconsistent data structure and a lot of dirty data in the data set. In order to conduct a comprehensive and accurate quality evaluation of hazardous chemical generation data, it is necessary to conduct comprehensive data collection to avoid misidentification of data failures or omissions caused by missed data. This ensures the accuracy of data quality evaluation, provides a basis for subsequent monitoring of hazardous chemical production processes, ensures the stability of hazardous chemical production, and greatly ensures the accuracy of risk assessment of hazardous chemical generation processes.

[0037] Based on this, the scheme of the present invention conducts comprehensive collection of the operating data of the hazardous chemical production enterprise to be evaluated. In order to avoid data omissions, during the initial data collection, the equipment data, production process data, log data and other data of the production process will be collected through the enterprise's main control system (such as DCS system) or other data management systems. However, if these data are processed directly later, the processing delay will increase due to the large amount of data, resulting in a longer cycle of the entire evaluation process, which is prone to failure delays.

[0038] Preferably, in order to solve this problem, the scheme of the present invention will also pre-process the data after completing data collection. The purpose of data pre-processing is to reduce redundant data and reduce the data volume. In the collected data, there is a large amount of invalid data, such as backup data, identification data, etc. These data have little effect on the subsequent data quality evaluation, but the data itself is not small. By filtering these redundant data through corresponding data filtering means and retaining only valid data, the data volume can be greatly reduced, the data evaluation efficiency can be improved, the evaluation time can be shortened, and the system response speed can be ensured.

[0039] Step S20: traverse each final-level indicator in the pre-constructed multi-layer evaluation model, and calculate the evaluation value of each final-level indicator based on the operating data.

[0040] Specifically, a multi-layer evaluation model is constructed, including: pushing preset data quality evaluation indicators and collecting user confirmation information; identifying the user's sensitive information based on the user's confirmation information, and determining the data quality evaluation indicators that need to be evaluated based on the sensitive information; wherein the data quality evaluation indicators include: data standardization indicators, data integrity indicators, data accuracy indicators, data consistency indicators, data timeliness indicators and data accessibility indicators; and performing a multi-layer evaluation model based on the determined data quality evaluation indicators that need to be evaluated.

[0041] Embodiment 1: The present invention adopts 6 first-level evaluation rules of standardization, completeness, accuracy, consistency, timeliness and accessibility of data quality and 21 second-level evaluation rules of data standards, data models, metadata, business rules, authoritative reference data, security specifications, data element integrity, data record integrity, data content correctness, data format compliance, data repetition rate, data uniqueness, dirty data occurrence rate, same data consistency, related data consistency, time period-based correctness, time point-based timeliness, temporality, accessibility and availability, to construct 16 quantitative evaluation indicators suitable for chemical safety data.

[0042] Preferably, it can be seen that if a complete data quality evaluation is required, it is necessary to evaluate in parallel or one by one from multiple aspects, and the amount of data processing is still very large. In fact, the user may only be interested in some of the indicators. For example, the user needs to accurately control the generation process, so he is relatively sensitive to the integrity and accuracy of the data. If the user first evaluates only these two indicators in a very short time, the user can only select these two indicators for quality evaluation.

[0043] Embodiment 2: If the user only selects the integrity evaluation indicator, the user's sensitive information is identified as data integrity, and data integrity is used as the top-level indicator to construct a multi-level evaluation model, and data factors that are directly or indirectly related to the top-level indicator are directly or indirectly used as subordinate indicators of the top-level indicator. In this way, the generation process of the multi-level evaluation model is differentiated, and differentiated models are generated according to the sensitive needs of users, ensuring that the generated model is suitable for the current user, reducing the waste of invalid processing time, and improving data processing efficiency. Of course, if the user wants a complete data quality evaluation, it is also possible to construct a complete evaluation model based on the solution of embodiment 1. The solution of the present invention proposes a personalized evaluation solution.

[0044] Preferably, the multi-layer evaluation model is carried out based on the determined data quality evaluation indicators that need to be evaluated, including: based on the determined data quality evaluation indicators that need to be evaluated, decomposing the associated factors to obtain the associated factors under each data quality evaluation indicator; based on each associated factor, generating subordinate indicators under the corresponding data quality evaluation indicator; taking all subordinate indicators as the final indicators, taking each data quality evaluation indicator as a secondary indicator, and taking the evaluation quantitative indicators of the operating data of the hazardous chemical production enterprise to be evaluated as the top-level indicators, to form a corresponding multi-layer evaluation model.

[0045] Embodiment three:

[0046] Preferably, the subordinate indicators of the data standardization indicator include: data type accuracy, which indicates the proportion of data that conforms to the specified data type in the total data; constraint information accuracy, which indicates the proportion of data that conforms to the specified constraints in the total data; validation rule accuracy, which indicates the proportion of data that conforms to the specified validation rules in the total data; default value accuracy, which indicates the proportion of data that conforms to the default value in the total data.

[0047] Specifically, data type accuracy: used to calculate the proportion of data that conforms to the specified data type to ensure the standardization of data; constraint information accuracy: used to calculate the proportion of data that conforms to the specified constraint conditions to ensure the standardization of data; validation rule accuracy: used to calculate the proportion of data that conforms to the specified validation rules to ensure the standardization of data; default value accuracy: used to calculate the proportion of data that conforms to the default value to ensure the standardization of data.

[0048] Preferably, the subordinate indicators of the data integrity indicator include: foreign key coverage rate, which indicates the proportion of data that meets the preset requirements of foreign key constraints in the total data; trigger compliance rate, which indicates the proportion of data that meets the preset requirements of triggers in the total data; primary key constraint compliance rate, which indicates the proportion of data that meets the preset requirements of unique primary key values ​​in the total data; number of unique constraints, which indicates the proportion of data that meets the preset requirements of unique constraints in the total data.

[0049] Specifically, foreign key coverage rate: used to calculate the proportion of data that meets the requirements of foreign key constraints to ensure data integrity; trigger compliance rate: used to calculate the proportion of data that meets the requirements of triggers to ensure data integrity; primary key constraint compliance rate: used to calculate the proportion of data that meets the requirements of unique primary key values ​​to ensure data integrity; number of unique constraints: used to calculate the proportion of data that meets the requirements of unique constraints to ensure data integrity.

[0050] Preferably, the subordinate indicators of the data accuracy indicator include: content correctness rate, which is used to indicate the accuracy of data content; data repetition rate, which is used to indicate the data repetition values ​​of each data column; data uniqueness compliance rate, which is used to indicate the proportion of data containing data repetition values ​​in the total data; dirty data occurrence rate, which is used to indicate the proportion of data containing data outliers in the total data.

[0051] Specifically, content accuracy rate: used to calculate the correctness of data content to ensure accuracy rules; data duplication rate: used to calculate which data columns contain the most duplicate values ​​to ensure data accuracy; data uniqueness compliance rate: used to calculate the proportion of data containing duplicate values ​​to ensure data accuracy; dirty data occurrence rate: used to calculate the proportion of data outliers to ensure data accuracy.

[0052] Preferably, subordinate indicators of the data consistency indicator include: data content consistency, which is used to indicate the consistency of data content in each data table of the operating data; data relationship consistency, which is used to indicate the consistency of data key values ​​in each data table of the operating data.

[0053] Preferably, the subordinate indicators of the data timeliness index include: time period accuracy rate, which is used to indicate the proportion of data that meets the specified time difference in the total data; time point timeliness rate, which is used to indicate the proportion of data that meets the time point timeliness in the total data; time series compliance rate, which is used to indicate the proportion of data that meets the time series in the total data.

[0054] Specifically, the time period accuracy rate is used to calculate the proportion of time differences that meet the requirements to ensure the timeliness of the data; the time point timeliness rate is used to calculate the proportion of data based on the timeliness of the time point to ensure the timeliness of the data; the time series compliance rate is used to calculate the proportion of data that meets the time series to ensure the timeliness of the data.

[0055] Preferably, the subordinate indicators of the data accessibility indicator include: availability, which is used to indicate the difficulty of obtaining data; usability, which is used to indicate the usability of data within its data life cycle.

[0056] Specifically, Figure 2 Calculating the evaluation value of each final-level indicator based on the operating data includes the following steps:

[0057] Step S201: Calculate the final indicator evaluation value of the data standardization indicator.

[0058] Specific:

[0059] 1) Data type accuracy: For key database data sets, use SQL statements to return the data type of each column in the specified table to ensure that the data type of each column conforms to its specified data type. Calculate the number of columns that conform to the specified data type, find their proportion, and obtain the data type accuracy a1.

[0060] 2) Constraint information accuracy: For key database data sets, use SQL statements to return the constraint information of all columns in the specified table, including primary keys, foreign keys, unique constraints, etc., to ensure that the value of each column meets the specified constraint conditions. Calculate the number of columns that meet the specified constraint conditions, find their proportion, and obtain the constraint information accuracy a2.

[0061] 3) Verification rule accuracy: For key database data sets, use SQL statements to return the verification rule information of all columns in the specified table to ensure that the value of each column complies with its specified verification rules. Calculate the number of columns that comply with its specified verification rules, find out their proportion, and obtain the verification rule accuracy a3.

[0062] 4) Default value accuracy: For key database data sets, use SQL statements to return the default value of each column in the specified table to ensure that the default value of each column meets the requirements. Calculate the number of default values ​​that meet the specified default value, find their proportion, and obtain the default value accuracy a4.

[0063] 5) Data skewness coefficient, which indicates the bias of the data set features. Since the data in the data set is biased, the training effect of the machine learning model will be affected, so it is necessary to detect the feature distribution of the data set. This method uses the skewness coefficient of the data set in the specified data column to feedback the degree of data set deviation. The data deviation calculation formula is as follows:

[0064]

[0065] Where n is the number of samples, μ is the mean of the specified data column, and σ is the standard deviation of the specified data column. If the data of the specified data column is full data, the data skewness coefficient defaults to 0, and the data skewness coefficient a5 is obtained.

[0066] Step S202: Calculate the final level indicator evaluation value of the data integrity indicator.

[0067] Specifically, it includes two rules: data element integrity and data record integrity. Element integrity includes the integrity of entities, attributes, and relationships. Specifically combined with the characteristics of hazardous chemical production safety data, the normative rules can be fitted into the following evaluation method:

[0068] 1) Foreign key coverage: For key data sets in the database, use SQL statements to return information about all foreign key constraints in the specified table, including foreign key columns and corresponding reference tables and reference columns.

[0069] 2) Trigger compliance rate: For key database data sets, SQL statements are used to return information about all triggers defined on the specified table, including trigger name, trigger event, trigger table, and trigger execution statement. By checking this information, it can be ensured that the definition and execution of triggers will not damage data integrity.

[0070] Calculate the number of foreign key constraints and triggers that meet the requirements, find their proportion, and obtain the foreign key constraint and trigger compliance rates b1 and b2.

[0071] 3) Primary key constraint compliance rate: For the key data sets of the database, use SQL statements to return all column names of the primary key constraints in the specified table, and calculate the proportion of columns with unique primary key values ​​as the primary key constraint compliance rate b3.

[0072] 4) For the key data set of the database, use SQL statements to return all column names with unique constraints in the specified table, and calculate the proportion of columns with unique constraints as the number of unique constraints b4.

[0073] Step S203: Calculate the final indicator evaluation value of the data accuracy indicator.

[0074] Specific:

[0075] 1) Content accuracy: The average relative distance is used to measure the accuracy of the actual value. The average relative distance is obtained using the following formula:

[0076]

[0077] V is the data set, v' is the value of the specified data column, v is the true value of the data column, k is the number of data in the specified data column, and Dis(v',v) is the distance formula. When the value range of the specified data column is continuous, the distance formula is as follows: p is the data dimension:

[0078]

[0079] When the value range of the specified data column is categorical, the distance formula is as follows:

[0080]

[0081] When the value range of the specified data column is other comparable types, the cosine distance is used as follows:

[0082]

[0083] The similarity of two word vectors can be calculated as the accuracy of the content by loading the trained chemical safety production word2vec model. The content accuracy c1 is obtained.

[0084] 2) Data repetition rate: For key data sets in the database, SQL statements are used to return duplicate data and group them. The number of rows in each group and the percentage of each group are then calculated to identify which data columns in the database contain the most duplicate values, which is called the data repetition rate c2.

[0085] 3) Data uniqueness compliance rate: Detect the uniqueness of a column or multiple columns in the database. For the key data sets of the database, use SQL statements to group the specified columns and calculate the number of rows in each group. If the number of rows in a group is greater than 1, it means that there are duplicate values ​​in the column. The data uniqueness compliance rate c3 is calculated.

[0086] 4) Dirty data occurrence rate: Using the isolation forest algorithm, randomly select the point "a" to be separated; select a random data point "b" between the minimum and maximum values ​​and different from "a"; if the value of "b" is lower than the value of "a", the value of "b" becomes the new lower limit; if the value of "b" is greater than the value of "a", the value of "b" becomes the new upper limit; as long as there are data points other than "a" between the upper and lower limits, repeat the process. Compared with isolating non-outliers, it requires fewer splits to isolate outliers, that is, outliers have a lower isolation number than non-outliers. Therefore, if the isolation number of a data point is lower than the threshold, the data point is defined as an outlier. The dirty data occurrence rate c4 is calculated by calling the IsolationForest algorithm in the sklearn.ensemble package.

[0087] Step S204: Calculate the final index evaluation value of the data consistency index.

[0088] Specific:

[0089] 1) Data content consistency: For key data sets in the database, SQL statements are used to return all data verification rules in the specified table. By checking these rules, it can be ensured that the data content meets the expected rules and conditions. For key data sets in the database, SQL statements are used to return records that exist in Table 1 but not in Table 2, as well as their key values. By checking these records, it can be ensured that the relationship between the two tables meets expectations.

[0090] 2) Data relationship consistency: For key data sets in the database, use SQL statements to return all foreign key constraints in the specified table. By checking these constraints, you can ensure that the relationship between the tables meets expectations. For key data sets in the database, use SQL statements to return records that exist in one table but not in another table, as well as their key values. By checking these records, you can ensure that the relationship between the two tables meets expectations.

[0091] Step S205: Calculate the final indicator evaluation value of the data timeliness indicator.

[0092] Specific:

[0093] 1) Time period accuracy: For key data sets in the database, use SQL statements to return the time difference column values, calculate the proportion of time difference values ​​that meet the requirements, and obtain the time period accuracy e1.

[0094] 2) Time point timeliness: For key database data sets, use SQL statements to return the maximum timestamp, minimum timestamp, and the time difference between them (in seconds). By comparing the maximum timestamp and the current timestamp, it can be determined whether the data transmission is delayed. If the maximum timestamp is much later than the current timestamp, it means that there is a delay in data transmission. Calculate the proportion of data based on time point timeliness and obtain the time point timeliness rate e2.

[0095] 3) Time series correlation coefficient: The maximum value of the Pearson correlation coefficient and the Spearman correlation coefficient between the specified data column and time is calculated as the time series correlation coefficient, and the data time series conformity rate e3 is obtained. The calculation formula is:

[0096]

[0097] Among them, v i ′ is the value of the specified data column, t i The value of the time column.

[0098] Step S206: Calculate the final index evaluation value of the data accessibility index.

[0099] Specific:

[0100] 1) Accessibility: Accessibility refers to whether the data can be easily accessed. Methods for checking accessibility include checking whether the database connection is normal, whether the network is available, and whether the database server is online.

[0101] 2) Availability: When checking availability during the life cycle, you need to check whether the data follows the best practices for data management, such as backup and restore strategies, data encryption, data archiving, etc. Monitor database connections and network status: You need to monitor database connections and network status to ensure that the connection between the database server and the application remains stable at all times. Implement data management best practices: You need to ensure that database data follows the best practices for data management, such as backup and restore strategies, data encryption, data archiving, etc. Test data recovery strategies: You need to regularly test the recovery strategy for database data to ensure that data can be quickly recovered in the event of a failure or other problem.

[0102] Embodiment 4: A data quality comprehensive evaluation model is constructed according to the above data quality evaluation rule quantization matrix to obtain an overall evaluation of the data quality.

[0103] Step S30: weighting each evaluation index based on the improved hybrid weighting model to obtain each weighted evaluation index.

[0104] Specifically, a judgment matrix for each evaluation index is constructed based on the 1-9 scaling method; subjective weights of each index are determined based on the judgment matrix; objective weights of each index are determined based on the entropy weight method; and the subjective weights and objective weights are combined based on the game theory method to obtain a combined weight.

[0105] Embodiment 5, as Figure 3 :

[0106] Step S301: Determine subjective weights using the analytic hierarchy process.

[0107] Specifically, Santy's 1-9 scale method is used to compare factors and indicators at different levels to determine their relative importance. Finally, the judgment matrix is ​​obtained:

[0108]

[0109] The next step is to calculate the weight vector. The weight of each factor and indicator is calculated based on the judgment matrix. The calculation method is to normalize each column of the judgment matrix and then add the weight of each row to get the total weight of each factor and indicator.

[0110] When the indicator level is more than one, its weight needs to be calculated hierarchically. First, confirm that the main level contains m1 elements. The total ranking weights of the levels are The bottom layer B contains m2 sub-factors They are for factor A j The weights are Current level B i Not in main level A j When b ij = 0, then the underlying sub-factor B i The total weight formula is:

[0111]

[0112] Then, a consistency check is performed. For judgment matrices at different levels, a consistency check is required to ensure the consistency of the data in the judgment matrix. If it passes, the subjective weight W1 is obtained. CI(j) and RI(j) are used to represent the subjective weight W1 of the main factor A. j The corresponding sub-factor B i The CI and RI values ​​calculated by single sorting are then tested for consistency using the following formula:

[0113]

[0114] When CR < 0.1, the consistency of the judgment matrix is ​​considered acceptable and passes the verification. If passed, the subjective weight W1 is obtained.

[0115] Step S302: Determine the objective weight using the entropy weight method.

[0116] For m1×m2 indicators, there are n data, and the record of the i-th indicator in the j-th data is x ij .

[0117] Specifically, the evaluation index data is normalized to obtain:

[0118]

[0119] Calculate the information entropy of each indicator. The smaller the information entropy, the greater the contribution of the indicator. The calculation formula is:

[0120]

[0121] If y ij =0, then define:

[0122] y ij y ij =0

[0123] Entropy value E i ∈[0, 1]. i The larger the value is, the greater the degree of differentiation of indicator i is, and the more information is obtained. Therefore, a higher weight should be given to this index. Therefore, the objective weight is:

[0124]

[0125] The weight of each indicator is calculated by information entropy. The larger the weight, the greater the impact of the indicator on the evaluation result. The consistency test of each indicator weight of the indicator system is carried out to ensure the reliability and validity of the data. In this way, the objective weight W2 is obtained.

[0126] Step S303: Game theory is used to perform subjective and objective comprehensive weighting to find an equilibrium solution.

[0127] Specifically, according to the idea of ​​game theory, the objective function is established, with the goal of minimizing the sum of the deviations between the indicator combination weight W and W1 and W2, and seeking the optimal linear combination coefficients α1 and α2. The indicator combination weight at this time is the optimal combination weight W':

[0128] min||WW i ||2 (i=1,2)

[0129] Finally, the comprehensive weight is obtained:

[0130]

[0131] Among them, W′ is the optimal combination weight; α′1 and α′2 are linear combination coefficients; W1 is the subjective weight; and W2 is the objective weight.

[0132] Step S40: Based on the evaluation values ​​of the final level indicators and the weighted evaluation indicators, the multi-level evaluation model is operated level by level to obtain a comprehensive evaluation value of the operating data of the hazardous chemical production enterprise to be evaluated.

[0133] Specifically, after completing the indicator weighting, a multi-level evaluation system can be calculated step by step based on the evaluation value calculation value of each indicator and the corresponding weight to obtain the final evaluation result of the user interest indicator. The evaluation result can be the evaluation result of a single indicator or the combined result of multiple indicators.

[0134] Preferably, the multi-layer evaluation model is operated step by step based on the evaluation value of each final-level indicator and each weighted evaluation indicator to obtain a comprehensive evaluation value of the operation data of the corresponding hazardous chemical production enterprise, including: in a preset grey clustering evaluation model, based on the value of each evaluation indicator and the preset value classification rule, the center point of each grey class is determined, and based on the center point of each grey class, the possibility function of each evaluation indicator and each grey class is obtained; based on the possibility function, the grey fusion coefficient of each evaluation indicator with respect to each grey class is calculated respectively, which is expressed as:

[0135]

[0136] Among them, j is the jth evaluation index; k is the kth gray class; is the possibility function of evaluation index j and gray class k; is the weight of the evaluation index j in the gray class k; based on the maximum value criterion, the gray class k corresponding to the object when the clustering coefficient is the largest is selected as the comprehensive evaluation value of the operating data of the corresponding hazardous chemical production enterprise.

[0137] Embodiment six:

[0138] Grey clustering evaluation models include grey correlation clustering, grey variable weight clustering, grey fixed weight clustering, and grey clustering with mixed possibility function. This paper uses a grey evaluation model based on the combination of hierarchical analysis method and mixed possibility function to evaluate hazardous chemical production safety data. The main steps are as follows:

[0139] First, the values ​​of hazardous chemical production safety data quality indicators are divided into s levels, and s gray subcategories of the indicators are determined. According to the index value range and expert experience, the index value range and division interval are determined by combining interpolation and extrapolation, and finally the center point of each gray class is determined. Finally, we get the possible degree function of index j about k gray class

[0140] when When grayscale

[0141]

[0142] When the evaluation index is a positive index, k = 1 grayscale When the evaluation index is a reverse index, k = 5 grayscale

[0143]

[0144] When the evaluation index is a positive index, k = 5 grayscale When the evaluation index is a reverse index, k = 1 grayscale

[0145]

[0146] Calculate the grey variable weight clustering coefficient of index j with respect to k grey classes:

[0147]

[0148] is the weight of index j in subclass k obtained by the AHP method. Finally, the clustering coefficient vector σ of object i is obtained i , and select the gray class k corresponding to the object when the clustering coefficient is the largest according to the maximum criterion to determine the evaluation category to which the object belongs.

[0149]

[0150]

[0151] Figure 4 This is a system structure diagram of a hazardous chemical production data quality evaluation system provided by an embodiment of the present invention. Figure 4 As shown, an embodiment of the present invention provides a hazardous chemical production data quality evaluation system, the system comprising: a collection unit, used to collect the operating data of the hazardous chemical production enterprise to be evaluated, and preprocess the operating data to obtain the preprocessed operating data; a processing unit, used to traverse each terminal indicator in the pre-constructed multi-layer evaluation model, and calculate the evaluation value of each terminal indicator based on the preprocessed operating data; a weighting unit, used to weight each evaluation indicator based on an improved hybrid weighting model to obtain each weighted evaluation indicator; an evaluation unit, used to perform step-by-step calculations on the multi-layer evaluation model based on the evaluation value of each terminal indicator and each weighted evaluation indicator to obtain a comprehensive evaluation value of the operating data of the hazardous chemical production enterprise to be evaluated.

[0152] An embodiment of the present invention further provides a computer-readable storage medium, on which instructions are stored, which, when executed on a computer, enable the computer to execute the above-mentioned hazardous chemical production data quality assessment method.

[0153] Those skilled in the art will understand that all or part of the steps in the method for implementing the above-mentioned embodiments can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including several instructions for making a single-chip microcomputer, a chip or a processor (processor) perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0154] The optional embodiments of the present invention are described in detail above in conjunction with the accompanying drawings. However, the embodiments of the present invention are not limited to the specific details in the above embodiments. Within the technical concept of the embodiments of the present invention, the technical scheme of the embodiments of the present invention can be subjected to a variety of simple modifications, and these simple modifications all belong to the protection scope of the embodiments of the present invention. It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the embodiments of the present invention will not further describe various possible combinations.

[0155] In addition, various embodiments of the present invention may be arbitrarily combined, and as long as they do not violate the concept of the embodiments of the present invention, they should also be regarded as the contents disclosed in the embodiments of the present invention.

Claims

1. A method for evaluating the quality of hazardous chemical production data, characterized in that: The method comprises: Collect the operating data of the hazardous chemical production enterprises to be evaluated; Traversing each final-level indicator in the pre-built multi-layer evaluation model, and calculating the evaluation value of each final-level indicator based on the operation data; Based on the improved hybrid weighting model, each evaluation index is weighted to obtain the weighted evaluation index; Based on the evaluation values ​​of each final-level indicator and each weighted evaluation indicator, the multi-layer evaluation model is operated step by step to obtain a comprehensive evaluation value of the operation data of the corresponding hazardous chemical production enterprise.

2. The method according to claim 1, characterized in that: The method further comprises constructing a multi-layer evaluation model, comprising: Push preset data quality evaluation indicators and collect user confirmation information; Based on the user's confirmation information, the user's sensitive information is identified, and based on the sensitive information, the data quality evaluation index to be evaluated is determined; wherein the data quality evaluation index includes: Data standardization indicators, data integrity indicators, data accuracy indicators, data consistency indicators, data timeliness indicators and data accessibility indicators; A multi-layer evaluation model is constructed based on the determined data quality evaluation indicators that need to be evaluated.

3. The method according to claim 2, characterized in that The construction of a multi-layer evaluation model based on the determined data quality evaluation indicators that need to be evaluated includes: Based on the determined data quality evaluation indicators that need to be evaluated, the correlation factors are decomposed to obtain the correlation factors under each data quality evaluation indicator; Based on each correlation factor, generate subordinate indicators under the corresponding data quality evaluation indicators; All subordinate indicators are taken as the final indicators, each data quality evaluation indicator is taken as the secondary indicator, and the evaluation quantitative indicators of the operating data of the hazardous chemical production enterprise to be evaluated are taken as the top-level indicators to form a corresponding multi-layer evaluation model.

4. The method according to claim 3, characterized in that The subordinate indicators of the data standardization indicators include: Data type accuracy, which indicates the proportion of data that conforms to the specified data type in the total data; Constraint information accuracy, which indicates the proportion of data that meets the specified constraint conditions in the total data; Validation rule accuracy, which indicates the percentage of data that meets the specified validation rules in the total data; Default value accuracy, which indicates the proportion of data that meets the default value in the total data; The data skewness coefficient indicates the bias of the data set characteristics.

5. The method according to claim 3, characterized in that: The subordinate indicators of the data integrity indicator include: Foreign key coverage rate, which indicates the proportion of data that meets the preset foreign key constraints in the total data; Trigger compliance rate, which indicates the proportion of trigger data that meets the preset requirements in the total data; Primary key constraint compliance rate, which indicates the percentage of data with unique primary key values ​​that meet the preset requirements in the total data; The number of unique constraints indicates the number of unique constraints that meet the preset requirements.

6. The method according to claim 3, characterized in that The subordinate indicators of the data accuracy indicator include: Content accuracy, used to indicate the accuracy of data content; Data repetition rate, used to indicate the data repetition value of each data column; Data uniqueness compliance rate, which is used to indicate the proportion of data containing duplicate values ​​in the total data; The dirty data occurrence rate is used to measure the proportion of data with data outliers in the total data.

7. The method according to claim 3, characterized in that The subordinate indicators of the data consistency indicator include: Data content consistency, used to indicate the consistency of data content in each data table of operating data; Data relationship consistency is used to indicate the consistency of data key values ​​in each data table of running data.

8. The method according to claim 3, characterized in that The subordinate indicators of the data timeliness indicator include: Time period accuracy, which is used to indicate the proportion of data that meets the specified time difference in the total data; Time point timeliness rate, which is used to indicate the proportion of data that meets the time point timeliness in the total data; The time series compliance rate is used to indicate the proportion of data that complies with time series in the total data.

9. The method according to claim 3, characterized in that: The subordinate indicators of the data accessibility indicator include: Accessibility, which is used to indicate how easy it is to obtain data; Availability is used to indicate the usability of data during its data life cycle.

10. The method according to claim 1, characterized in that The improved hybrid weighting model is used to weight each evaluation index, including: Construct a judgment matrix for each evaluation index based on the 1-9 scaling method; Determining the subjective weight of each indicator based on the judgment matrix; The objective weight of each indicator is determined based on the entropy weight method.

11. The method according to claim 10, characterized in that After completing the weighting of each evaluation indicator, the method further includes: Combining the subjective weight and the objective weight based on the game theory method to obtain the combined weight includes: An optimization problem is constructed with the goal of minimizing the sum of the combined weight, the subjective weight, and the objective weight; The linear combination coefficients in the optimization problem are adjusted step by step until the iteration termination condition is met, and the optimal combination weight is obtained, which is expressed as: Among them, W′ is the optimal combination weight; α′1 and α′2 are linear combination coefficients; W1 is the subjective weight; W2 is the objective weight.

12. The method according to claim 10, characterized in that Based on the evaluation values ​​of the final level indicators and the weighted evaluation indicators, the multi-level evaluation model is operated level by level to obtain the comprehensive evaluation value of the operation data of the corresponding hazardous chemical production enterprise, including: In the preset grey clustering evaluation model, based on the values ​​of each evaluation index and the preset value classification rules, the center point of each grey class is determined, and based on the center point of each grey class, the possibility function of each evaluation index and each grey class is obtained; Based on the possibility function, the grey fusion coefficient of each evaluation index with respect to each grey class is calculated respectively, which is expressed as: Among them, j is the jth evaluation index; k is the kth gray category; is the possibility function of evaluation index j and gray class k; is the weight of evaluation index j in gray class k; Based on the maximum value criterion, the gray class k corresponding to the object when the clustering coefficient is the largest is selected as the comprehensive evaluation value of the operating data of the corresponding hazardous chemical production enterprise.

13. A data quality evaluation system for hazardous chemical production, characterized in that: The system comprises: The collection unit is used to collect the operating data of the hazardous chemical production enterprise to be evaluated; A processing unit, used to traverse each final-level indicator in the pre-constructed multi-layer evaluation model, and calculate the evaluation value of each final-level indicator based on the pre-processed operation data; A weighting unit, used for weighting each evaluation index based on an improved hybrid weighting model to obtain each weighted evaluation index; The evaluation unit is used to perform step-by-step calculations on the multi-layer evaluation model based on the evaluation values ​​of each final-level indicator and each weighted evaluation indicator to obtain a comprehensive evaluation value of the operation data of the corresponding hazardous chemical production enterprise.

14. A computer-readable storage medium having instructions stored thereon, which, when executed on a computer, enables the computer to execute the hazardous chemical production data quality assessment method as described in any one of claims 1 to 12.

Images