A data storage method for data quality issues based on relational database
By constructing new detection result data structures and bit operation calculation methods in relational databases, custom index sequences to organize and store problem data, solving the problems of low storage efficiency and large space in the existing technology, and achieving rapid Improved analytical and statistical efficiency.
Patent Information
- Application Number
- CN202411087926.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2044-08-09
AI Technical Summary
The existing problem data storage methods consume a lot of physical resources in relational databases, have low analysis and statistics efficiency, and are difficult to quickly investigate and classify data quality problems, and occupy a large space.
By building a new detection result data structure, using bit operation calculation methods, customizing fields, quality dimensions and rule index sequences, processing, sorting and storing problem data, using binary numerical values to represent violation information, supporting multiple rules to simultaneous filtering, reducing storage space and computational complexity.
It realizes rapid screening and statistics of the overall characteristics and key attributes of problem data, reduces storage space and memory consumption, and improves computing efficiency.
Smart Images

Figure CN118626495B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data processing, and in particular to a method for storing data with data quality problems based on a relational database. Background Art
[0002] In a business environment, the degree to which data meets the purpose of use of data consumers and can meet the specific needs of business scenarios is called data quality. In different business scenarios, data consumers have different requirements for data quality. Some people focus on the accuracy and consistency of data, while others focus on the timeliness and relevance of data. Therefore, as long as the data can meet the purpose of use, it can be said that the data quality meets the requirements.
[0003] Problem data refers to erroneous data generated during the data quality inspection process, which is usually described as inaccurate, incomplete, inconsistent or unreliable data. This data may contain errors, missing values, duplicates, inconsistent formats or outdated information.
[0004] The existing problem data storage method generally first organizes and stores the problem data in a unified manner, and then saves and records the problem data by saving the problem fields of the current record and the rule information violated by the problem fields. Using this method to store problem data consumes more physical resources, and the analysis and statistics of problem data are more complicated and the execution efficiency is low, making it difficult to effectively extract the characteristics of problem data.
[0005] In order to facilitate the rapid investigation of problem data in a relational database and the rapid statistical analysis of the classification of quality problems related to data quality, and to achieve the above requirements with a smaller space occupancy, the present invention proposes a data quality problem data storage method based on a relational database. Summary of the invention
[0006] In order to make up for the defects of the prior art, the present invention provides a simple and efficient data storage method for data quality problems based on a relational database.
[0007] The present invention is achieved through the following technical solutions:
[0008] A data quality problem data storage method based on a relational database improves the overall query and analysis efficiency of problem data by constructing a new test result data structure and using bitwise calculation methods;
[0009] The specific steps are as follows:
[0010] Step S1: Customize the fields to be detected in the detection data according to the requirements, and arrange them in a customized order to form a field index sequence;
[0011] Step S2: determine the quality dimensions involved in data quality detection, divide the quality dimensions into validity, consistency, uniqueness, timeliness and completeness, and arrange them in a custom fixed order to form a dimension index sequence;
[0012] Then, the quality rules involved in data quality detection are determined, and divided into corresponding data quality dimensions according to the characteristics of the quality rules. The quality rules are also arranged in a custom fixed order to form a rule index sequence;
[0013] Step S3: After the data quality test is completed and the problem data is obtained, the quality results are processed and sorted according to the quality dimension sequence defined in step S2, that is, the current problem data is assigned a placeholder value according to the set quality dimension sequence, and it is determined in turn whether the current problem data violates the detection standard of the quality dimension indexed in the corresponding sequence:
[0014] If the detection criteria of the current index dimension are violated, the current placeholder value is set to 1;
[0015] If the current index criteria are not violated, the current placeholder is set to 0;
[0016] After all the placeholder assignment operations are completed, a binary value is obtained based on the current problem data, and the binary value is converted to decimal and stored as the dimension violation information of the current detection result. The decimal value is the detection result of the quality dimension of the current problem record;
[0017] Step S4: After the quality dimension detection results are sorted, the quality results are processed and sorted according to the quality rule sequence defined in step S2, that is, the current problem data is assigned a placeholder value according to the set quality rule sequence, and it is determined in turn whether the current problem data violates the detection standard of the quality rule of the corresponding sequence index:
[0018] If the criteria for the current index are violated, the current placeholder is set to 1;
[0019] If the current index criteria are not violated, the current placeholder is set to 0;
[0020] After all the placeholder assignment operations are completed, a binary value can be sorted out according to the current problem data, and the binary value is converted into decimal and stored as the rule violation information of the current detection result. The decimal value is the detection result of the quality rule of the current problem record;
[0021] Step S5: After the quality rule detection results are sorted out, sort out the detection results of whether there are problems in the fields;
[0022] That is, placeholder values are assigned to the question fields in the current question data according to the field order set in step S1:
[0023] If there is a problem with the field at the current index position, the record is 1;
[0024] If there is no problem, the current placeholder is set to 0;
[0025] After all the placeholder assignment operations are completed, a binary value is sorted out according to the problem field contained in the current problem data, and the binary value is converted to decimal and stored as the field violation information of the current detection result. The decimal value is the field violation result of the current problem record;
[0026] After step S6 and step S3 to step S5 are sorted out, the problem fields in the problem data, i.e., the corresponding quality rule identifiers involved, are sorted out and stored;
[0027] In step S6, the storage rule is to store in groups according to a custom fixed order, and store the question field as a unit:
[0028] If there are multiple fields, they are separated and stored using separators, and the corresponding violated quality rules are stored as one unit;
[0029] If there are multiple quality rules, use separators to separate and store them;
[0030] The results are stored as detailed test result information of the problem data, where the problem fields correspond one-to-one with the quality rules;
[0031] Step S7, after the detailed detection result information is stored, by setting the calculation standard value of the quality rule and the quality standard value of the problem data, and performing bitwise calculation and evaluation with the problem data detection result of the quality rule and the field violation result of the problem record, query statistics of the quality rules, problem data and quality dimensions are realized.
[0032] In step S7, when analyzing the number of data in the statistical problem records that violate the uniqueness dimension judgment standard, firstly define the operation standard value of the problem records to be screened according to the dimension order defined in step S2;
[0033] If the predefined dimension order is uniqueness, validity, completeness, timeliness, and consistency, and you need to obtain data that violates the consistency dimension standard, define the calculation standard value as 00001, convert it into the corresponding decimal 1, and perform a bitwise AND operation on the data standard value and the dimension detection result of the current problem data:
[0034] If the test result is greater than 0, it means that the current problem data violates the current dimension detection standard;
[0035] If the result is equal to 0, it means that the current problem data does not violate the current dimension detection standard.
[0036] In step S7, the bitwise AND operation supports simultaneous screening of multiple detection rules. When it is necessary to simultaneously query the problem data that violates timeliness and consistency, the calculation standard value is defined as 00011, which is converted into the corresponding decimal value of 3. The bitwise AND operation is performed on the data standard value and the dimension detection result of the current problem data:
[0037] If the data obtained is the operation standard value itself, which is 3, it means that the current problem data violates the current dimension detection standard, that is, the standard of timeliness and consistency;
[0038] If it is not 3 but greater than 0, it means that both test criteria cannot be met at the same time;
[0039] If it is equal to 0, it means that neither of the two detection criteria is met.
[0040] The system for implementing the data quality problem data storage method based on a relational database includes a field index sequence module, a dimension index sequence module, a rule index sequence module, a quality dimension detection module, a quality rule detection module, a problem field detection module, a sorting and storage module and a query statistics module;
[0041] The field index sequence module is responsible for customizing the fields that need to be detected in the detection data according to the requirements, and arranging them in a custom order to form a field index sequence;
[0042] The dimension index sequence module is responsible for determining the quality dimensions involved in data quality detection, dividing the quality dimensions into validity, consistency, uniqueness, timeliness and completeness, and arranging them in a custom fixed order to form a dimension index sequence;
[0043] The rule index sequence module is responsible for determining the quality rules involved in data quality detection, and dividing them into corresponding data quality dimensions according to the characteristics of the quality rules. The quality rules are also arranged in a custom fixed order to form a rule index sequence.
[0044] The quality dimension detection module is responsible for processing and arranging the quality results according to the quality dimension sequence defined by the dimension index sequence module after the data quality detection is completed and the problem data is obtained, that is, the current problem data is assigned a placeholder value according to the set quality dimension sequence, and the current problem data is judged in turn whether it violates the detection standard of the quality dimension of the corresponding sequence index:
[0045] If the detection criteria of the current index dimension are violated, the current placeholder value is set to 1;
[0046] If the current index criteria are not violated, the current placeholder is set to 0;
[0047] After all the placeholder assignment operations are completed, a binary value is obtained based on the current problem data, and the binary value is converted to decimal and stored as the dimension violation information of the current detection result. The decimal value is the detection result of the quality dimension of the current problem record;
[0048] The quality rule detection module is responsible for processing and arranging the quality results according to the quality rule sequence defined by the rule index sequence module after the quality dimension detection results are sorted out. That is, according to the set quality rule sequence, the current problem data is assigned a placeholder value, and the current problem data is judged in turn whether it violates the detection standard of the quality rule of the corresponding sequence index:
[0049] If the criteria for the current index are violated, the current placeholder is set to 1;
[0050] If the current index criteria are not violated, the current placeholder is set to 0;
[0051] After all the placeholder assignment operations are completed, a binary value can be sorted out according to the current problem data, and the binary value is converted into decimal and stored as the rule violation information of the current detection result. The decimal value is the detection result of the quality rule of the current problem record;
[0052] The problem field detection module is responsible for sorting out the detection results of whether there are problems in the fields after the quality rule detection results are sorted out;
[0053] That is, according to the field order set by the field index sequence module, placeholder values are assigned to the problem fields in the current problem data:
[0054] If there is a problem with the field at the current index position, the record is 1;
[0055] If there is no problem, the current placeholder is set to 0;
[0056] After all the placeholder assignment operations are completed, a binary value is sorted out according to the problem field contained in the current problem data, and the binary value is converted to decimal and stored as the field violation information of the current detection result. The decimal value is the field violation result of the current problem record;
[0057] The sorting and storage module is responsible for sorting and storing the problem fields in the problem data, i.e., the corresponding quality rule identifiers involved;
[0058] In the sorting and storage module, the storage rule is to group and store in a custom fixed order, and store the question field as a unit:
[0059] If there are multiple fields, they are separated and stored using separators, and the corresponding violated quality rules are stored as one unit;
[0060] If there are multiple quality rules, use separators to separate and store them;
[0061] The results are stored as detailed test result information of the problem data, where the problem fields correspond one-to-one with the quality rules;
[0062] The query statistics module is responsible for setting the calculation standard values of quality rules and the quality standard values of problem data, and performing bitwise calculations and evaluations on the problem data detection results of the quality rules and the field violation results of the problem records, so as to realize query statistics on quality rules, problem data and quality dimensions.
[0063] When the query statistics module analyzes the number of data that violates the uniqueness dimension judgment standard in the statistical problem records, it first defines the operation standard value of the problem records to be screened according to the dimension order defined by the degree index sequence module;
[0064] If the predefined dimension order is uniqueness, validity, completeness, timeliness, and consistency, and you need to obtain data that violates the consistency dimension standard, define the calculation standard value as 00001, convert it into the corresponding decimal 1, and perform a bitwise AND operation on the data standard value and the dimension detection result of the current problem data:
[0065] If the test result is greater than 0, it means that the current problem data violates the current dimension detection standard;
[0066] If the result is equal to 0, it means that the current problem data does not violate the current dimension detection standard.
[0067] In the query statistics module, the bitwise AND operation supports simultaneous screening of multiple detection rules. When it is necessary to simultaneously query the problem data that violates timeliness and consistency, the calculation standard value is defined as 00011, which is converted into the corresponding decimal value of 3. The bitwise AND operation is performed on the data standard value and the dimension detection result of the current problem data:
[0068] If the data obtained is the operation standard value itself, which is 3, it means that the current problem data violates the current dimension detection standard, that is, the standard of timeliness and consistency;
[0069] If it is not 3 but greater than 0, it means that both test criteria cannot be met at the same time;
[0070] If it is equal to 0, it means that neither of the two detection criteria is met.
[0071] The beneficial effects of the present invention are: the data quality problem data storage method based on the relational database can quickly screen and count the overall characteristics and key attributes of the problem data, greatly improve the computing efficiency, reduce the space for storing problem data, and reduce the storage memory consumption by constructing a new storage structure. BRIEF DESCRIPTION OF THE DRAWINGS
[0072] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0073] Attached Figure 1 It is a schematic diagram of a data storage method for data quality problems based on a relational database according to the present invention.
[0074] Attached Figure 2 The figure is a schematic diagram of the calculation flow of the problem data result value of the present invention.
[0075] Attached Figure 3 This is a schematic diagram of the process of saving violation results of problem data fields of the present invention.
[0076] Attached Figure 4 This is a schematic diagram of the problem screening and statistical process of the present invention. DETAILED DESCRIPTION
[0077] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0078] This data quality problem data storage method based on a relational database improves the overall query and analysis efficiency of problem data by constructing a new test result data structure and using bit operation calculation methods;
[0079] As attached Figure 1 As shown, the specific steps are as follows:
[0080] Step S1: Customize the fields to be detected in the detection data according to the requirements, and arrange them in a customized order to form a field index sequence;
[0081] Step S2: determine the quality dimensions involved in data quality detection, divide the quality dimensions into validity, consistency, uniqueness, timeliness and completeness, and arrange them in a custom fixed order to form a dimension index sequence;
[0082] Then, the quality rules involved in data quality detection are determined, and divided into corresponding data quality dimensions according to the characteristics of the quality rules. The quality rules are also arranged in a custom fixed order to form a rule index sequence;
[0083] Step S3: After the data quality test is completed and the problem data is obtained, the quality results are processed and sorted according to the quality dimension sequence defined in step S2, that is, the current problem data is assigned a placeholder value according to the set quality dimension sequence, and it is determined in turn whether the current problem data violates the detection standard of the quality dimension indexed in the corresponding sequence:
[0084] If the detection criteria of the current index dimension are violated, the current placeholder value is set to 1;
[0085] If the current index criteria are not violated, the current placeholder is set to 0;
[0086] After all the placeholder assignment operations are completed, a binary value is obtained based on the current problem data, and the binary value is converted to decimal and stored as the dimension violation information of the current detection result. The decimal value is the detection result of the quality dimension of the current problem record;
[0087] Step S4: After the quality dimension detection results are sorted, the quality results are processed and sorted according to the quality rule sequence defined in step S2, that is, the current problem data is assigned a placeholder value according to the set quality rule sequence, and it is determined in turn whether the current problem data violates the detection standard of the quality rule of the corresponding sequence index:
[0088] If the criteria for the current index are violated, the current placeholder is set to 1;
[0089] If the current index criteria are not violated, the current placeholder is set to 0;
[0090] After all the placeholder assignment operations are completed, a binary value can be sorted out according to the current problem data, and the binary value is converted into decimal and stored as the rule violation information of the current detection result. The decimal value is the detection result of the quality rule of the current problem record;
[0091] Step S5: After the quality rule detection results are sorted out, sort out the detection results of whether there are problems in the fields;
[0092] That is, placeholder values are assigned to the question fields in the current question data according to the field order set in step S1:
[0093] If there is a problem with the field at the current index position, the record is 1;
[0094] If there is no problem, the current placeholder is set to 0;
[0095] After all the placeholder assignment operations are completed, a binary value is sorted out according to the problem field contained in the current problem data, and the binary value is converted to decimal and stored as the field violation information of the current detection result. The decimal value is the field violation result of the current problem record;
[0096] After the arrangement of step S6 and step S3 to step S5 is completed, the problem fields in the problem data, i.e., the corresponding quality rule identifiers involved, are arranged and stored, as shown in the attached figure. Figure 3 As shown;
[0097] In step S6, the storage rule is to store in groups according to a custom fixed order, and store the question field as a unit:
[0098] If there are multiple fields, they are separated and stored using separators, and the corresponding violated quality rules are stored as one unit;
[0099] If there are multiple quality rules, use separators to separate and store them;
[0100] The results are stored as detailed test result information of the problem data, where the problem fields correspond one-to-one with the quality rules;
[0101] If you want to view the quality rules that a problem record specifically violates, you can view the storage information in step S6, through which you can view the specific detection result information of the problem data, that is, the detailed detection result information.
[0102] Step S7, after the detailed detection result information is stored, by setting the calculation standard value of the quality rule and the quality standard value of the problem data, and performing bitwise calculation and evaluation with the problem data detection result of the quality rule and the field violation result of the problem record, query statistics of the quality rules, problem data and quality dimensions are realized.
[0103] As attached Figure 2 As shown, in step S7, when analyzing the number of data in the statistical problem records that violate the uniqueness dimension judgment standard, firstly define the operation standard value of the problem records to be screened according to the dimension order defined in step S2;
[0104] If the predefined dimension order is uniqueness, validity, completeness, timeliness, and consistency, and you need to obtain data that violates the consistency dimension standard, define the calculation standard value as 00001, convert it into the corresponding decimal 1, and perform a bitwise AND operation on the data standard value and the dimension detection result of the current problem data:
[0105] If the test result is greater than 0, it means that the current problem data violates the current dimension detection standard;
[0106] If the result is equal to 0, it means that the current problem data does not violate the current dimension detection standard.
[0107] As attached Figure 4 As shown, in step S7, the bitwise AND operation supports simultaneous screening of multiple detection rules. When it is necessary to simultaneously query the problem data that violates timeliness and consistency, the calculation standard value is defined as 00011, which is converted into the corresponding decimal value of 3, and the bitwise AND operation is performed on the data standard value and the dimension detection result of the current problem data:
[0108] If the data obtained is the operation standard value itself, which is 3, it means that the current problem data violates the current dimension detection standard, that is, the standard of timeliness and consistency;
[0109] If it is not 3 but greater than 0, it means that both test criteria cannot be met at the same time;
[0110] If it is equal to 0, it means that neither of the two detection criteria is met.
[0111] By flexibly setting different calculation standard values, we can conduct a comprehensive analysis and screening of the problem data, so as to find the main problem points of the problem, such as single filter type matching, multiple filter types arbitrary matching, multiple filter types full matching, etc.
[0112] The system for implementing the data quality problem data storage method based on a relational database includes a field index sequence module, a dimension index sequence module, a rule index sequence module, a quality dimension detection module, a quality rule detection module, a problem field detection module, a sorting and storage module and a query statistics module;
[0113] The field index sequence module is responsible for customizing the fields that need to be detected in the detection data according to the requirements, and arranging them in a custom order to form a field index sequence;
[0114] The dimension index sequence module is responsible for determining the quality dimensions involved in data quality detection, dividing the quality dimensions into validity, consistency, uniqueness, timeliness and completeness, and arranging them in a custom fixed order to form a dimension index sequence;
[0115] The rule index sequence module is responsible for determining the quality rules involved in data quality detection, and dividing them into corresponding data quality dimensions according to the characteristics of the quality rules. The quality rules are also arranged in a custom fixed order to form a rule index sequence.
[0116] The quality dimension detection module is responsible for processing and arranging the quality results according to the quality dimension sequence defined by the dimension index sequence module after the data quality detection is completed and the problem data is obtained, that is, the current problem data is assigned a placeholder value according to the set quality dimension sequence, and the current problem data is judged in turn whether it violates the detection standard of the quality dimension of the corresponding sequence index:
[0117] If the detection criteria of the current index dimension are violated, the current placeholder value is set to 1;
[0118] If the current index criteria are not violated, the current placeholder is set to 0;
[0119] After all the placeholder assignment operations are completed, a binary value is obtained based on the current problem data, and the binary value is converted to decimal and stored as the dimension violation information of the current detection result. The decimal value is the detection result of the quality dimension of the current problem record;
[0120] The quality rule detection module is responsible for processing and arranging the quality results according to the quality rule sequence defined by the rule index sequence module after the quality dimension detection results are sorted out. That is, according to the set quality rule sequence, the current problem data is assigned a placeholder value, and the current problem data is judged in turn whether it violates the detection standard of the quality rule of the corresponding sequence index:
[0121] If the criteria for the current index are violated, the current placeholder is set to 1;
[0122] If the current index criteria are not violated, the current placeholder is set to 0;
[0123] After all the placeholder assignment operations are completed, a binary value can be sorted out according to the current problem data, and the binary value is converted into decimal and stored as the rule violation information of the current detection result. The decimal value is the detection result of the quality rule of the current problem record;
[0124] The problem field detection module is responsible for sorting out the detection results of whether there are problems in the fields after the quality rule detection results are sorted out;
[0125] That is, according to the field order set by the field index sequence module, placeholder values are assigned to the problem fields in the current problem data:
[0126] If there is a problem with the field at the current index position, the record is 1;
[0127] If there is no problem, the current placeholder is set to 0;
[0128] After all the placeholder assignment operations are completed, a binary value is sorted out according to the problem field contained in the current problem data, and the binary value is converted to decimal and stored as the field violation information of the current detection result. The decimal value is the field violation result of the current problem record;
[0129] The sorting and storage module is responsible for sorting and storing the problem fields in the problem data, i.e., the corresponding quality rule identifiers involved;
[0130] In the sorting and storage module, the storage rule is to group and store in a custom fixed order, and store the question field as a unit:
[0131] If there are multiple fields, they are separated and stored using separators, and the corresponding violated quality rules are stored as one unit;
[0132] If there are multiple quality rules, use separators to separate and store them;
[0133] The results are stored as detailed test result information of the problem data, where the problem fields correspond one-to-one with the quality rules;
[0134] If you want to view the specific quality rules that a problem record violates, you can view the storage information of the sorting storage module. Through this information, you can view the specific detection result information of the problem data, that is, the detailed detection result information.
[0135] The query statistics module is responsible for setting the calculation standard values of quality rules and the quality standard values of problem data, and performing bitwise calculations and evaluations on the problem data detection results of the quality rules and the field violation results of the problem records, so as to realize query statistics on quality rules, problem data and quality dimensions.
[0136] When the query statistics module analyzes the number of data that violates the uniqueness dimension judgment standard in the statistical problem records, it first defines the operation standard value of the problem records to be screened according to the dimension order defined by the degree index sequence module;
[0137] If the predefined dimension order is uniqueness, validity, completeness, timeliness, and consistency, and you need to obtain data that violates the consistency dimension standard, define the calculation standard value as 00001, convert it into the corresponding decimal 1, and perform a bitwise AND operation on the data standard value and the dimension detection result of the current problem data:
[0138] If the test result is greater than 0, it means that the current problem data violates the current dimension detection standard;
[0139] If the result is equal to 0, it means that the current problem data does not violate the current dimension detection standard.
[0140] In the query statistics module, the bitwise AND operation supports simultaneous screening of multiple detection rules. When it is necessary to simultaneously query the problem data that violates timeliness and consistency, the calculation standard value is defined as 00011, which is converted into the corresponding decimal value of 3. The bitwise AND operation is performed on the data standard value and the dimension detection result of the current problem data:
[0141] If the data obtained is the operation standard value itself, which is 3, it means that the current problem data violates the current dimension detection standard, that is, the standard of timeliness and consistency;
[0142] If it is not 3 but greater than 0, it means that both test criteria cannot be met at the same time;
[0143] If it is equal to 0, it means that neither of the two detection criteria is met.
[0144] The above is a detailed introduction to a data storage method for data quality problems based on a relational database in an example of the present invention. This section uses specific examples to illustrate the principles and implementation methods of the invention. The above examples are only used to help understand the core ideas of the present invention. Without departing from the principles of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
Claims
1. A data storage method for data quality issues based on relational database, Features: By constructing a new test result data structure and using bitwise calculations, the efficiency of overall query and analysis of problem data is improved; The specific steps are as follows: Step S1: Customize the fields to be detected in the detection data according to the requirements, and arrange them in a customized order to form a field index sequence; Step S2: determine the quality dimensions involved in data quality detection, divide the quality dimensions into validity, consistency, uniqueness, timeliness and completeness, and arrange them in a custom fixed order to form a dimension index sequence; Then, the quality rules involved in data quality detection are determined, and divided into corresponding data quality dimensions according to the characteristics of the quality rules. The quality rules are also arranged in a custom fixed order to form a rule index sequence; Step S3: After the data quality test is completed and the problem data is obtained, the quality results are processed and sorted according to the quality dimension sequence defined in step S2, that is, the current problem data is assigned a placeholder value according to the set quality dimension sequence, and it is determined in turn whether the current problem data violates the detection standard of the quality dimension indexed in the corresponding sequence: If the detection criteria of the current index dimension are violated, the current placeholder value is set to 1; If the current index criteria are not violated, the current placeholder is set to 0; After all the placeholder assignment operations are completed, a binary value is obtained based on the current problem data, and the binary value is converted to decimal and stored as the dimension violation information of the current detection result. The decimal value is the detection result of the quality dimension of the current problem record; Step S4: After the quality dimension detection results are sorted, the quality results are processed and sorted according to the quality rule sequence defined in step S2, that is, the current problem data is assigned a placeholder value according to the set quality rule sequence, and it is determined in turn whether the current problem data violates the detection standard of the quality rule of the corresponding sequence index: If the criteria for the current index are violated, the current placeholder is set to 1; If the current index criteria are not violated, the current placeholder is set to 0; After all the placeholder assignment operations are completed, a binary value can be sorted out according to the current problem data, and the binary value is converted into decimal and stored as the rule violation information of the current detection result. The decimal value is the detection result of the quality rule of the current problem record; Step S5: After the quality rule detection results are sorted out, sort out the detection results of whether there are problems in the fields; That is, placeholder values are assigned to the question fields in the current question data according to the field order set in step S1: If there is a problem with the field at the current index position, the record is 1; If there is no problem, the current placeholder is set to 0; After all the placeholder assignment operations are completed, a binary value is sorted out according to the problem field contained in the current problem data, and the binary value is converted to decimal and stored as the field violation information of the current detection result. The decimal value is the field violation result of the current problem record; After step S6 and step S3 to step S5 are sorted out, the problem fields in the problem data, i.e., the corresponding quality rule identifiers involved, are sorted out and stored; Step S7, after the detailed detection result information is stored, by setting the calculation standard value of the quality rule and the quality standard value of the problem data, and performing bitwise calculation and evaluation with the problem data detection result of the quality rule and the field violation result of the problem record, query statistics of the quality rules, problem data and quality dimensions are realized.
2. The data storage method for data quality problems based on a relational database according to claim 1, Features: In step S6, the storage rule is to store in groups according to a custom fixed order, and store the question field as a unit: If there are multiple fields, they are separated and stored using separators, and the corresponding violated quality rules are stored as one unit; If there are multiple quality rules, use separators to separate and store them; The results are stored as detailed detection result information of the problem data, where the problem fields correspond one-to-one to the quality rules.
3. The data storage method for data quality problems based on a relational database according to claim 2, Features: In step S7, when analyzing the number of data in the statistical problem records that violate the uniqueness dimension judgment standard, firstly define the operation standard value of the problem records to be screened according to the dimension order defined in step S2; If the predefined dimension order is uniqueness, validity, completeness, timeliness, and consistency, and you need to obtain data that violates the consistency dimension standard, define the calculation standard value as 00001, convert it into the corresponding decimal 1, and perform a bitwise AND operation on the data standard value and the dimension detection result of the current problem data: If the test result is greater than 0, it means that the current problem data violates the current dimension detection standard; If the result is equal to 0, it means that the current problem data does not violate the current dimension detection standard.
4. The data storage method for data quality problems based on a relational database according to claim 3, Features: In step S7, the bitwise AND operation supports simultaneous screening of multiple detection rules. When it is necessary to simultaneously query the problem data that violates timeliness and consistency, the calculation standard value is defined as 00011, which is converted into the corresponding decimal value of 3. The bitwise AND operation is performed on the data standard value and the dimension detection result of the current problem data: If the data obtained is the operation standard value itself, which is 3, it means that the current problem data violates the current dimension detection standard, that is, the standard of timeliness and consistency; If it is not 3 but greater than 0, it means that both test criteria cannot be met at the same time; If it is equal to 0, it means that neither of the two detection criteria is met.
5. A system for implementing the data quality problem data storage method based on a relational database as described in any one of claims 1 to 4, Features: It includes field index sequence module, dimension index sequence module, rule index sequence module, quality dimension detection module, quality rule detection module, problem field detection module, sorting and storage module and query statistics module; The field index sequence module is responsible for customizing the fields that need to be detected in the detection data according to the requirements, and arranging them in a custom order to form a field index sequence; The dimension index sequence module is responsible for determining the quality dimensions involved in data quality detection, dividing the quality dimensions into validity, consistency, uniqueness, timeliness and completeness, and arranging them in a custom fixed order to form a dimension index sequence; The rule index sequence module is responsible for determining the quality rules involved in data quality detection, and dividing them into corresponding data quality dimensions according to the characteristics of the quality rules. The quality rules are also arranged in a custom fixed order to form a rule index sequence. The quality dimension detection module is responsible for processing and arranging the quality results according to the quality dimension sequence defined by the dimension index sequence module after the data quality detection is completed and the problem data is obtained, that is, the current problem data is assigned a placeholder value according to the set quality dimension sequence, and the current problem data is judged in turn whether it violates the detection standard of the quality dimension of the corresponding sequence index: If the detection criteria of the current index dimension are violated, the current placeholder value is set to 1; If the current index criteria are not violated, the current placeholder is set to 0; After all the placeholder assignment operations are completed, a binary value is obtained based on the current problem data, and the binary value is converted to decimal and stored as the dimension violation information of the current detection result. The decimal value is the detection result of the quality dimension of the current problem record; The quality rule detection module is responsible for processing and arranging the quality results according to the quality rule sequence defined by the rule index sequence module after the quality dimension detection results are sorted out. That is, according to the set quality rule sequence, the current problem data is assigned a placeholder value, and the current problem data is judged in turn whether it violates the detection standard of the quality rule of the corresponding sequence index: If the criteria for the current index are violated, the current placeholder is set to 1; If the current index criteria are not violated, the current placeholder is set to 0; After all the placeholder assignment operations are completed, a binary value can be sorted out according to the current problem data, and the binary value is converted into decimal and stored as the rule violation information of the current detection result. The decimal value is the detection result of the quality rule of the current problem record; The problem field detection module is responsible for sorting out the detection results of whether there are problems in the fields after the quality rule detection results are sorted out; That is, according to the field order set by the field index sequence module, placeholder values are assigned to the problem fields in the current problem data: If there is a problem with the field at the current index position, the record is 1; If there is no problem, the current placeholder is set to 0; After all the placeholder assignment operations are completed, a binary value is sorted out according to the problem field contained in the current problem data, and the binary value is converted to decimal and stored as the field violation information of the current detection result. The decimal value is the field violation result of the current problem record; The sorting and storage module is responsible for sorting and storing the problem fields in the problem data, i.e., the corresponding quality rule identifiers involved; The query statistics module is responsible for setting the calculation standard values of quality rules and the quality standard values of problem data, and performing bitwise calculations and evaluations on the problem data detection results of the quality rules and the field violation results of the problem records, so as to realize query statistics on quality rules, problem data and quality dimensions.
6. The system for implementing the data storage method for data quality problems based on a relational database according to claim 5, Features: In the sorting and storage module, the storage rule is to group and store in a custom fixed order, and store the question field as a unit: If there are multiple fields, they are separated and stored using separators, and the corresponding violated quality rules are stored as one unit; If there are multiple quality rules, use separators to separate and store them; The results are stored as detailed detection result information of the problem data, where the problem fields correspond one-to-one to the quality rules.
7. The system for implementing the data storage method for data quality problems based on a relational database according to claim 5, Features: When the query statistics module analyzes the number of data that violates the uniqueness dimension judgment standard in the statistical problem records, it first defines the operation standard value of the problem records to be screened according to the dimension order defined by the degree index sequence module; If the predefined dimension order is uniqueness, validity, completeness, timeliness, and consistency, and you need to obtain data that violates the consistency dimension standard, define the calculation standard value as 00001, convert it into the corresponding decimal 1, and perform a bitwise AND operation on the data standard value and the dimension detection result of the current problem data: If the test result is greater than 0, it means that the current problem data violates the current dimension detection standard; If the result is equal to 0, it means that the current problem data does not violate the current dimension detection standard.
8. The system for implementing the data storage method for data quality problems based on a relational database according to claim 5, Features: In the query statistics module, the bitwise AND operation supports simultaneous screening of multiple detection rules. When it is necessary to simultaneously query the problem data that violates timeliness and consistency, the calculation standard value is defined as 00011, which is converted into the corresponding decimal value of 3. The bitwise AND operation is performed on the data standard value and the dimension detection result of the current problem data: If the data obtained is the operation standard value itself, which is 3, it means that the current problem data violates the current dimension detection standard, that is, the standard of timeliness and consistency; If it is not 3 but greater than 0, it means that both test criteria cannot be met at the same time; If it is equal to 0, it means that neither of the two detection criteria is met.
