Autonomous preprocessing method for structured data based on multi-dimensional data standard

Abstract

The invention relates to an autonomous preprocessing method for structured data based on a multi-dimensional data standard, and belongs to the field of data governance. The method comprises the following steps: defining a multi-dimensional data standard; the cleaning conversion rule is quoted when a standard field in the multi-dimensional data standard is defined, and a cleaning conversion strategy of the standard field for problem data is determined; the data labeling rule is quoted when a standard field in the multi-dimensional data standard is defined, and a rule for autonomously marking the data based on the standard field is determined; defining a data processing task name, a scheduling mode and task description basic information; configuring a source data table and a detail data table; performing autonomous quality evaluation; automatic cleaning conversion; and performing autonomous data annotation. According to the method, the manual operation of configuring the data preprocessing process by the user is reduced, the threshold of data governance is lowered, and the efficiency of data governance is improved.

Description

Technical Field

[0001] This invention belongs to the field of data governance, specifically relating to an autonomous preprocessing method for structured data based on multidimensional data standards. Background Technology

[0002] With the rapid development of information technology and the in-depth advancement of the construction of Digital China, the innovative application of digital technology has become an important driving force for high-quality development in the political, economic, cultural, social, and ecological civilization fields. Data generated in the course of lawful performance of duties and operations in various fields is a fundamental resource for the construction of Digital China. However, due to the lack of data standards and constraints in the early stages of informatization construction in some fields, the quality of data varies greatly, resulting in the ineffective release of data value.

[0003] To improve data quality, data governance typically employs manually configured ETL processes for preprocessing operations such as null value and duplicate value removal. Data labeling is also primarily done manually by governance personnel. Furthermore, data standards, data cleaning and transformation rules, and data labeling rules are not effectively integrated during data governance. While manually configuring ETL processes can effectively address problematic data and improve data quality, it requires deep involvement from data governance personnel in the data preprocessing process. This invention proposes a multi-dimensional data standard definition method that effectively integrates traditional data standard definitions with data cleaning and transformation rules and data labeling rules. This allows a data processing engine to autonomously preprocess data based on multi-dimensional data standards. This method lowers the operational threshold for data governance personnel in data preprocessing and improves the efficiency of data preprocessing. Summary of the Invention

[0004] (a) Technical problems to be solved

[0005] The technical problem this invention aims to solve is how to provide an autonomous preprocessing method for structured data based on multidimensional data standards, in order to address the lack of definition of data cleaning and transformation rules and data labeling rule attributes in existing technologies. Currently, the preprocessing operation for structured data mainly relies on manual configuration of ETL data transformation processes, making it difficult to achieve preprocessing operations such as cleaning, transformation, and data labeling of structured data autonomously by the program.

[0006] (II) Technical Solution

[0007] To address the aforementioned technical problems, this invention proposes an autonomous preprocessing method for structured data based on multidimensional data standards. This method includes the following steps:

[0008] Step 1: Define multidimensional data standards: Based on the standard field definitions of the data standards, definitions of cleaning and transformation rules and data annotation rules are added;

[0009] Step 2: The cleaning and transformation rules are referenced when defining the standard fields in the multidimensional data standard to determine the cleaning and transformation strategy for the problem data.

[0010] Step 3: The data annotation rules are referenced when defining the standard fields in the multidimensional data standard to determine the rules for autonomously labeling data based on these standard fields.

[0011] Step 4: Define the data processing task name, scheduling method, and basic task description information. The scheduling method includes manual scheduling and automatic scheduling by the program according to the timing strategy.

[0012] Step 5: Configure the source data table and detailed data table. The source data table stores the original data being imported, while the detailed data table stores high-quality data that has been cleaned, transformed, and labeled to achieve a unified data standard. Configure the mapping relationship between the fields in the source data table and the fields in the detailed data table. The fields when creating the detailed data table are derived from the standard fields in the multidimensional data standard. The configuration uses a method of automatic mapping as the main approach and manual mapping as a supplement. The automatic mapping uses the cosine similarity algorithm to recommend the standard fields that are most similar to the fields in the source data table.

[0013] Step Six: Autonomous Quality Assessment: The data processing tasks successfully configured in Steps Four and Five support periodic automatic execution and manual execution. When executing the task, the data processing engine will conduct a quality assessment of the data in the source data table based on multidimensional data standards.

[0014] Step 7, Autonomous Cleaning and Transformation: Based on the quality assessment results of the original data in Step 6, the data preprocessing engine obtains the database, table, and field information of the data with quality problems, as well as the specific quality problems; and performs cleaning and transformation processing on the original data according to the cleaning and transformation rules defined in the multidimensional data standard fields corresponding to the fields containing the problem data.

[0015] Step 8, Autonomous Data Labeling: After Step 7 is completed, high-quality cleaned and transformed data will be generated. Based on the data labeling rules corresponding to the data standard fields, the data will be automatically labeled, and the labeled data will be persisted to the detailed database.

[0016] (III) Beneficial Effects

[0017] This invention proposes an autonomous preprocessing method for structured data based on multidimensional data standards. The main advantages of this invention are as follows:

[0018] a) The data standard integrates data cleaning and transformation rule attributes and data annotation rule attributes, expanding the definition of traditional data standards. The resulting multidimensional data standard will support the data preprocessing engine to autonomously perform preprocessing operations such as data cleaning, transformation, and data annotation.

[0019] (b) The data preprocessing engine autonomously assesses the quality of the raw data from the data source based on the standard data attributes defined in the multidimensional data standard. The quality assessment results include information on the database, table, and fields containing the problematic data, as well as quality issues such as null values ​​or duplicate data. Based on the quality assessment results, the problematic data is cleaned, transformed, and labeled according to the cleaning, transformation, and data labeling rules defined in the multidimensional data standard. This reduces the manual operation of configuring the data preprocessing workflow for users, allowing them to focus on their business operations.

[0020] c) Users do not need to worry about the methods of data quality assessment, cleaning and transformation, and manual annotation. They only need to focus on which data source to import the data into which detailed database. Data quality assessment, cleaning and transformation, and data annotation will be automatically implemented by the data preprocessing engine based on multidimensional data standards, which lowers the threshold of data governance and improves the efficiency of data governance. Attached Figure Description

[0021] Figure 1 This is a diagram of the architecture of the present invention;

[0022] Figure 2 This is the main flowchart for data processing;

[0023] Figure 3 A flowchart for multidimensional data standard management;

[0024] Figure 4 This is a flowchart of the data processing process;

[0025] Figure 5 This is a data processing flowchart of the present invention. Detailed Implementation

[0026] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0027] The "GB / T 44109-2024 Implementation Guidelines for Big Data Governance in Information Technology" only defines the business, technical, and management attributes of data standards, such as their Chinese and English names, business definitions, business rules, statistical dimensions, basis for formulation, data types, data length, data precision, standard-setting personnel, department to which the standard belongs, and department using the standard. It lacks definitions for data cleaning and transformation rules and data annotation rules. In the field of data governance, current preprocessing operations for structured data mainly rely on manual configuration of ETL data transformation processes, making it difficult to achieve preprocessing operations such as cleaning, transformation, and data annotation of structured data autonomously by a program.

[0028] To address the aforementioned technical challenges, this invention proposes a multidimensional data standard definition method that integrates data cleaning and transformation rules with annotation rule attributes. Based on this multidimensional data standard, a data processing engine autonomously performs quality assessment, cleaning and transformation, and data annotation on structured data. This enables an autonomous and efficient preprocessing capability for structured data, providing fundamental data support for user business operations, promoting data sharing, fully leveraging data value, and effectively supporting the urgent needs of digital transformation across various industries. This invention primarily includes multidimensional data standard definition, cleaning and transformation rule definition, data annotation rule definition, data preprocessing tasks, and a data preprocessing engine.

[0029] 1. Multidimensional Data Standard Definition: Based on the data standard definition dimensions in "GB / T 44109-2024 Information Technology Big Data Data Governance Implementation Guide", new definitions of data cleaning and transformation rules and data annotation rules have been added, which are the basis for independent data preprocessing.

[0030] 2. Definition of cleaning and transformation rules: Defines the data cleaning and transformation rules in the multidimensional data standard, which will support the program to perform preprocessing operations such as cleaning and transformation of data autonomously.

[0031] 3. Data annotation rule definition: Defines the data annotation rules in the multidimensional data standard, which will support the program to perform preprocessing operations to annotate the data autonomously.

[0032] 4. Data Preprocessing Task: Defines the data source information, detailed database information, and the mapping relationship between data source table fields and detailed database table fields. The data source provides the raw data, and the detailed database stores high-quality data after preprocessing such as cleaning, transformation, and data labeling. The data preprocessing task supports both manual and scheduled initiation.

[0033] 5. Data Preprocessing Engine: Using data from the data source as raw data, it performs operations such as quality assessment, cleaning and transformation, and data labeling on the raw data based on multidimensional data standards.

[0034] This invention relates to a method for defining a multidimensional data standard. Based on this standard, a program autonomously performs quality assessment, cleaning, transformation, and data annotation on structured data, aiming to address the need for efficient preprocessing of structured data. This method uses a relational database as the data source. Users only need to configure which data source to import data into which detailed database. The data processing engine will then perform preprocessing operations such as autonomous quality assessment, cleaning, transformation, and data annotation on the raw data from the data source based on the field names, Chinese field comments, field types, field lengths, non-null attributes, primary key attributes, cleaning and transformation rules, and data annotation rules defined in the multidimensional data standard. The preprocessed data will then be stored in the detailed database.

[0035] This invention expands the traditional definition of data standards by adding data cleaning and transformation rule attributes and data annotation rule attributes, forming a multi-dimensional data standard. This supports data processing engines in autonomously performing quality assessment, cleaning, transformation, and data annotation operations on structured data. This invention can be applied to various relational databases, including but not limited to DM, ShenTong, Kingbase, Oracle, MySQL, and PostgreSQL. The data processing flow involves a source database and a detailed database. The source database provides the raw data source; the detailed database's tables all have fields conforming to the data standard and are used to store high-quality data after cleaning, transformation, and data annotation. The data in the detailed database fully complies with the requirements of the data standard.

[0036] The main business process of this invention involves a data preprocessing engine evaluating the quality of the original data source based on the attribute definitions (field name, Chinese field comments, field type, field length, non-null attributes, primary key attributes, data dictionary, etc.) of multidimensional standard data. This evaluation generates a quality assessment result, which includes key information such as the database, table, and field information of the problematic data, as well as specific quality issues like null values ​​and duplicate values. The quality assessment result serves as input for the data processing engine to autonomously perform data cleaning and transformation. The data preprocessing engine then cleans, transforms, and annotates the data in the original database according to the cleaning and transformation rules and data annotation rules defined by the multidimensional standard data. The cleaned, transformed, and annotated data is then stored in a detailed database.

[0037] The data fields in the data source may be standard fields that conform to data standards, or they may be non-standard fields that do not conform to data standards. If the data source field is a standard field, the data preprocessing engine will directly perform quality assessment and data preprocessing operations on the raw data of the data source according to the definition of the field in the multidimensional data standard, such as field name, field type, primary key attribute, non-null attribute, data dictionary, cleaning and transformation rules, and data labeling rules. If the data source field is a non-standard field, the attribute definition of the standard field will be obtained according to the field mapping relationship between the data source table field and the detailed database table field (all data table fields in the detailed database are standard fields). When configuring the data source and detailed database, the mapping relationship between the data source table field and the detailed database table field is configured mainly by automatic program association and supplemented by manual association.

[0038] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0039] This invention aims to expand the definition of traditional data standards by adding definitions for cleaning and transformation rules and data annotation rules, forming a multi-dimensional data standard. This allows a data preprocessing engine to autonomously perform preprocessing operations such as quality assessment, cleaning and transformation, and data annotation on structured data based on this multi-dimensional data standard. This data preprocessing method can be completed autonomously by the data preprocessing engine, eliminating the need for manual configuration of ETL cleaning and transformation processes and manual data annotation, thus improving data preprocessing efficiency and providing reliable foundational data for user business development.

[0040] Multidimensional data standard management business processes, such as Figure 2 As shown:

[0041] Step 1: Define Multidimensional Data Standards. Building upon the standard field definitions of traditional data standards (such as field names, Chinese comments, field types, field lengths, NOT NULL attributes, primary key attributes, etc.), this step adds definitions for cleaning and transformation rules and data annotation rules. Cleaning and transformation rules can be defined using SQL statements or user-defined functions; data annotation rules can also be defined using SQL statements or user-defined functions. Upon successful rule definition, a unique rule ID will be generated.

[0042] When defining standard fields for multidimensional data standards, the program first checks whether the standard field exists based on the standard field name and Chinese annotation. If either of these has a duplicate value, the system will prompt "Standard field already exists". If neither the standard field name nor the Chinese annotation has a duplicate, the attribute information of the standard field needs to be filled in. If the standard field type is a dictionary, a data dictionary needs to be associated with the standard field. The cleaning and transformation rules and data annotation rules are associated with the multidimensional data standards through the rule ID.

[0043] Cleaning and conversion rules define the business process:

[0044] Step Two: The cleaning and transformation rules are referenced when defining standard fields in the multidimensional data standard to determine the cleaning and transformation strategy for the problematic data. When defining cleaning and transformation rules, the system first checks if the rule exists based on its name. If the rule already exists, the system will prompt "Rule already exists"; if the rule does not exist, the system needs to fill in the cleaning and transformation rules for the problematic data, rule description, and other attribute information. The cleaning and transformation rules can be defined by SQL statements or user-defined functions. If the program verifies the input rule attribute information, the attribute information of the cleaning and transformation rule is persisted, and a unique rule ID is generated, completing the rule definition; if the cleaning and transformation rule attribute information fails verification, the attributes of the cleaning and transformation rule need to be redefined until the defined attribute information passes verification.

[0045] The cleaning and transformation rules include, but are not limited to: (1) removing duplicate values; (2) retaining duplicate values; (3) removing null values; (4) filling with fixed values; (5) filling null values ​​with the maximum value of a random array; (6) filling null values ​​with the minimum value of a random array; (7) filling null values ​​with the average value of a random array; and (8) filling null values ​​with the median value of a random array. Rules (5), (6), (7), and (8) apply only to numeric data filling, and the random arrays are generated as follows:

[0046]

[0047] Data set D represents all data records in the column containing null values; Indicates from dataset Random selection A set formed by data points. If the set... The number of data entries is less than Then the set With sets The same, will Let it be a set The number of data entries.

[0048] random array The formula for finding the maximum value in the middle is as follows:

[0049]

[0050] random array The formula for finding the minimum value is as follows:

[0051]

[0052] random array The formula for taking the average of the values ​​is as follows:

[0053]

[0054] random array The formula for taking the median is as follows:

[0055] , odd number

[0056] This represents the set d after sorting in order. Data points.

[0057] , Even number

[0058] This represents the set d after sorting in order. Data points.

[0059] This represents the set d after sorting in order. Data points.

[0060] Labeling rules define the business process:

[0061] Step 3: Data labeling rules are referenced when defining standard fields in the multidimensional data standard. This determines the rules for the program to autonomously label data based on these standard fields. For example, for the "Age" field, ages less than or equal to 6 years old are labeled "Infant," and ages greater than or equal to 7 years old but less than or equal to 17 years old are labeled "Teenage," etc. This invention provides a tool for defining labeling rules. When defining data labeling rules, the program first checks whether the labeling rule exists based on the input rule name. If the rule already exists, the system will prompt "Rule already exists." If the rule does not exist, the labeling rule logic, rule description, and other attribute information need to be filled in. The logic of the labeling rule can be expressed by SQL statements or by user-defined functions. If the program verifies the input rule attribute information, it persists the attribute information of the labeling rule, and the labeling rule definition is complete. If the rule attribute information verification fails, the attributes of the labeling rule need to be redefined until the defined attribute information passes verification.

[0062] Configure the data preprocessing task flow as follows: Figure 4 As shown:

[0063] Step 4: Define basic information such as data processing task name, scheduling method, and task description. The scheduling method includes manual scheduling and automatic scheduling by the program according to the timing strategy.

[0064] Step 5: Configure the source data table and detailed data table. The source data table stores the original data being imported; data standards may vary, and data quality may differ. The detailed data table stores high-quality data that has undergone cleaning, transformation, and data labeling, resulting in standardized data. Configure the mapping relationship between the fields in the source data table and the fields in the detailed data table. The fields created in the detailed data table are derived from standard fields in the multidimensional data standard, so the fields in the detailed table are standard fields that meet the standard requirements. The configuration primarily uses automatic mapping, supplemented by manual mapping. Automatic mapping uses a cosine similarity algorithm to recommend the standard field most similar to the field in the source data table. If the source data table field has Chinese comments, the similarity between the Chinese comments of the source data table field and the Chinese comments of the detailed data table field is calculated; if the source data table field does not have Chinese comments, the similarity between the field name of the source data table and the field name of the detailed data table is calculated. The program will map the standard field with the highest similarity to the field in the source data table. The mapping relationship supports manual correction. The similarity algorithm is shown below:

[0065]

[0066] Where A represents the word frequency vector of the Chinese comments or field names of the source data table fields, and B represents the word frequency vector of the Chinese comments or field names of the standard fields. Represents the first in vector A Numbers, Represents the first element in vector B. There are two numbers, n, where n represents the dimension of vectors A and B.

[0067] The data processing task execution flow is as follows: Figure 5 As shown:

[0068] Data processing engine:

[0069] Step Six: Autonomous Quality Assessment. Data processing tasks successfully configured in Steps Four and Five support periodic automatic execution and manual execution. During task execution, the data processing engine will perform a quality assessment of the source data table based on multidimensional data standards. If the source data table fields are standard fields, the data processing engine can assess the uniqueness, completeness, and validity of the data based on the field types, primary key attributes, NOT NULL attributes, field lengths, and data dictionary (dictionary information is only associated when the field type is a dictionary). If the imported data uses non-standard fields, since Step Three has already mapped the source data table fields to the standard-compliant detail data table fields, the data processing engine can obtain the required field types, primary key attributes, NOT NULL attributes, field lengths, and data dictionary from the detail data table fields to perform a quality assessment of the source data table.

[0070] Key information from the quality assessment results includes details about the databases, tables, and fields where quality issues occurred, as well as the specific issues themselves, such as null values, duplicate values, or dictionary data not conforming to dictionary standards. This key information, including the databases, tables, fields, and quality issues, will serve as input for data cleaning and transformation, supporting the data processing engine in autonomously cleaning and transforming the raw data from the source tables.

[0071] Step Seven: Autonomous Cleaning and Transformation. Based on the quality assessment results of the raw data in Step Six, the data preprocessing engine can obtain information about the databases, tables, and fields where quality issues exist, as well as the specific quality problems. The engine then cleans and transforms the raw data according to the cleaning and transformation rules defined in the multidimensional data standard fields corresponding to the fields containing the problematic data.

[0072] Step 8: Autonomous Data Labeling. After Step 7 is completed, high-quality cleaned and transformed data will be generated. Based on the data labeling rules corresponding to the data standard fields, the data can be automatically labeled, and the labeled data will be persisted to the detailed database.

[0073] In the "GB / T 44109-2024 Information Technology Big Data Data Governance Implementation Guide" and current engineering practices in the field of data governance, data standard management, data quality management, data dictionary, and data annotation are independent of each other. Preprocessing operations such as cleaning, transformation, and data annotation are primarily based on these independent data standards, data dictionaries, cleaning and transformation rules, and annotation rules, relying on manual configuration of the data preprocessing workflow. This invention proposes a multi-dimensional data standard definition method that integrates data cleaning and transformation rule attributes and data annotation rule attributes. Based on this multi-dimensional data standard, it proposes a method for preprocessing structured data autonomously by a data preprocessing engine. The main advantages are as follows:

[0074] a) The data standard integrates data cleaning and transformation rule attributes and data annotation rule attributes, expanding the definition of traditional data standards. The resulting multidimensional data standard will support the data preprocessing engine to autonomously perform preprocessing operations such as data cleaning, transformation, and data annotation.

[0075] (b) The data preprocessing engine autonomously assesses the quality of the raw data from the data source based on the standard data attributes defined in the multidimensional data standard. The quality assessment results include information on the database, table, and fields containing the problematic data, as well as quality issues such as null values ​​or duplicate data. Based on the quality assessment results, the problematic data is cleaned, transformed, and labeled according to the cleaning, transformation, and data labeling rules defined in the multidimensional data standard. This reduces the manual operation of configuring the data preprocessing workflow for users, allowing them to focus on their business operations.

[0076] c) Users do not need to worry about the methods of data quality assessment, cleaning and transformation, and manual annotation. They only need to focus on which data source to import the data into which detailed database. Data quality assessment, cleaning and transformation, and data annotation will be automatically implemented by the data preprocessing engine based on multidimensional data standards, which lowers the threshold of data governance and improves the efficiency of data governance.

[0077] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A self-preprocessing method for structured data based on multidimensional data standards, characterized in that, The method includes the following steps: Step 1: Define multidimensional data standards: Based on the standard field definitions of the data standards, definitions of cleaning and transformation rules and data annotation rules are added; Step 2: The cleaning and transformation rules are referenced when defining the standard fields in the multidimensional data standard to determine the cleaning and transformation strategy for the problem data. Step 3: The data annotation rules are referenced when defining the standard fields in the multidimensional data standard to determine the rules for autonomously labeling data based on these standard fields. Step 4: Define the data processing task name, scheduling method, and basic task description information. The scheduling method includes manual scheduling and automatic scheduling by the program according to the timing strategy. Step 5: Configure the source data table and detailed data table. The source data table stores the original data being imported, while the detailed data table stores high-quality data that has been cleaned, transformed, and labeled to achieve a unified data standard. Configure the mapping relationship between the fields in the source data table and the fields in the detailed data table. The fields when creating the detailed data table are derived from the standard fields in the multidimensional data standard. The configuration uses a method of automatic mapping as the main approach and manual mapping as a supplement. The automatic mapping uses the cosine similarity algorithm to recommend the standard fields that are most similar to the fields in the source data table. Step Six: Autonomous Quality Assessment: The data processing tasks successfully configured in Steps Four and Five support periodic automatic execution and manual execution. When executing the task, the data processing engine will conduct a quality assessment of the data in the source data table based on multidimensional data standards. Step 7, Autonomous Cleaning and Transformation: Based on the quality assessment results of the original data in Step 6, the data preprocessing engine obtains the database, table, and field information of the data with quality problems, as well as the specific quality problems; and performs cleaning and transformation processing on the original data according to the cleaning and transformation rules defined in the multidimensional data standard fields corresponding to the fields containing the problem data. Step 8, Autonomous Data Labeling: After Step 7 is completed, high-quality cleaned and transformed data will be generated. Based on the data labeling rules corresponding to the data standard fields, the data will be automatically labeled, and the labeled data will be persisted to the detailed database.

2. The autonomous preprocessing method for structured data based on multidimensional data standards as described in claim 1, characterized in that, In step one, the standard field definitions for data standards include: field name, Chinese comments, field type, field length, non-null attribute, and primary key attribute; the cleansing and transformation rules are defined using SQL statements or user-defined functions; the data annotation rules are defined using SQL statements or user-defined functions, and a unique rule ID will be generated after the rule is successfully defined.

3. The autonomous preprocessing method for structured data based on multidimensional data standards as described in claim 2, characterized in that, When defining the standard fields of the multidimensional data standard in step one, first determine whether the standard field exists based on the standard field name and Chinese annotation. If either of them has a duplicate value, the system will prompt "the standard field already exists". If neither the standard field name nor the Chinese annotation is duplicated, the attribute information of the standard field needs to be filled in. If the standard field type is a dictionary, a data dictionary needs to be associated with the standard field. The cleaning and transformation rules and data annotation rules are associated with the multidimensional data standard through the rule ID.

4. The autonomous preprocessing method for structured data based on multidimensional data standards as described in claim 1, characterized in that, In step two, when defining the cleaning and transformation rules, the system first checks whether the rule exists based on its name. If the rule already exists, the system will prompt "The rule already exists." If the rule does not exist, the system needs to fill in the cleaning and transformation rules for the problem data and the rule description attribute information. If the program verifies the entered rule attribute information, the system will persist the attribute information of the cleaning and transformation rule and generate a unique rule ID, thus completing the rule definition. If the cleaning and transformation rule attribute information verification fails, the system needs to redefine the attributes of the cleaning and transformation rule until the defined attribute information passes the verification.

5. The autonomous preprocessing method for structured data based on multidimensional data standards as described in claim 4, characterized in that, The cleaning and transformation rules include, but are not limited to: (1) removing duplicate values; (2) retaining duplicate values; (3) removing null values; (4) filling with fixed values; (5) filling null values ​​with the maximum value of a random array; (6) filling null values ​​with the minimum value of a random array; (7) filling null values ​​with the average value of a random array; (8) filling null values ​​with the median value of a random array; wherein, rules (5), (6), (7), and (8) are only for filling numeric data.

6. The autonomous preprocessing method for structured data based on multidimensional data standards as described in claim 5, characterized in that, The random array is generated in the following way: Data set D represents all data records in the column containing null values; Indicates from dataset Random selection A set of data points. If the set... The number of data entries is less than Then the set With sets The same, will Let it be a set The number of data entries.

7. The autonomous preprocessing method for structured data based on multidimensional data standards as described in claim 6, characterized in that, random array The formula for finding the maximum value in the middle is as follows: random array The formula for finding the minimum value is as follows: random array The formula for taking the average of the values ​​is as follows: random array The formula for taking the median is as follows: , odd number in, This represents the set d after sorting in order. One data point; , Even number This represents the set d after sorting in order. Data This represents the set d after sorting in order. Data.

8. The autonomous preprocessing method for structured data based on multidimensional data standards as described in claim 1, characterized in that, Step three includes: When defining data annotation rules, first determine whether the annotation rule exists based on the input rule name. If the rule already exists, the system will prompt "The rule already exists"; if the rule does not exist, the annotation rule logic and rule description attribute information need to be filled in; if the program verifies the input rule attribute information, the attribute information of the annotation rule is persisted, and the annotation rule definition is completed; if the rule attribute information verification fails, the attributes of the data annotation rule need to be redefined until the defined attribute information passes the verification.

9. The autonomous preprocessing method for structured data based on multidimensional data standards as described in claim 1, characterized in that, In step five, If the source data table field has Chinese comments, calculate the similarity between the Chinese comments of the source data table field and the Chinese comments of the detail data table field; if the source data table field does not have Chinese comments, calculate the similarity between the field name of the source data table and the field name of the detail data table; map and associate the standard field with the highest similarity to the source data table field, and the mapping relationship supports manual correction; the similarity algorithm is as follows: Where A represents the word frequency vector of the Chinese comments or field names of the source data table fields, and B represents the word frequency vector of the Chinese comments or field names of the standard fields. Represents the first in vector A Numbers, Represents the first element in vector B. There are two numbers, n, where n represents the dimension of vectors A and B.

10. The autonomous preprocessing method for structured data based on multidimensional data standards as described in claim 1, characterized in that, In step six, if the source data table field is a standard field, the data processing engine evaluates the uniqueness, completeness, and validity of the data in the source data table based on the field type, primary key attribute, non-null attribute, field length, and data dictionary defined in the standard field. If the imported data is a non-standard field, since step three has already mapped the source data table field to the detailed data table field that meets the standard requirements, the data processing engine obtains the field type, primary key attribute, non-null attribute, field length, and data dictionary required by the standard field based on the detailed data table field to perform a quality assessment of the source data table.

Images