Data rule design method and device for intelligent visual online test

Abstract

The invention relates to the technical field of big data management, and particularly provides a data rule design method and device for intelligent visual online testing, and the method comprises the following steps: S1, defining a data rule: defining the rule from three aspects of custom quality dimension, a basic rule base and a business rule base; s2, the JavaScript script is verified, and the JavaScript S3, performing matching by using a regular expression, and searching and replacing a complex text mode; s4, online testing and formatting are carried out, regular expressions or JS scripts are selected to carry out online testing and editing on data rule contents, meanwhile, the rule contents are displayed as verification or processing results of the data, in addition, the JS scripts carry out parameter analysis on rule methods, and attribute classification and filling are carried out on parameters; and S5, task scheduling and optimization. Compared with the prior art, the method can provide data meanings and business rule standards which are commonly obeyed by an upper business level.

Description

Technical Field

[0001] The present invention relates to the technical field of big data governance, and specifically provides a data rule design method and device for intelligent visual online testing. Background Art

[0002] In the digital age, big data, cloud computing, artificial intelligence, the Internet of Things, etc. have generated massive, heterogeneous data. The digital age has brought about efficient, convenient, and personalized ways of living and working. At the same time, the digital age has brought about a series of challenges and problems, such as data standards and data security. Data standards define the format and meaning of data fields, and regulate the entry, storage, and use of data, thereby reducing the possibility of data errors and confusion and improving the reliability and availability of data.

[0003] How to ensure data integrity, accuracy, consistency and timeliness by using one or more data rule methods is an urgent problem to be solved by technicians in this field. Summary of the invention

[0004] The present invention aims at the above-mentioned deficiencies of the prior art and provides a data rule design method for intelligent visual online testing with strong practicability.

[0005] A further technical task of the present invention is to provide a data rule design device for intelligent visual online testing that is reasonably designed, safe and applicable.

[0006] The technical solution adopted by the present invention to solve its technical problem is:

[0007] A data rule design method for intelligent visual online testing has the following steps:

[0008] S1. Define data rules from three aspects: custom quality dimension, basic rule base and business rule base;

[0009] S2, JavaScript script verification;

[0010] S3, use regular expressions to match, find, and replace complex text patterns;

[0011] S4, online testing and formatting, select regular expressions or JS scripts to test and edit the data rule content online, and display the rule content as the verification or processing result of the data. In addition, the JS script parses the parameters of the rule method, classifies and fills the parameters by attributes;

[0012] S5. Task scheduling and optimization.

[0013] Further, in step S1, the quality dimensions include accuracy, redundancy, completeness, relevance, uniqueness, standardization, consistency and timeliness;

[0014] The basic rule base has built-in SQL detection data rules, and SQL is implemented through a custom class (com.inspur.gcloud.dqm.common.rule.check.impl.DqmCheckBySql)

[0015] Implement SQL detection that adapts to multiple data sources, adapt and verify the SQL syntax of different data sources in the SQL detection executor, and give friendly prompts.

[0016] Furthermore, the business rule library is associated with the actual business scenario. The business rules associate the basic rules with the data source or table to be detected. The business rules are defined from the data rules and normative quality dimensions. The detection type is set to single table detection, MySQL data source and library, table and target value, and the library and table storing the data source are set.

[0017] The rule list includes basic validation rules, parameter name, parameter description, parameter type and parameter value.

[0018] Furthermore, in step S2, it further includes:

[0019] S2-1. Create an execution engine manager factory;

[0020] The open source javax.script.ScriptEngineManager interface instantiates the script execution engine manager factory;

[0021] S2-2. Create a JavaScript execution engine;

[0022] The execution engine manager factory.getEngineByName("JavaScript") method creates the JavaScript execution engine engine;

[0023] S2-3, JavaScript verification;

[0024] The JavaScript execution engine engine.eval(param) debugs and verifies the regular expression, where param is the regular expression content to be verified.

[0025] Furthermore, in step S3, it further includes:

[0026] S3-1. Create a regular expression using the RegExp constructor.

[0027] S3-2, extract with regular expression, use the match() method of regular expression to extract the part of the string that matches the regular expression;

[0028] S3-3, regular expression verification, through multiple tests and repeated debugging to meet business needs.

[0029] Furthermore, in step S4, it further includes:

[0030] S4-1, Data rule JavaScript script, regular expression online testing, formatting: Based on the open source technologies ScriptEngine and RegExp, the data rule JavaScript script and regular expression verification are implemented respectively;

[0031] S4-2. Metadata collection can adapt to various heterogeneous environments and support the collection of full metadata from traditional relational databases and big data platforms, including data entities in the process and the logic of data entity processing.

[0032] Furthermore, in step S5, task scheduling of modules corresponding to data integration development, data quality, data standards, and data modeling is currently provided, and regular expressions or JS script verification expressions already available in the platform are configured in a certain process link of data integration development.

[0033] A data rule design device for intelligent visual online testing, comprising: at least one memory and at least one processor;

[0034] The at least one memory is used to store a machine-readable program;

[0035] The at least one processor is used to call the machine-readable program to execute a data rule design method for intelligent visual online testing.

[0036] Compared with the prior art, the data rule design method and device for intelligent visual online testing of the present invention have the following outstanding beneficial effects:

[0037] The present invention uses mature Java Scripting API and Regex to perform JavaScript script and regular expression verification respectively, and provides an interactive interface for visual online JavaScript script and regular expression verification, providing a friendly interface experience. Similarly, the basic Regex regular expression and JavaScript script provided are associated with existing data quality, data standards, data integration development, etc. to perform data quality null value rate, dictionary distribution and other explorations, metadata collection, and data integration development part process stage verification, providing the upper business level with commonly observed data meanings and business rule standards. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Attached Figure 1 The invention is a flowchart of a data rule design method for intelligent visual online testing. DETAILED DESCRIPTION

[0040] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] A best embodiment is given below:

[0042] like Figure 1 As shown, a data rule design method for intelligent visual online testing in this embodiment has the following steps:

[0043] S1. Define data rules from three aspects: custom quality dimension, basic rule base and business rule base;

[0044] Further including:

[0045] Among them, the quality dimension describes the dimensions of quality detection, including accuracy, redundancy, completeness, relevance, uniqueness, standardization, consistency, and timeliness.

[0046] The basic rule base system has built-in SQL detection, duplication detection and other common data rules.

[0047] In this invention, SQL detection is taken as an example. The data source types supported by SQL detection are as follows: MySQL, Oracle, SQLserver, Greenplum, Postgresql, Hive, NewSQL, DaM eng, KingBase, HighGo, Oscar, ClickHouse and other data source SQL detection, through the custom class (com.inspur.gcloud.dqm.common.rule.check.impl.DqmCheckBySql)

[0048] Implement SQL detection that adapts to multiple data sources, and adapt and verify the SQL syntax of different data sources in the SQL detection executor to give friendly prompts. The present invention has built-in a large number of commonly used basic rule libraries. In principle, the basic rule library does not allow online modification and is maintained through background testing. Select the quality result based on the return result of the basic rule class (currently three quality results are provided: Boolean value, percentage value and text value).

[0049] The business rule library is associated with the actual business scenario, and the business rules associate the basic rules with the data source / table to be detected. The business rules can be maintained. In the present invention, the business rules are defined from the data rules and normative quality dimensions, and the detection type is set to single table detection, MySQL data source and library, table and target value, and the library and table storing the data source are set; the rule list includes basic verification rules (digital verification, ID card number detection, date verification), parameter name, parameter description, parameter type, parameter value (different values ​​are set according to the business), etc.

[0050] S2, JavaScript script verification;

[0051] Online testing of JavaScript scripts is very convenient, and you can easily write, run and debug JavaScript codes.

[0052] include:

[0053] S2-1. Create an execution engine manager factory:

[0054] The present invention instantiates a script execution engine manager factory through an open source javax.script.ScriptEngineManager interface.

[0055] S2-2. Create JavaScript execution engine:

[0056] Create a JavaScript execution engine engine through the execution engine manager factory.getEngineByName("JavaScript") method.

[0057] S2-3, JavaScript verification, debug and verify the regular expression through the JavaScript execution engine engine.eval(param), where param is the regular expression content to be verified.

[0058] S3, use regular expressions to match, find, and replace complex text patterns;

[0059] include:

[0060] S3-1. Regular expression creation:

[0061] Use the RegExp constructor to create a regular expression. For example: var re = new RegExp().

[0062] S3-2, regular expression extraction:

[0063] Use the match() method of a regular expression to extract the part of a string that matches the regular expression.

[0064] S3-3, regular expression verification;

[0065] Through multiple tests and repeated debugging, business needs can be quickly met, such as verifying whether a string is entirely Chinese characters.

[0066] S4, online testing and formatting, select regular expressions or JS scripts to test and edit the data rule content online, and display the rule content as the verification or processing result of the data. In addition, the JS script parses the parameters of the rule method, classifies and fills the parameters by attributes;

[0067] Regular expressions are verified online through the RegExp and match functions, and JavaScript scripts are verified online through the JavaScript execution engine ScriptEngine.

[0068] include:

[0069] S4-1, data rule JavaScript script, regular expression online test, formatting:

[0070] At present, the present invention implements JavaScript script and regular expression verification of data rules respectively according to the open source technology ScriptEngine (Java dynamically executes JavaScript script) and RegExp.

[0071] Take S4-2, data rule association metadata collection product as an example: metadata collection can adapt to various heterogeneous environments and support the collection of full metadata from traditional relational databases and big data platforms, including data entities in the process (descriptions of libraries, tables, and fields) and the logic of data entity processing.

[0072] The metadata collection task provides a variety of adapters for different data sources, realizing end-to-end automated collection. The data source layer covers the mainstream relational, non-relational, structured, and unstructured data sources on the market, such as mysql, sqlserver, oracle, postgresql, hive, es, hdfs, etc.

[0073] You can filter the table by whitelist or blacklist, and you can also formulate task execution strategies, manually trigger or automatically trigger, etc.

[0074] S5, task scheduling and optimization;

[0075] Configure relevant data rules and other configuration items in task management, and complete tasks by configuring scheduled tasks to run periodically or immediately. By running tasks multiple times, the accuracy, completeness, and consistency of data rules can be continuously corrected and optimized.

[0076] Currently, it provides task scheduling for modules corresponding to data integration development, data quality, data standards, and data modeling. In a certain process link of data integration development, the regular expression or JS script verification expression already in the platform is configured.

[0077] Similarly, data quality, data standards, and data modeling are associated with predicted regular expressions and JS script verification expressions. Run module tasks manually or schedule tasks to reduce cumbersome business processes. Taking the task management of data quality inspection as an example, create an inspection task to configure the inspection task name, inspection task description, task scheduling resources (either scheduling resources or task execution engine configuration), and provide scheduling modes: manual mode, polling mode, timing mode, daily mode, weekly mode, custom mode, etc., and whether the task is monitored; in task management, add inspection items to associate the configured data rules (regular expressions or JavaScript expressions).

[0078] Finally, the operation test report is viewed through the task instance. The test results provide relevant information such as test rules, quality results, quality dimensions, test results, quality scores, and whether the standards are met.

[0079] The definition of data rules includes providing data rule information such as data rule name, rule code, directory, rule type, implementation type, rule content, and rule description. The rule types include validation rules and conversion rules, and the implementation types include JavaScript scripts and Regex regular expressions.

[0080] Regular expressions use common characters and special characters (metacharacters) to match patterns in text. Such patterns can be used for operations such as text searching, replacing, and extracting.

[0081] Based on the above method, a data rule design device for intelligent visual online testing in this embodiment includes: at least one memory and at least one processor;

[0082] The at least one memory is used to store a machine-readable program;

[0083] The at least one processor is used to call the machine-readable program to execute a data rule design method for intelligent visual online testing.

[0084] The above-mentioned specific implementations are only specific cases of the present invention. The patent protection scope of the present invention includes but is not limited to the above-mentioned specific implementations. Any technical solutions that conform to the above-mentioned specific implementations of the present invention and any appropriate changes or substitutions made by ordinary technicians in the relevant technical field shall fall within the patent protection scope of the present invention.

[0085] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A data rule design method for intelligent visual online testing, characterized in that: The steps are as follows: S1. Define data rules from three aspects: custom quality dimension, basic rule base and business rule base; S2, JavaScript script verification; S3, use regular expressions to match, find, and replace complex text patterns; S4, online testing and formatting, select regular expressions or JS scripts to test and edit the data rule content online, and display the rule content as the verification or processing result of the data. In addition, the JS script parses the parameters of the rule method, classifies and fills the parameters by attributes; S5. Task scheduling and optimization.

2. The data rule design method for intelligent visual online testing according to claim 1 is characterized in that: In step S1, the quality dimensions include accuracy, redundancy, completeness, relevance, uniqueness, standardization, consistency and timeliness; The basic rule base has built-in SQL detection data rules, and SQL is implemented through a custom class (com.inspur.gcloud.dqm.common.rule.check.impl.DqmCheckBySql) Implement SQL detection that adapts to multiple data sources, adapt and verify the SQL syntax of different data sources in the SQL detection executor, and give friendly prompts.

3. The data rule design method for intelligent visual online testing according to claim 2 is characterized in that: The business rule library is associated with the actual business scenario. The business rules associate the basic rules with the data source or table to be detected. The business rules are defined from the data rules and normative quality dimensions. The detection type is set to single table detection, MySQL data source and library, table and target value, and the library and table storing the data source are set. The rule list includes basic validation rules, parameter name, parameter description, parameter type and parameter value.

4. The data rule design method for intelligent visual online testing according to claim 3 is characterized in that: In step S2, further comprising: S2-1. Create an execution engine manager factory; The open source javax.script.ScriptEngineManager interface instantiates the script execution engine manager factory; S2-2. Create a JavaScript execution engine; The execution engine manager factory.getEngineByName("JavaScript") method creates the JavaScript execution engine engine; S2-3, JavaScript verification; The JavaScript execution engine engine.eval(param) debugs and verifies the regular expression, where param is the regular expression content to be verified.

5. The data rule design method for intelligent visual online testing according to claim 4 is characterized in that: In step S3, further comprising: S3-1. Create a regular expression using the RegExp constructor. S3-2, extract with regular expression, use the match() method of regular expression to extract the part of the string that matches the regular expression; S3-3, regular expression verification, through multiple tests and repeated debugging to meet business needs.

6. The data rule design method for intelligent visual online testing according to claim 5 is characterized in that: In step S4, further comprising: S4-1, Data rule JavaScript script, regular expression online testing, formatting: Based on the open source technologies ScriptEngine and RegExp, the data rule JavaScript script and regular expression verification are implemented respectively; S4-2. Metadata collection can adapt to various heterogeneous environments and support the collection of full metadata from traditional relational databases and big data platforms, including data entities in the process and the logic of data entity processing.

7. The data rule design method for intelligent visual online testing according to claim 6 is characterized in that: In step S5, task scheduling of modules corresponding to data integration development, data quality, data standards, and data modeling is currently provided, and regular expressions or JS script verification expressions already available in the platform are configured in a certain process link of data integration development.

8. A data rule design device for intelligent visual online testing, characterized in that: include: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to call the machine-readable program to execute the method according to any one of claims 1 to 8.

Images