Data verification rule configuration method and system and data distribution system

Abstract

The invention belongs to the technical field of data verification, and particularly relates to a data verification rule configuration method and system and a data distribution system. The configuration method comprises the following steps: S1, a programmer uses a unified code writing logic in advance to build a running environment of a configuration module; s2, business personnel convert the business demand into logical operation among a plurality of basic demands; and S3, the business personnel in the configuration module reserves the logical operation among the basic demands, converts each basic demand into the logical operation among the plurality of basic verification units, and completes verification rule configuration for the current business demand in the configuration module. According to the data verification method and device, business personnel can conveniently and independently complete configuration and maintenance of the verification rules, the data verification efficiency and verification effect are improved, and meanwhile the stability of data verification is guaranteed.

Description

Technical Field

[0001] This application belongs to the field of data verification technology, and in particular relates to a data verification rule configuration method, configuration system and data diversion system. Background Technology

[0002] In current enterprise-level big data governance practices, data verification before output is a crucial step in ensuring data quality. Data verification enables data segmentation, and subsequent targeted processing of this segmented data is essential to improving the effectiveness of enterprise-level big data governance. For example, data verification can separate data with an engineering cycle exceeding 10 years from data less than 10 years old.

[0003] For mainstream data verification methods, programmers configure verification rules by writing programming languages ​​based on business needs to achieve data verification.

[0004] However, in existing technologies, programmers are typically required to define validation logic by writing SQL statements, which presents the following problems: ① Once a business requirement is divided and handled by multiple programmers; or due to adjustments in business requirements, other programmers need to add, delete, or overwrite existing SQL statements; then, the different coding logic of different programmers may lead to inconsistent data validation efficiency, unstable validation results, or even worse results.

[0005] ② When programmers don't truly understand the business needs of business users, or when business users discover that validation rules need immediate adjustment during use, the limitations of writing SQL statements prevent them from promptly and independently modifying the validation rules without the help of programmers. This results in low efficiency in maintaining validation rules, particularly hindering business users from independently configuring and maintaining them.

[0006] This application content

[0007] The purpose of this application is to overcome the shortcomings of the prior art and provide a data verification rule configuration method, which makes it easier for business personnel to independently complete the configuration and maintenance of verification rules, improve the efficiency and effectiveness of data verification, and ensure the stability of data verification.

[0008] To achieve the above objectives, this application adopts the following technical solution: A method for configuring data validation rules includes the following steps: S1, programmers pre-build the runtime environment of the configuration module using a unified code writing logic; S2, business personnel transform business requirements into logical operations between several basic requirements; S3 allows business personnel to retain the logical operations between basic requirements within the configuration module, while transforming each basic requirement into logical operations between several basic verification units, and then complete the configuration of verification rules for the current business requirement within the configuration module.

[0009] Preferably, in S3: the basic verification unit is a triplet of {first element, verification relation, second element}, and the verification relation refers to the logical relationship between the first element and the second element in the current basic verification unit; The test relation falls into one of the following categories: Text equals: Used to determine if two strings are the same; Text is not equal to: Used to determine whether two strings are different; Greater than >, less than <, equal to =, greater than or equal to ≥, less than or equal to ≤: used for numerical comparison; Earlier than: Used to determine whether a certain date is earlier than another date; "Date later than": Used to determine whether a given date is later than another date; Contains: Used to determine whether a string contains a specified substring. "Does not contain": Used to determine whether a string does not contain a specified substring; Matching: Forward matching based on regular expressions; Mismatch: Negation matching based on regular expressions.

[0010] Preferably, after S3, the following content is also included: For the completed verification rule, the first element in each basic verification unit of the verification rule is replaced with a placeholder and recorded as the verification rule template. If the current verification rule template is not stored, the current verification rule template is stored; otherwise, the current verification rule template is discarded.

[0011] Preferably, after S1 and before S2, it also includes S1': S1': After the business personnel determine the keywords based on the business requirements, they use the keywords to search for the validation rule template. If a corresponding validation rule template is found, the content in the placeholders in the validation rule template is replaced with the corresponding business requirement fields in sequence to complete the validation rule configuration for the current business requirement. If no corresponding validation rule template is found, then S2 is executed.

[0012] Preferably, when new business needs arise, return to S1'.

[0013] This application also provides a data validation rule configuration system, including: a runtime environment module, a decomposition module, and a configuration module; the runtime environment module is used to build the runtime environment of the configuration module through unified code writing logic; the decomposition module is used to convert business requirements into logical operations between several basic requirements and then send them to the configuration module; the configuration module is used to retain the logical operations between basic requirements while converting each basic requirement into logical operations between several basic validation units; each module is programmed or configured to execute the steps of the data validation rule configuration method described above.

[0014] This application also provides a data offloading method, including the following steps: Step 1: Configure data validation rules according to business requirements; configure data validation rules using one of the methods described above. Step 2: Intercept and distribute the input data using the configured data validation rules.

[0015] This application also provides a data splitting system, including: a configuration unit and a splitting unit. The configuration unit is used to configure data verification rules according to business requirements; the splitting unit is used to execute the configured data verification rules to intercept and split the input data; each module is programmed or configured to execute the steps of a data splitting method as described above.

[0016] This application also provides a computer-readable storage medium, characterized in that: the computer-readable storage medium stores a computer program that is programmed or configured to execute a data verification rule configuration method as described above.

[0017] This application also provides a computer program product, including a computer program / instruction, characterized in that: when the computer program / instruction is executed by a processor, it implements the steps of the data verification rule configuration method described above.

[0018] The beneficial effects of this application are as follows: (1) The data verification rule configuration method of this application makes it easier for business personnel to independently complete the configuration and maintenance of verification rules, improve the efficiency and effect of data verification, and ensure the stability of data verification.

[0019] (2) In the data verification rule configuration method of this application, programmers only need to build the runtime environment of the configuration module in advance using a unified code writing logic, which improves the consistency of the code writing logic and lays the foundation for business personnel to configure and maintain the verification rules in the future. The later configuration of verification rules for the changing business needs is carried out by business personnel who have a deep understanding of the business needs; this directly avoids the situation where business needs are divided among different programmers or different programmers adjust the business needs code at different times. Naturally, it also avoids the uneven data verification efficiency, unstable verification effect, or even worse effect caused by different programmers' different code writing logic.

[0020] (3) In the data verification rule configuration method of this application, business personnel do not need to perform add, delete and modify operations based on past business requirement code (which is beyond the ability of most business personnel). Instead, they only need to have the ability to logically decompose business requirements to quickly get started, efficiently and independently complete the configuration and maintenance of verification rules. Business personnel do not need to have coding ability, which is especially suitable for configuring verification rules for sudden business requirements, and also reduces the implementation and maintenance costs of business requirements; completely eliminates the technical dependence on programmers, greatly shortens the verification rule configuration cycle (from several days to minutes), and improves the agility of data governance response.

[0021] (4) In the data verification rule configuration method of this application, business personnel first decompose the business requirements into logical operations between several basic requirements, and then further convert each basic requirement into logical operations between several basic verification units. That is to say, for business personnel, the smallest unit to implement the complex verification rule configuration of business requirements is the basic verification unit, and the basic verification unit is a triple of {first element, verification relation, second element}, which has strong readability. Therefore, this application not only makes it easy for business personnel to configure verification rules, but also ensures that the verification effect after configuration is stable and high. Even if other business personnel modify the previous verification rules, there is no situation where they cannot understand the previous verification rules, reducing communication losses.

[0022] (5) In the data verification rule configuration method of this application, in the early stage, business personnel rely on the specific verification rules of various business needs to be constructed step by step. However, as the specific verification rules are converted into verification rule templates and stored, business personnel can gradually obtain specific verification rules by directly calling the relevant verification rule templates through keywords according to business needs (this step is also based on the strong readability of the basic verification unit). Therefore, as time goes by, the efficiency of configuration and maintenance of verification rules will become higher and higher. It is especially suitable for enterprise-level scenarios with massive table structures that are similar, realizing "one-time configuration, hundreds of effects", and promoting the large-scale implementation of data quality management.

[0023] (6) Whether this application configures specific verification rules by having business personnel construct the logical operations between basic verification units step by step, or by calling the verification rule template and replacing the placeholders with specific fields, the code behind the specific verification rules exists in blocks (such as the code corresponding to the basic verification unit and the code corresponding to the verification rule template) compared to existing technologies. Therefore, the consumption of computing resources is greatly reduced, making the subsequent data diversion application more efficient. Furthermore, whether it is the basic verification unit or the verification rule template, business personnel can only adjust the logical decomposition between them, not the code behind them. However, the logical decomposition of the same business requirement by different business personnel is basically the same. Therefore, this also avoids the situation where the verification efficiency and verification effect are unstable due to different business personnel configuring the verification rules.

[0024] (7) The data verification rule configuration method of this application enables business personnel to understand and construct verification rules in a way that conforms to business thinking; at the same time, it lays a structural foundation for subsequent rule templates and cross-scenario reuse of verification rules, and significantly improves the expressive power and logical clarity of verification rules.

[0025] (8) A data verification rule configuration method of this application uses placeholders to de-field the verification rule template, which avoids the same logic being stored repeatedly and saves storage resources; and in the process of configuring the verification rule, the verification rule template is called as much as possible, which further ensures the consistency of verification rules across business personnel and systems, and reduces the configuration error rate and the maintenance cost of verification rules.

[0026] (9) The data diversion method of this application can efficiently, accurately and effectively realize data diversion, and the data diversion is highly flexible and can meet the ever-changing business needs. Attached Figure Description

[0027] Figure 1This is a flowchart illustrating a data verification rule configuration method according to this application. Detailed Implementation

[0028] To make the technical solution of this application clearer and more explicit, the application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Solutions derived by those skilled in the art through equivalent substitution and conventional reasoning of the technical features of the technical solution of this application without creative effort all fall within the protection scope of this application.

[0029] Example 1

[0030] This application provides a method for configuring data verification rules, such as... Figure 1 As shown, it includes the following steps: S1: Programmers pre-build the runtime environment of the configuration module using a unified code writing logic.

[0031] S2, business personnel transform business requirements into logical operations between several basic requirements.

[0032] S3 allows business personnel to retain the logical operations between basic requirements within the configuration module, while transforming each basic requirement into logical operations between several basic verification units, and then complete the configuration of verification rules for the current business requirement within the configuration module.

[0033] Within the configuration module, the validation rules for business requirements are presented as the logical operation results between several basic validation units.

[0034] Let's illustrate S2 with an example: If the business requirement is to offload data whose "engineering cycle must not exceed 10 years", then we need to intercept and offload data whose engineering cycle exceeds 10 years. Therefore, the business personnel transformed this business requirement into three mutually exclusive but logically parallel basic requirements: Basic requirement 1: If the project cycle unit is "year", then the project cycle value should be ≤10.

[0035] Basic requirement 2: If the project cycle unit is "month", then the project cycle value should be ≤120.

[0036] Basic requirement 3: If the project cycle unit is "day", then the project cycle value should be ≤3652. Calculated based on a ten-year period including two leap years.

[0037] That is, business requirements should be transformed into: basic requirement 1 ∪ basic requirement 2 ∪ basic requirement 3.

[0038] In S3: The basic verification unit is in the form of a triple of {first element, verification relation, second element}. The verification relation refers to the logical relationship between the first element and the second element in the current basic verification unit.

[0039] Both the first element and the second element fall into one of the following categories: The values ​​of a field (such as "project name" or "company market value"), field attributes (such as field value length), numerical values ​​(such as 10), strings (such as "approval"), dates (such as 2000-01-01), and constraint expressions (such as regular expressions, the number of times a string appears, and the result of arithmetic operations involving the field value length).

[0040] The verification relationship falls into one of the following categories: Text equals: Used to determine if two strings are the same; Text is not equal to: Used to determine whether two strings are different; Greater than >, less than <, equal to =, greater than or equal to ≥, less than or equal to ≤: used for numerical comparison; Earlier than: Used to determine whether a certain date is earlier than another date; "Date later than": Used to determine whether a given date is later than another date; Contains: Used to determine whether a string contains a specified substring. "Does not contain": Used to determine whether a string does not contain a specified substring; Matching: Forward matching based on regular expressions; Mismatch: Negation matching based on regular expressions.

[0041] Let's take S3 as an example: For example, basic requirement 1 is "If the project cycle unit is 'years', then the project cycle value should be ≤10". This means we need to intercept and redirect data with project cycle values ​​greater than 10 years. Basic requirement 1 is transformed into a logical operation between two basic verification units: "Basic verification unit 1, and, basic verification unit 2". Basic verification unit 1 is {project cycle value, >, 10}, and basic verification unit 2 is {project cycle unit, text equal to, year}.

[0042] Optionally, the following may also be included after S3: For a completed verification rule, replace the first element in each basic verification unit of the verification rule with a placeholder and record it as the verification rule template. If the current verification rule template is not stored, store the current verification rule template; otherwise, discard the current verification rule template.

[0043] Optionally, business personnel can bind corresponding template names, business tags, creators, creation times, and applicable scenario descriptions to the validation rule templates. This makes it easier for business personnel to retrieve and use the corresponding validation rule templates later.

[0044] S1' is included after S1 and before S2: S1': After the business personnel determine the keywords based on the business requirements, they use the keywords to search for the validation rule template. If a corresponding validation rule template is found, the content in the placeholders in the validation rule template is replaced with the corresponding business requirement fields in sequence to complete the validation rule configuration for the current business requirement. If no corresponding validation rule template is found, then S2 is executed.

[0045] To complete the validation of a business requirement, it may be necessary to perform logical operations using several validation rule templates.

[0046] Optionally, when new business needs arise, return to S1'.

[0047] In S1', the keyword can be the template name, such as " {0} "More than 10 years", "10 years", among which {0} represents a placeholder; it can be a business tag, such as "time verification", "contract term compliance", or "field length verification"; it can also be the creator, creation time, or applicable scenario description. In S1', keywords are used to search for verification rule templates through binding relationships.

[0048] If there are m new fields (used to replace the content at the placeholders), batch replacing the content at the placeholders in n validation rule templates with these m new fields will quickly yield m×n specific validation rules. For validation rule templates with only one placeholder, the new fields can be replaced in multiple validation rule templates at once (the replacement order between the new fields and multiple placeholders does not need to be considered in validation rule templates with only one placeholder). Batch importing the content at the placeholders can effectively improve configuration efficiency.

[0049] This application provides a data verification rule configuration method that facilitates business personnel to independently configure and maintain verification rules, improves data verification efficiency and effectiveness, and ensures data verification stability.

[0050] In the data validation rule configuration method of this application, programmers only need to pre-build the runtime environment of the configuration module using a unified code writing logic, which improves the consistency of code writing logic and lays the foundation for subsequent configuration and maintenance of validation rules by business personnel. Later, the configuration of validation rules for changing business needs is all done by business personnel who have a deep understanding of those needs; this directly avoids the situation where business needs are divided among different programmers, or where different programmers adjust the business needs code at different times. It also avoids the inconsistent data validation efficiency, unstable validation results, or even deterioration in results caused by different programmers' code writing logic.

[0051] In the data validation rule configuration method of this application, business personnel do not need to perform add, delete, or modify operations based on past business requirement code (which is beyond the capabilities of most business personnel). Instead, they only need to have the ability to logically decompose business requirements to quickly get started, efficiently, and independently complete the configuration and maintenance of validation rules. Business personnel do not need to have coding skills, which is especially suitable for configuring validation rules for sudden business requirements, and also reduces the implementation and maintenance costs of business requirements; it completely eliminates the technical dependence on programmers, significantly shortens the validation rule configuration cycle (from several days to minutes), and improves the agility of data governance response.

[0052] In this application, business personnel first break down business requirements into logical operations between several basic requirements. Then, each basic requirement is further transformed into logical operations between several basic verification units. In other words, for business personnel, the smallest unit for configuring complex verification rules for business requirements is the basic verification unit. This basic verification unit is a triple of {first element, verification relation, second element}, which is highly readable. Therefore, this application not only makes it easy for business personnel to configure verification rules, but also ensures stable and high-quality verification results after configuration. Even if other business personnel modify previous verification rules, there is no issue of not understanding the previous rules, reducing communication overhead.

[0053] On the other hand, in the early stages, this application relies on business personnel to build specific verification rules for various business needs step by step. However, as the specific verification rules are converted into verification rule templates and stored, business personnel can gradually obtain specific verification rules by directly calling relevant verification rule templates through keywords according to business needs (this step is also based on the strong readability of the basic verification unit). Therefore, as time goes on, the efficiency of configuring and maintaining verification rules will become higher and higher. It is especially suitable for enterprise-level scenarios with massive table structures that are similar, realizing "one-time configuration, hundreds of effects", and promoting the large-scale implementation of data quality management.

[0054] Whether configuring specific validation rules by having business personnel construct the logical operations between basic validation units step by step, or by calling validation rule templates and replacing placeholders with specific fields, the code behind these validation rules exists in blocks (such as the code corresponding to the basic validation units and the code corresponding to the validation rule templates) compared to existing technologies. This significantly reduces the consumption of computational processing resources, resulting in higher efficiency for subsequent data splitting applications. Furthermore, business personnel can only adjust the logical breakdown between basic validation units and validation rule templates, not the underlying code. However, different business personnel will generally use the same logical breakdown for the same business requirement. This avoids instability in validation efficiency and effectiveness caused by different business personnel configuring validation rules.

[0055] This application provides a method for configuring data verification rules: ① It enables business personnel to understand and construct validation rules in a way that aligns with business thinking; at the same time, it lays a structural foundation for subsequent rule templates and cross-scenario reuse of validation rules, significantly improving the expressive power and logical clarity of validation rules.

[0056] ② Use placeholders to create fieldless validation rule templates to avoid storing the same logic repeatedly, thus saving storage resources. Furthermore, during the configuration of validation rules, try to call the validation rule template first, which further ensures the consistency of validation rules across business personnel and systems, reducing the configuration error rate and the maintenance cost of validation rules.

[0057] According to technical personnel, using the data verification rule configuration method of this application, adding a new verification rule has been shortened from 2 days to as little as 5 minutes; if a new verification rule is added by calling a verification rule template, it only takes 30 seconds.

[0058] Example 2

[0059] This application also provides a data offloading method, including the following steps: Step 1: Configure data validation rules according to business requirements; configure data validation rules using a data validation rule configuration method as described in Example 1. Step 2: Intercept and distribute the input data using the configured data validation rules.

[0060] If we want to split the original mixed data into three mutually exclusive data sets that meet conditions A, B, and C respectively, we can first configure data validation rules. Using the original mixed data as input, we can intercept data that does not meet condition A and output data that does meet condition A. Then, we can use the intercepted data that does not meet condition A as input again, and by configuring data validation rules, intercept data that does not meet condition B and output data that meets condition B. Finally, we can use the intercepted data that does not meet condition B as input again, and by configuring data validation rules, intercept data that does not meet condition C and output data that meets condition C. This completes the process of splitting the original mixed data into three mutually exclusive data sets that meet conditions A, B, and C respectively.

[0061] The data diversion method proposed in this application can efficiently, accurately and effectively divert data, and the data diversion is highly flexible, which can meet the ever-changing business needs.

[0062] Example 3

[0063] This application also provides a data verification rule configuration system, including: The runtime environment module, the decomposition module, and the configuration module; The runtime environment module is used to build the runtime environment for the configuration module through a unified code writing logic; The decomposition module is used to transform business requirements into logical operations between several basic requirements and then send them to the configuration module. The configuration module is used to retain the logical operations between basic requirements while transforming each basic requirement into logical operations between several basic verification units.

[0064] Each module is programmed or configured to perform the steps of a data validation rule configuration method as described in Example 1.

[0065] Example 4

[0066] This application also provides a data offloading system, including: The system consists of a configuration unit and a flow splitting unit. The configuration unit is used to configure data validation rules according to business requirements; the flow splitting unit is used to execute the configured data validation rules and intercept and split the input data.

[0067] Each module is programmed or configured to perform the steps of a data offloading method as described in Example 2.

[0068] Example 5

[0069] This application also provides a computer-readable storage medium storing a computer program that is programmed or configured to perform a data verification rule configuration method as described in Embodiment 1.

[0070] This application also provides a computer program product, including a computer program / instructions that are executed by a processor to implement the steps of a data verification rule configuration method as described in Embodiment 1.

[0071] The technologies, shapes, and structures not described in detail in this application are all well-known technologies. It should also be noted that the above are merely preferred embodiments created in this application and are not intended to limit the creation of this application. The components or steps in the embodiments of this application can be decomposed and / or recombined, and these decompositions and / or recombinations should be considered as equivalent solutions of this application and should all fall within the protection scope of this application.

Claims

1. A method for configuring data validation rules, characterized in that, Includes the following steps: S1, programmers pre-build the runtime environment of the configuration module using a unified code writing logic; S2, business personnel transform business requirements into logical operations between several basic requirements; S3: In the configuration module, business personnel retain the logical operations between basic requirements, and then transform each basic requirement into logical operations between several basic verification units before completing the verification rule configuration for the current business requirement.

2. The data verification rule configuration method according to claim 1, characterized in that, In S3: the basic verification unit is a triple of {first element, verification relation, second element}, and the verification relation refers to the logical relationship between the first element and the second element in the current basic verification unit; The test relation falls into one of the following categories: Text equals: Used to determine if two strings are the same; Text is not equal to: Used to determine whether two strings are different; Greater than >, less than <, equal to =, greater than or equal to ≥, less than or equal to ≤: used for numerical comparison; Earlier than: Used to determine whether a certain date is earlier than another date; "Date later than": Used to determine whether a given date is later than another date; Contains: Used to determine whether a string contains a specified substring. "Does not contain": Used to determine whether a string does not contain a specified substring; Matching: Forward matching based on regular expressions; Mismatch: Negation matching based on regular expressions.

3. The data verification rule configuration method according to claim 2, characterized in that, Following S3, the following content is also included: For a completed verification rule, replace the first element in each basic verification unit of the verification rule with a placeholder and record it as the verification rule template. If the current verification rule template is not stored, store the current verification rule template; otherwise, discard the current verification rule template.

4. The data verification rule configuration method according to claim 3, characterized in that, S1' is included after S1 and before S2: S1': After the business personnel determine the keywords based on the business requirements, they use the keywords to search for the validation rule template. If a corresponding validation rule template is found, the content in the placeholders of the validation rule template is replaced with the corresponding business requirement fields in sequence to complete the validation rule configuration for the current business requirement. If no corresponding validation rule template is found, then S2 is executed.

5. The data verification rule configuration method according to claim 4, characterized in that: When new business needs arise, return to S1'.

6. A data verification rule configuration system, characterized in that, include: The system consists of a runtime environment module, a decomposition module, and a configuration module. The runtime environment module is used to build the runtime environment for the configuration module through a unified code writing logic. The decomposition module is used to convert business requirements into logical operations between several basic requirements and then send them to the configuration module; the configuration module is used to retain the logical operations between the basic requirements while converting each basic requirement into logical operations between several basic verification units; each module is programmed or configured to execute the steps of a data verification rule configuration method as described in any one of claims 1-5.

7. A data splitting method, characterized in that, Includes the following steps: Step 1: Configure data validation rules according to business requirements; Data verification rules are configured using a data verification rule configuration method as described in any one of claims 1-5; Step 2: Intercept and distribute the input data using the configured data validation rules.

8. A data offloading system, characterized in that, include: The configuration unit is used to configure data verification rules according to business requirements. The splitting unit is used to execute the configured data verification rules to intercept and split the input data; each module is programmed or configured to perform the steps of the data splitting method as described in claim 7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program that is programmed or configured to perform a data verification rule configuration method as described in any one of claims 1-5.

10. A computer program product comprising a computer program / instructions, characterized in that: When the computer program / instruction is executed by the processor, it implements the steps of a data verification rule configuration method as described in any one of claims 1-5.

Images