Data management plug-in and verification conversion method

Abstract

The invention discloses a data governance plug-in and a verification conversion method, and relates to the technical field of data processing, and the data governance plug-in comprises a data preloading module, an SQL verification module, a complex conversion module and a data storage module. According to the invention, the verification capability and efficiency are improved, the limitation of traditional verification is broken through, the data of multiple data sources is loaded to the memory by using the data preloading module, and cross-data-source and cross-table complex verification can be realized. In the memory environment, the data association relationship is constructed based on the primary key and the foreign key, so that frequent database query is avoided, and the verification efficiency is greatly improved. For example, in a verification scene of personnel and an organizational structure table, the accuracy of personnel unit coding can be rapidly judged, the strict requirement of a complex business scene for data accuracy is effectively met, and compared with a traditional verification mode, the time overhead of data transmission and processing is greatly reduced.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data governance plug-in and a verification conversion method. Background Art

[0002] During data processing, routine validation is fundamental to ensuring data quality. Common validations include validation of named entity data such as ID cards, mobile phone numbers, IP addresses, and email addresses. This type of validation primarily uses established rules to determine the correctness of the data format. For example, ID card numbers must conform to specific encoding rules, including area codes, date of birth codes, sequence codes, and checksums. Regular expressions are used to match their format and verify length to ensure data validity. Non-null validation checks whether data fields contain empty values, null, or special characters representing null, preventing missing data from impacting subsequent processing. Longitude and latitude verification aims to determine the accuracy of the latitude and longitude information in the data. Traditionally, this approach utilizes the geo search function in Elasticsearch (ES) to identify the county to which the latitude and longitude belong and compares it with the attribute field to verify accuracy. Meanwhile, routine conversions primarily perform simple formatting and processing on the data itself. Whitespace removal removes whitespace characters on both sides of a string to unify the data format for easier matching and analysis. Case conversion standardizes the case of data to avoid data recognition issues caused by case discrepancies. Specific character removal eliminates unnecessary special characters from data based on business needs, making the data more standardized. String truncation extracts portions of the original string at a specified position and length to meet specific business display or processing requirements. Date and time standardization typically uses the SimpleDateFormatter formatting tool to convert date and time data into a unified format for easier storage, comparison, and display. For numerical precision standards, BigDecimal is often used for rounding operations to control the number of decimal places and ensure data accuracy in numerical calculations and presentation.

[0003] At present, although existing technologies provide basic solutions for routine verification and conversion, their limitations are gradually becoming apparent as business scenarios become increasingly complex. Most existing verification technologies only analyze the data itself and cannot meet complex verification requirements across data sources and tables. In actual business, for example, in an enterprise's customer information management system, it may be necessary to verify whether the customer's department code is in the enterprise's organizational structure table. This involves data association verification between different data sources and data tables, which is difficult to achieve with existing technologies. For complex verification scenarios, such as those involving multi-table associations and containing complex query conditions, existing technologies lack effective processing methods and cannot efficiently obtain a list of data IDs that do not meet the conditions, resulting in affected efficiency and accuracy of data verification. In terms of data conversion, existing technologies can only perform simple data conversion itself, which is difficult to meet data conversion requirements under complex business logic. In some financial business scenarios, data needs to be converted under multiple conditions according to complex business rules, and existing technologies cannot provide flexible and powerful conversion functions.

[0004] Currently, no effective solutions have been proposed for the problems in related technologies. Summary of the Invention

[0005] In response to the problems in related technologies, the present invention proposes a data governance plug-in and verification conversion method, aiming to solve the problem that existing data verification and conversion technologies cannot meet complex business needs, and provide a technical solution that can realize cross-data source and cross-table verification, handle complex SQL verification, and meet complex business conversion, thereby improving the accuracy, efficiency and flexibility of data processing to adapt to diverse business scenarios.

[0006] The technical solution of the present invention is achieved as follows:

[0007] on the one hand:

[0008] A data governance plug-in, comprising: a data preloading module, an SQL validation module, a complex conversion module and a data storage module, wherein;

[0009] The data preloading module is used to load data from multiple data sources into the memory to provide a data basis for the operation of the SQL verification module and the complex conversion module;

[0010] The SQL verification module is used to calibrate and execute specific SQL statements, verify the data, and return a list of data IDs that do not meet the rules;

[0011] The complex conversion module is used to calibrate the select xxx as key,xxx as value fromxxx query statement based on business needs to achieve complex data conversion;

[0012] Data storage module: stores data from the data source, data loaded in the memory, verification results, and converted data.

[0013] Furthermore, it also includes: a data update monitoring module, which is used to monitor data changes in the data source, trigger data loading and updating operations according to a preset update strategy, and ensure the accuracy of memory data.

[0014] Furthermore, the SQL verification module includes: a query statement generator, which is used to dynamically generate SQL query statements according to different business rules and verification requirements.

[0015] Furthermore, the complex conversion module includes: a function library extension interface for allowing users to customize functions and integrate them into the system.

[0016] on the other hand:

[0017] A data governance plug-in verification conversion method includes the following steps:

[0018] Perform complex validation in advance. Using data preloading technology, data from multiple data sources is preloaded into memory. Cross-data source and cross-table validation is performed in the memory environment. By building data associations, complex data logic can be judged.

[0019] Perform SQL validation, based on complex validation, calibrate specific SQL statements, execute SQL queries, and return a list of data IDs that do not meet the validation rules;

[0020] Perform complex conversions, calibrate the select xxx as key,xxx as value from xxx query statement, and flexibly convert data according to business rules, including converting the specified fields to the target value.

[0021] The complex verification step includes the following steps:

[0022] Preload data in advance, determine the data sources that need to be verified based on business needs, and use data reading tools to read and load data from multiple data sources into memory;

[0023] Build data association relationships. In memory, build association relationships between data based on the primary key and foreign key information of the data to form the data set required for data verification;

[0024] Perform validation operations, write validation logic code based on the constructed data set, validate data across data sources and tables, and determine whether the data complies with business rules.

[0025] The SQL verification step includes the following steps:

[0026] Set SQL statements and calibrate and adapt SQL query statements based on the specific needs of complex verification. In multi-table association verification scenarios, use JOIN statements to implement inter-table associations and set verification conditions through the WHERE clause;

[0027] Execute SQL query, execute the designed SQL query statement in the database, and obtain the list of data IDs that do not meet the verification rules.

[0028] The complex conversion step includes the following steps:

[0029] Determine business requirements in advance, including at least the conversion fields, target values, and business logic;

[0030] Calibrate query statements. According to business needs, calibrate the select xxx as key,xxx as value fromxxx query statement and adjust fields and logic to achieve flexible data conversion.

[0031] Execute the query statement, execute the calibrated query statement in the database, and complete the data conversion operation.

[0032] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a verification conversion method for a data governance plug-in.

[0033] Beneficial effects of the present invention:

[0034] 1. The present invention improves verification capabilities and efficiency, breaks through the limitations of traditional verification, and uses a data pre-loading module to load data from multiple data sources into memory, which can realize complex verification across data sources and tables. In a memory environment, data association relationships are constructed based on primary keys and foreign keys to avoid frequent database queries, greatly improving verification efficiency. For example, in the verification scenario of personnel and organizational structure tables, the accuracy of personnel unit codes can be quickly determined, effectively meeting the strict requirements of complex business scenarios for data accuracy. Compared with traditional verification methods, the time overhead of data transmission and processing is greatly reduced.

[0035] 2. The present invention enhances verification flexibility and adaptability. The query statement generator of the SQL verification module can dynamically generate SQL query statements according to different business rules and verification requirements. For complex verification scenarios with multiple tables and complex conditions, SQL statements can be accurately written to obtain a list of data IDs that do not meet the conditions, ensuring data accuracy and business compliance, allowing the system to flexibly respond to various complex verification requirements. At the same time, it meets the needs of complex business conversion. The complex conversion module uses a specific query statement structure to flexibly implement complex data conversion according to business rules. The provided function library extension interface allows users to customize functions to meet diverse calculation and processing needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 is a principle block diagram of a data governance plug-in according to an embodiment of the present invention;

[0038] Figure 2 This is a flow chart of a verification conversion method for a data governance plug-in according to an embodiment of the present invention;

[0039] Figure 3 This is a schematic diagram of a complex verification scenario of a data governance plug-in and a verification conversion method according to an embodiment of the present invention;

[0040] Figure 4 This is a schematic diagram of an SQL verification scenario of a data governance plug-in and a verification conversion method according to an embodiment of the present invention;

[0041] Figure 5 This is a schematic diagram of a complex conversion scenario of a data governance plug-in and a verification conversion method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention are within the scope of protection of the present invention.

[0043] According to an embodiment of the present invention, a data governance plug-in is provided.

[0044] like Figure 1As shown, the data governance plug-in according to an embodiment of the present invention includes: a data preloading module 1, an SQL verification module 2, a complex conversion module 3 and a data storage module 4, wherein;

[0045] Data preloading module 1 is used to load data from multiple data sources into memory, providing a data basis for the operations of SQL verification module 2 and complex conversion module 3;

[0046] The data preloading module 1 includes a data update monitoring module 11 for monitoring data changes in a data source and triggering data loading and updating operations according to a preset update strategy to ensure the accuracy of memory data.

[0047] This technical solution, through data preloading technology, breaks through the limitation of traditional verification that only targets the data itself. When performing complex verification, the system will pre-load the data of the relevant data source into the memory. For example, in the verification scenario of the personnel information table and the organization table, the system will load the unit code data in the organization table into the memory. Then, when verifying the personnel unit code in the personnel information table, a quick judgment is made directly in the memory to check whether the personnel unit code exists in the organization table. This method greatly improves the efficiency of verification, avoids frequent database query operations, and reduces the time overhead of data transmission and processing. At the same time, through memory judgment, complex verification logic across data sources and tables can be easily implemented to meet the strict requirements for data accuracy in complex business scenarios.

[0048] SQL verification module 2, used to identify and execute specific SQL statements, verify the data, and return a list of data IDs that do not meet the rules;

[0049] The SQL verification module 2 includes a query statement generator, which is used to dynamically generate SQL query statements according to different business rules and verification requirements, thereby improving the flexibility and adaptability of the system.

[0050] This technical solution adopts an SQL verification method for particularly complex verification scenarios, such as those involving multi-table associations and containing complex query conditions. In this verification method, the system will write corresponding SQL query statements based on complex business needs. By executing these SQL statements, the system can query a list of data IDs that do not meet specific conditions. For example, in a complex business scenario involving a customer order table, a product table, and an inventory table, it is necessary to verify whether the number of products in the order exceeds the inventory quantity, and whether the customer credit rating of the order meets the requirements. At this time, you can write a complex SQL statement to associate these three tables and set corresponding query conditions to obtain a list of data IDs that do not meet the conditions. Then, according to this list, the records corresponding to the data ID are verified and processed in detail to ensure the accuracy of the data and the compliance of the business.

[0051] Complex conversion module 3 is used to calibrate the select xxx as key,xxx as valuefrom xxx query statement based on business needs to achieve complex data conversion;

[0052] The complex conversion module 3 includes a function library extension interface that allows users to customize functions and integrate them into the system to meet the diverse computing and processing requirements in complex business conversion.

[0053] This technical solution implements complex data conversion by using the statement structure of "select xxx as key,xxx as value from xxx;". Among them, "key" represents the field to be converted, and "value" represents the conversion target value. In a business scenario, it is necessary to convert the coordinates start_stake of the starting point of the goods into a specific kilometer format pile_in_km_format. You can write corresponding query statements to calculate and convert the coordinate data according to business rules, and convert it into kilometer format data that meets the requirements to meet the special needs of data display and processing in the business. This conversion method can flexibly perform complex conversion operations on data according to different business rules, greatly improving the flexibility and adaptability of data processing.

[0054] Data storage module 4: stores data from the data source, data loaded in the memory, verification results and converted data.

[0055] According to an embodiment of the present invention, a verification conversion method for a data governance plug-in is provided.

[0056] like Figure 2-Figure 5 As shown, the verification conversion method of the data governance plug-in according to an embodiment of the present invention includes:

[0057] Step S1: Pre-complex validation is performed. Using data preloading technology, data from multiple data sources is preloaded into memory. Cross-data source and cross-table validation is performed on the data in the memory environment. By building data associations, complex data logic is judged.

[0058] The complex verification step includes the following steps:

[0059] Preload data in advance, determine the data sources that need to be verified based on business needs, and use data reading tools to read and load data from multiple data sources into memory; the Java cache framework GuavaCache can be used to achieve efficient data loading and management.

[0060] Build data association relationships. In memory, build association relationships between data based on the primary key and foreign key information of the data to form the data set required for data verification;

[0061] Perform validation operations, write validation logic code based on the constructed data set, validate data across data sources and tables, and determine whether the data complies with business rules.

[0062] Step S2: Perform SQL verification. Based on complex verification, a specific SQL statement is identified. After executing the SQL query, a list of data IDs that do not meet the verification rules is returned.

[0063] The SQL verification step includes the following steps:

[0064] Set SQL statements and calibrate and adapt SQL query statements based on the specific needs of complex verification. In multi-table association verification scenarios, use JOIN statements to implement inter-table associations and set verification conditions through the WHERE clause;

[0065] Execute SQL queries, execute the designed SQL query statements in the database, and obtain a list of data IDs that do not meet the verification rules; you can use database access technologies such as JDBC to execute SQL statements and obtain results.

[0066] Step S3, perform complex conversion, calibrate the query statement select xxx as key,xxx as value from xxx, and flexibly convert the data according to business rules, including converting the specified field to be converted (key) into the target value (value).

[0067] The complex conversion step includes the following steps:

[0068] Determine business requirements in advance, including at least the conversion fields, target values, and business logic;

[0069] Calibrate query statements. According to business needs, calibrate the select xxx as key,xxx as value fromxxx query statement and adjust fields and logic to achieve flexible data conversion.

[0070] Execute the query statement, execute the calibrated query statement in the database, and complete the data conversion operation.

[0071] In addition, according to an embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the verification conversion method of the data governance plug-in are performed.

[0072] In summary, with the help of the above technical solution of the present invention, the following effects can be achieved:

[0073] 1. The present invention improves verification capabilities and efficiency, breaks through the limitations of traditional verification, and uses a data pre-loading module to load data from multiple data sources into memory, which can realize complex verification across data sources and tables. In a memory environment, data association relationships are constructed based on primary keys and foreign keys to avoid frequent database queries, greatly improving verification efficiency. For example, in the verification scenario of personnel and organizational structure tables, the accuracy of personnel unit codes can be quickly determined, effectively meeting the strict requirements of complex business scenarios for data accuracy. Compared with traditional verification methods, the time overhead of data transmission and processing is greatly reduced.

[0074] 2. The present invention enhances verification flexibility and adaptability. The query statement generator of the SQL verification module can dynamically generate SQL query statements according to different business rules and verification requirements. For complex verification scenarios with multiple tables and complex conditions, SQL statements can be accurately written to obtain a list of data IDs that do not meet the conditions, ensuring data accuracy and business compliance, allowing the system to flexibly respond to various complex verification requirements. At the same time, it meets the needs of complex business conversion. The complex conversion module uses a specific query statement structure to flexibly implement complex data conversion according to business rules. The provided function library extension interface allows users to customize functions to meet diverse calculation and processing needs.

[0075] The foregoing is merely a preferred embodiment of the present invention and is not intended to limit the present invention. A person skilled in the art will readily appreciate other embodiments of the present invention after considering the disclosure in the specification and examples. This application is intended to cover any variations, uses, or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered merely exemplary, and the true scope and spirit of the present invention are indicated by the claims.

[0076] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A data governance plug-in, characterized in that: include: Data preloading module (1), SQL verification module (2), complex conversion module (3) and data storage module (4), wherein; The data preloading module (1) is used to load data from multiple data sources into the memory, and is used to provide a data basis for the operation of the SQL verification module (2) and the complex conversion module (3); The SQL verification module (2) is used to calibrate and execute specific SQL statements, verify the data, and return a list of data IDs that do not meet the rules; Complex conversion module (3), used to calibrate the select xxx as key,xxx as value fromxxx query statement based on business needs to achieve complex data conversion; Data storage module (4): stores data from the data source, data loaded in the memory, verification results and converted data.

2. The data governance plug-in according to claim 1, characterized in that: The data preloading module (1) further comprises: a data update monitoring module (11) for monitoring data changes in a data source, triggering a data loading and updating operation according to a preset update strategy, and ensuring the accuracy of the memory data.

3. The data governance plug-in according to claim 1, characterized in that: The SQL verification module (2) comprises: a query statement generator, which is used to dynamically generate SQL query statements according to different business rules and verification requirements.

4. The data governance plug-in according to claim 1, characterized in that: The complex conversion module (3) includes a function library expansion interface for allowing users to define functions and integrate them into the system.

5. A data governance plug-in verification conversion method, used in the data governance plug-in verification conversion method according to any one of claims 1 to 4, characterized in that: The following steps are involved: Perform complex validation in advance. Using data preloading technology, data from multiple data sources is preloaded into memory. Cross-data source and cross-table validation is performed in the memory environment. By building data associations, complex data logic can be judged. Perform SQL validation, based on complex validation, calibrate specific SQL statements, execute SQL queries, and return a list of data IDs that do not meet the validation rules; Perform complex conversions, calibrate the select xxx as key,xxx as value from xxx query statement, and flexibly convert data according to business rules, including converting the specified fields to the target value.

6. The verification conversion method of the data governance plug-in according to claim 5 is characterized in that: The complex verification step includes the following steps: Preload data in advance, determine the data sources that need to be verified based on business needs, and use data reading tools to read and load data from multiple data sources into memory; Build data association relationships. In memory, build association relationships between data based on the primary key and foreign key information of the data to form the data set required for data verification; Perform validation operations, write validation logic code based on the constructed data set, validate data across data sources and tables, and determine whether the data complies with business rules.

7. The verification conversion method of the data governance plug-in according to claim 5 is characterized in that: The SQL verification step includes the following steps: Set SQL statements and calibrate and adapt SQL query statements based on the specific needs of complex verification. In multi-table association verification scenarios, use JOIN statements to implement inter-table associations and set verification conditions through the WHERE clause; Execute SQL query, execute the designed SQL query statement in the database, and obtain the list of data IDs that do not meet the verification rules.

8. The verification conversion method of the data governance plug-in according to claim 5 is characterized in that: The complex conversion steps include the following steps: Determine business requirements in advance, including at least the conversion fields, target values, and business logic; Calibrate query statements. According to business needs, calibrate the select xxx as key,xxx as value from xxx query statement and adjust fields and logic to achieve flexible data conversion. Execute the query statement, execute the calibrated query statement in the database, and complete the data conversion operation.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the verification and conversion method of the data governance plug-in described in any one of claims 5-8 are implemented.

Images