Data verification method, device and equipment based on business-driven engine modeling

Abstract

The invention relates to the technical field of big data processing, in particular to a data verification method, device and equipment based on business-driven engine modeling, and the method comprises the steps: obtaining a plurality of target business demands, and generating a corresponding verification rule according to to-be-verified data corresponding to each target business demand; when it is judged that the first preset database corresponding to each piece of to-be-verified data exists in the second preset database, synchronizing the to-be-verified data to the target database; and when the verification strategy of each piece of to-be-verified data meets a preset verification condition, extracting at least one piece of target data from the plurality of pieces of to-be-verified data, and performing data preprocessing on the target data based on the verification rule of each piece of to-be-verified data to obtain a verification result corresponding to each target business demand. Therefore, the problems that in the related technology, data verification depends on manual verification, consistency verification of massive and heterogeneous data is difficult to deal with, the decision data quality cannot be guaranteed, and the processing efficiency is low are solved.

Description

Technical Field

[0001] The present application relates to the field of big data processing technology, and in particular to a data verification method, device and equipment based on business-driven engine class modeling. Background Art

[0002] With the advancement of enterprise digital transformation, the importance of business data in enterprise management decision-making has become increasingly prominent. However, due to the huge amount of data and diverse sources, as well as the different standards for understanding and using data among departments, how to ensure the accuracy and consistency of data has become a major challenge.

[0003] In the related art, data verification for business needs relies on traditional manual passive verification methods.

[0004] However, the traditional manual verification method has the following problems:

[0005] (1) The data verification methods of related technologies cannot truly reflect the business situation and have low accuracy;

[0006] (2) Due to the huge amount of data, the indicators that need to be verified are in the millions, and some indicators even have requirements in the tens of millions. Therefore, the data verification methods of related technologies have low verification performance and cannot meet the verification needs of huge amounts of data;

[0007] (3) Due to uneven data quality and inconsistent data formats from different databases, there are quality issues such as data duplication and missing data;

[0008] (4) Different indicators have different time and member statistical granularity, such as time: real-time, day, week, month, assessment month, quarter, assessment quarter, assessment period and year, members: head office, branch, team, which requires targeted verification solutions, making verification more complicated;

[0009] (5) Different departments have inconsistent understandings of the calculation logic of business indicators and cannot come up with a unified definition;

[0010] (6) There are information islands between departments, and the verification process requires a lot of collaborative communication, resulting in a large amount of information gap;

[0011] Therefore, manual verification methods for data verification are difficult to cope with the consistency verification of massive and heterogeneous data, which will lead to the inability to guarantee the quality of decision-making data and low processing efficiency. Summary of the invention

[0012] The present application provides a data verification method, device, equipment, storage medium and program product based on business-driven engine class modeling to solve the problems that data verification relies on manual verification, is difficult to cope with consistency verification of massive and heterogeneous data, and leads to the inability to guarantee the quality of decision data and low processing efficiency.

[0013] The first aspect of the present application provides a data verification method based on business-driven engine class modeling, including the following steps: obtaining multiple target business requirements, and determining the data to be verified corresponding to each target business requirement, and generating verification rules for each data to be verified based on the data to be verified; determining a first preset database of the data to be verified, and judging whether each first preset database exists in a second preset database respectively, and if each first preset database exists in the second preset database respectively, synchronizing the data to be verified to the target database; designing a verification strategy for each data to be verified based on the data to be verified, and when the verification strategy for each data to be verified meets the preset verification conditions, extracting at least one target data from the multiple data to be verified, and performing data preprocessing on the target data based on the verification rules of each data to be verified, to obtain a verification result corresponding to each target business requirement.

[0014] Preferably, obtaining the multiple target business requirements and determining the data to be verified corresponding to each target business requirement includes: obtaining the data indicator corresponding to each target business requirement and the calculation logic corresponding to the data indicator, the time dimension corresponding to the data indicator, the target organization corresponding to the data indicator, and the statistical dimension of the members corresponding to the data indicator; obtaining the data to be verified corresponding to each target business requirement based on the data indicator and the calculation logic corresponding to the data indicator, the time dimension corresponding to the data indicator, the target organization corresponding to the data indicator, and the statistical dimension of the members corresponding to the data indicator.

[0015] Preferably, designing the verification strategy for each data to be verified based on the data to be verified includes: designing a verification script framework for the data to be verified corresponding to each target business requirement; and obtaining the verification strategy for each data to be verified based on the verification script framework.

[0016] Preferably, after designing the verification strategy for each data to be verified based on the data to be verified, it also includes: if the verification strategy for each data to be verified does not meet the preset verification conditions, the verification script framework is expanded to obtain a new verification strategy for each data to be verified according to the expanded verification script framework, and the new verification strategy for each data to be verified is judged until the new verification strategy for each data to be verified meets the preset verification conditions.

[0017] Preferably, the target data is preprocessed based on the verification rules of each data to be verified to obtain a verification result corresponding to each target business requirement, including: extracting data processing information of the target data within a preset time; comparing the data processing information with the data to be verified, if the comparison results are consistent, then it is determined that the data processing information has been successfully verified and a verification success result is obtained; otherwise, it is determined that the data processing information has failed verification and a verification failure result is obtained, and a verification failure report is generated.

[0018] Preferably, after determining that the data processing information verification fails, obtaining the verification failure result, and generating a verification failure report, it also includes: analyzing the verification failure result, and performing data repair based on the analysis result, comparing the repaired data processing information with the data to be verified until the comparison results are consistent, and ending the verification.

[0019] The second aspect of the present application provides a data verification device based on business-driven engine class modeling, including: an acquisition module, used to obtain multiple target business needs, and determine the data to be verified corresponding to each target business need, and generate verification rules for each data to be verified based on the data to be verified; a determination module, used to determine a first preset database of the data to be verified, and determine whether each first preset database exists in a second preset database respectively, if each first preset database exists in the second preset database respectively, then synchronize the data to be verified to the target database; a design module, used to design a verification strategy for each data to be verified based on the data to be verified, and when the verification strategy of each data to be verified meets the preset verification conditions, extract at least one target data from the multiple data to be verified, and perform data preprocessing on the target data based on the verification rules of each data to be verified to obtain a verification result corresponding to each target business need.

[0020] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the data verification method based on business-driven engine class modeling as described in the above embodiment.

[0021] The fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the data verification method based on business-driven engine class modeling as described in the above embodiment.

[0022] The fifth aspect of the present application provides a computer program product, including a computer program or instructions, characterized in that when the computer program or instructions are executed, the data verification method based on business-driven engine class modeling as described in the above embodiment is implemented.

[0023] Therefore, this application has at least the following beneficial effects:

[0024] This application determines the corresponding data to be verified based on multiple target business needs, generates verification rules for each data to be verified according to the data to be verified, and synchronizes them to the corresponding database according to the data type of the data to be verified, and designs a verification strategy for each data to be verified according to the data to be verified, so as to verify different data to be verified using corresponding verification strategies to obtain verification results corresponding to each target business need, which can adapt to the elasticity of business evolution, can quickly respond to changes in demand, realize consistency verification of multi-source heterogeneous data and large amounts of data, improve data quality, and improve data processing efficiency.

[0025] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0027] Figure 1 A flowchart of a data verification method based on business-driven engine class modeling provided according to an embodiment of the present application;

[0028] Figure 2 It is a flowchart of a method for verifying business data and big data platform data according to an embodiment of the present application;

[0029] Figure 3 This is an example diagram of a data verification device based on business-driven engine class modeling according to an embodiment of the present application;

[0030] Figure 4 Schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0031] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0032] In related technologies, data verification of various activities:

[0033] 1) Confirm business requirements to confirm the rules required to generate data, and reach agreement with various departments on the confirmed calculation rules;

[0034] 2) Confirm the database and data table used in the current activity, and design different data retrieval schemes according to different databases;

[0035] 3) Write verification scripts. Most solutions use Python language.

[0036] 4) Run the script on the pub platform to generate a verification conclusion report to facilitate problem location until all data is verified to be accurate.

[0037] However, different departments have inconsistent understandings of the calculation logic of business indicators, and a unified definition is required, which takes up too many resources. Therefore, this application proposes a method to complete multi-level full-link verification from the source of data collection to the final output results, covering different system levels, and forming an end-to-end consistency monitoring closed loop, aiming to use scenario-driven methodology to make the verification plan highly consistent with business needs and evolve in synergy; continuous evaluation and optimization mechanisms ensure that its inspection effectiveness is always synchronized with the status of the business system; data verification methods have changed the traditional passive method of relying on manual verification, and built an active, automated, and intelligent data quality defense line, which has enhanced the reliability of the results and provided a solid foundation for complex business decisions; it has eliminated the verification blind spots of isolated links in the past, and formed a verification closed loop system with mutual input and output. Enhanced the reliability of results.

[0038] The following describes the data verification method, device, vehicle, storage medium and program product based on business-driven engine class modeling according to the embodiments of the present application with reference to the accompanying drawings. Specifically, Figure 1 A flow chart of a data verification method based on business-driven engine class modeling provided in an embodiment of the present application.

[0039] like Figure 1 As shown, the data verification method based on business-driven engine class modeling includes the following steps:

[0040] In step S101, a plurality of target business requirements are obtained, and the data to be verified corresponding to each target business requirement is determined, and a verification rule for each data to be verified is generated according to the data to be verified.

[0041] Among them, the target business needs include users and head office and branch office personnel's daily understanding of their own or their subordinates' performance in selling insurance products, order status, monthly sales, and types of insurance products with concentrated performance. Head office and branch office personnel include: agents, agent team leaders, agent clerks / sales personnel, etc., without specific restrictions.

[0042] It can be understood that the embodiments of the present application can obtain multiple target business requirements, and determine the data to be verified corresponding to each target business requirement, generate verification rules for each data to be verified based on the data to be verified, so as to use the corresponding verification strategies to verify different data to be verified to obtain the verification results corresponding to each target business requirement, which can adapt to the elasticity of business evolution and can quickly respond to changes in demand.

[0043] It should be noted that the data to be verified usually comes from the enterprise's internal SaaS, Huoshui, Core, Renfa and other business system databases and data warehouses. The databases and tables to be used need to be specifically confirmed based on the indicators, without specific limitations.

[0044] Specifically, use scripts to obtain big data platform data - data to be verified, adb data: business department data - business data to be processed, extract data from both parties and save them in a fixed format that is configurable; clean and organize business data according to the calculation logic of the indicators.

[0045] For example, the cleaned data is: the indicator is [the number of short-term insurance policies issued by the agent within one year]; (1) obtain all agent IDs; (2) query the agent's order table; (3) filter the insurance policies that were issued within one year and are of the short-term insurance type; (4) generate the number of policies for the agent.

[0046] In an embodiment of the present application, multiple target business requirements are obtained, and the data to be verified corresponding to each target business requirement is determined, including: obtaining the data indicators corresponding to each target business requirement and the calculation logic corresponding to the data indicators, the time dimension corresponding to the data indicators, the target organization corresponding to the data indicators, and the statistical dimension of the members corresponding to the data indicators; based on the data indicators and the calculation logic corresponding to the data indicators, the time dimension corresponding to the data indicators, the target organization corresponding to the data indicators, and the statistical dimension of the members corresponding to the data indicators, the data to be verified corresponding to each target business requirement is obtained.

[0047] It can be understood that the embodiments of the present application can obtain the data to be verified corresponding to each target business demand based on the data indicators and the calculation logic corresponding to the data indicators, the time dimension corresponding to the data indicators, the target organization corresponding to the data indicators, and the statistical dimension of the members corresponding to the data indicators, so as to facilitate the subsequent verification of the required data to be verified to obtain the verification results.

[0048] It should be noted that at the beginning of the demand, the demand is confirmed with each business department, including the indicators used in the current version, the calculation logic of the indicators, the time dimension of the indicators, the statistical dimension of the institutions or members of the indicators, the libraries and tables involved and other basic information, to determine the expected results of the verification.

[0049] In step S102, the first preset database of the data to be verified is determined, and it is judged whether each first preset database exists in the second preset database respectively. If each first preset database exists in the second preset database respectively, the data to be verified is synchronized to the target database.

[0050] The target database may be polardb or adb, without specific limitation.

[0051] It is understandable that the embodiments of the present application can synchronize the data to be verified to facilitate subsequent consistency verification of multi-source heterogeneous data and large amounts of data.

[0052] It should be noted that the first preset database may be a library table of the current indicator, and the second preset database may be a wide table database of the data platform, without any specific limitation.

[0053] In step S103, a verification strategy for each data to be verified is designed based on the data to be verified, and when the verification strategy for each data to be verified meets the preset verification conditions, at least one target data is extracted from the multiple data to be verified, and the target data is preprocessed based on the verification rules of each data to be verified to obtain a verification result corresponding to each target business requirement.

[0054] Among them, the preset verification conditions include the verification of the indicator time dimension and the verification of the indicator calculation logic, without specific limitations.

[0055] It can be understood that the embodiments of the present application can design a verification strategy for each data to be verified based on the data to be verified, and when the verification strategy for each data to be verified meets the preset verification conditions, extract at least one target data from multiple data to be verified, and perform data preprocessing on the target data based on the verification rules of each data to be verified to obtain a verification result corresponding to each target business requirement, thereby verifying different data to be verified using corresponding verification strategies to obtain a verification result corresponding to each target business requirement, which can adapt to the elasticity of business evolution, can quickly respond to changes in demand, realize consistency verification of multi-source heterogeneous data and large amounts of data, improve data quality, and improve data processing efficiency.

[0056] Specifically, the verification scheme and verification scenarios are designed: 1. Verification of the time dimension of the indicator: for example, the time when the agent last issued a long-term insurance policy, verify whether the time values ​​calculated by the development side and the test side are consistent, and convert them into a string format of '2024-08-01 10:00:00'; 2. Verification of the indicator calculation logic: for example, the premium of the agent's most recent policy, verify whether the result is correct, and support setting the gap value, for example, 1002, 58 yuan, the gap is 0, 01 yuan, then the calculation of 1002, 57 yuan and 1002, 59 yuan can be judged as correct.

[0057] In an embodiment of the present application, a verification strategy for each data to be verified is designed based on the data to be verified, including: designing a verification script framework for the data to be verified corresponding to each target business requirement; and obtaining a verification strategy for each data to be verified based on the verification script framework.

[0058] It can be understood that the embodiments of the present application can design a verification script framework for the data to be verified corresponding to each target business requirement; based on the verification script framework, the verification strategy for each data to be verified is obtained, thereby realizing consistency verification of multi-source heterogeneous data and a large amount of data, improving data quality, and improving data processing efficiency.

[0059] It should be noted that the SQL language is designed according to the requirements and the version of the database client, and the accuracy of the SQL results is preliminarily verified. For example, write SQL, run it on the server, and check whether the data query results are correct.

[0060] Specifically, the framework of designing the verification script: 1. Whether the data source is already supported. For example, if the current demand is to add a new business database table, then it is necessary to design a new data extractor and data validator for the database table. 2. Check whether the existing framework of the system supports indicator verification. If it does not support the extended system, design an extended verification solution.

[0061] In an embodiment of the present application, after designing a verification strategy for each data to be verified based on the data to be verified, it also includes: if the verification strategy for each data to be verified does not meet the preset verification conditions, the verification script framework is expanded to obtain a new verification strategy for each data to be verified according to the expanded verification script framework, and the new verification strategy for each data to be verified is judged until the new verification strategy for each data to be verified meets the preset verification conditions.

[0062] It can be understood that in the embodiment of the present application, if the verification strategy for each data to be verified does not meet the preset verification conditions, the verification script framework can be expanded to obtain a new verification strategy for each data to be verified based on the expanded verification script framework, and the verification strategy for each new data to be verified can be judged until the verification strategy for each new data to be verified meets the preset verification conditions, thereby achieving consistency verification of multi-source heterogeneous data and a large amount of data, improving data quality, and improving data processing efficiency.

[0063] In an embodiment of the present application, data preprocessing is performed on the target data based on the verification rules of each data to be verified to obtain a verification result corresponding to each target business requirement, including: extracting data processing information of the target data within a preset time; comparing the data processing information with the data to be verified, if the comparison results are consistent, then it is determined that the data processing information verification is successful, and a verification success result is obtained; otherwise, it is determined that the data processing information verification has failed, and a verification failure result is obtained, and a verification failure report is generated.

[0064] It can be understood that the embodiments of the present application can extract data processing information of the target data within a preset time; compare the data processing information with the data to be verified to generate a corresponding verification report, thereby more intuitively reflecting the processing results and improving the user experience.

[0065] It should be noted that the data to be verified is compared with the cleaned business data for verification. For example, the short_num and last_short_time fields of different agents are compared based on test to determine whether they are consistent, thereby generating a result report. The result report is analyzed, and the cause of the error in the data verified in the report is analyzed in real time, and the result analysis cause is fed back.

[0066] According to the data verification method based on business-driven engine class modeling proposed in the embodiment of the present application, the corresponding data to be verified is determined based on multiple target business needs, verification rules for each data to be verified are generated according to the data to be verified, and synchronized to the corresponding database according to the data type of the data to be verified, and a verification strategy for each data to be verified is designed according to the data to be verified, so that the corresponding verification strategy is used to verify different data to be verified to obtain the verification result corresponding to each target business need. The method can adapt to the elasticity of business evolution, can quickly respond to changes in demand, realize consistency verification of multi-source heterogeneous data and large amounts of data, improve data quality, and improve data processing efficiency.

[0067] The following will be combined Figure 3 The data verification method based on business-driven engine class modeling is described. This application consists of four modules: data extractor, data verifier, data comparator and data analyzer. The data extractor extracts data from adb, polardb and dataworks, and saves the source data in dictionary format; the data verifier cleans and verifies the saved dictionary metadata to generate unified format data; the data comparator compares and verifies the processed business system data with the cleaned data to be verified; the data analyzer realizes result analysis and process optimization according to the visual verification results of the verification; the specific steps are as follows:

[0068] Step 1: Metadata-driven business model definition layer, supporting visual business flow modeling

[0069] 1) Confirm the business logic of the indicator data according to the basic requirements document. The indicator data is mainly used to help the head office and branch personnel understand their or their subordinates' performance in selling insurance products, order status, monthly sales, and performance-focused insurance types, etc. For example, the head office and branch personnel include agents, agent team leaders, and agent clerks / sales personnel.

[0070] 2) Determine the source of indicator data. The data displayed on the IDL dashboard usually comes from the enterprise's internal SaaS, Huoshui, Core, Renfa and other business system databases and data warehouses; the database and table to be used must be confirmed based on the indicators.

[0071] 3) Data target, load the extracted data into the server-side database of the IDL dashboard to provide a data source for the dashboard visualization.

[0072] 4) Data flow: configure the data interface, export the data in the source database in incremental or full form, and insert it into the target database after format conversion.

[0073] Step 2: Generate a use case generation engine based on Python language that supports business rule configuration and intelligently combines test scenarios.

[0074] 1) Data acquisition, configuration of library table information, access to the database, and acquisition of data to be verified and business data to be processed.

[0075] 2) Data preprocessing: Check, remove duplicates, complete, calculate and other preprocessing of the original data to obtain the indicator dimension data required by the dashboard. Build format and structure checks at the source of the data to verify the input quality.

[0076] 1. Unified integration and processing of strong type conversion data from different database sources into json format;

[0077] 2. For individual fields such as time type, perform string conversion to prevent subsequent code from formatting the datetime type and reporting errors;

[0078] 3. Unify the field definitions of each data source. For example, the agent cloud insurance ID may be called account_id, app_id, aid in different data sources, and is uniformly defined as app_id;

[0079] 4. Unified verification of different data sources to ensure compliance with field types. For example, if the agent registration time type is string type, it will be checked and can be null, but cannot be int, float, etc. Set up encryption and integrity verification on the data transmission channel to prevent loss in transit.

[0080] Among them, (1) data source transmission adopts Advanced Encryption Standard, abbreviated as AES symmetric encryption algorithm, which performs encryption and decryption after end-to-end data transmission to ensure data security;

[0081] (2) After the client requests the encrypted data, it will use the pre-defined key to decrypt it and obtain the current JSON data for subsequent inspection;

[0082] (3) After the client receives the decrypted data, it will perform a missing check on the data source using the MD5 value to prevent missing data during intermediate transmission.

[0083] Add parameter boundary checks to core business processing units to avoid improper input. Replay and verify key business processes to ensure the correctness of results. Add statistical consistency comparisons to output data to determine the causes of differences.

[0084] 3) Data association: business data from different systems are associated by user, event, time and other dimensions to support aspect analysis.

[0085] 4) Data calculation: according to the requirements of the dashboard, the calculation formulas of various indicators are executed to obtain the final report result data. For the final output results, the expected business verification items are executed.

[0086] Calculation formula example: Management income = Yipei income + direct performance income + job level activity allowance; Long-term insurance average performance = total long-term insurance performance / total number of Starry Sky employees (number of people who join, leave, or place orders)

[0087] 5) Data organization: organize and reconstruct indicator data to support the generation of dashboard charts and reports.

[0088] 6) Data visualization: based on user selection, dynamically call the result data to generate corresponding charts, dashboards and other visual presentations.

[0089] 7) Capture business operation data in real time and provide feedback to optimize and verify scenarios and models.

[0090] Step 3: A cross-level data consistency monitoring mechanism is implemented to achieve an end-to-end verification closed loop.

[0091] 1. Determine whether the basic data types are consistent: strings correspond to strings, and integers correspond to integers;

[0092] 2. The data values ​​are consistent;

[0093] 3. The number of data items is consistent, such as some business fields, the policy number issued by the agent, the number of items is consistent, and each policy number also needs to be consistent).

[0094] Each chain is associated through a unified tracking ID, such as assessment ID + team ID + time dimension, to form an end-to-end verification path. The chains complement each other and work together to ensure output quality.

[0095] 1) At the data access layer, verify the integrity of the database table in real time.

[0096] 1. Verify whether some common fields of the table exist, which can be used as a business reference, such as the modification time, creation time, and auto-increment ID of the data, but not as a strong type check;

[0097] 2. Verify specific fields, such as enumeration value fields. The enumeration values ​​are A and B. Verify that the value of the field must not contain C. This is a strong type check to avoid data acquisition exceptions.

[0098] 2) At the script ETL layer, establish correctness checks for data conversion to ensure that the business is accurately reflected.

[0099] 3) At the script data validation layer, statistical consistency checks are performed to prevent aggregation errors.

[0100] 4) Use an adaptive statistical algorithm to dynamically determine the threshold range of data distribution.

[0101] Step 4: The real-time data pipeline connects the business system to support business scenario simulation verification.

[0102] 1) Configure email or form or html basic information

[0103] 1. Specify the email addresses of relevant business personnel or upload in batches;

[0104] 2. Dynamically configure the data to be displayed in the HTML report template (JS language), parse and format the JSON data through JS to display the verification results on the reverse page, and obtain the verification report in time.

[0105]

[0106] 2) Visual verification report, supporting adjustment of report content and format.

[0107] 3) At the data application layer, associate business processes, re-execute core business cases, and confirm the correctness of the results.

[0108] 4) With the help of visualization technology, the detection effect is enhanced through human-computer collaboration.

[0109] 5) Build a data association graph and analyze the abnormal relationships between nodes.

[0110] 6) Build a heterogeneous model integration system to integrate the results of multiple detection models.

[0111] 7) Continuously monitor verification quality and efficiency indicators to identify optimization directions for the next phase.

[0112] Step 5: Continuously optimize the verification strategy in the dynamic business PROFILE module of machine learning.

[0113] 1) Use supervised learning methods to train classification models to identify known anomaly types.

[0114] 2) Use unsupervised learning cluster analysis to find outliers in the data.

[0115] 3) Apply time series analysis and forecasting methods to discover abnormal trends in data changes.

[0116] In summary, based on a deep understanding of business scenarios, this application needs to build an automated, intelligent, and dynamic data verification system based on multi-source heterogeneous data and an intelligent verification project driven by configurable rules, that is, a business-driven engine-type modeled data verification system; it has the flexibility to adapt to business evolution and can quickly respond to changes in demand.

[0117] Next, a data verification device based on business-driven engine class modeling proposed in an embodiment of the present application will be described with reference to the accompanying drawings.

[0118] Figure 3 It is a block diagram of a data verification device based on business-driven engine class modeling according to an embodiment of the present application.

[0119] like Figure 3 As shown, the data verification device 10 based on business-driven engine class modeling includes: an acquisition module 100, a determination module 200 and a design module 300.

[0120] Among them, the acquisition module 100 is used to acquire multiple target business needs, and determine the data to be verified corresponding to each target business need, and generate verification rules for each data to be verified based on the data to be verified; the determination module 200 is used to determine the first preset database of the data to be verified, and judge whether each first preset database exists in the second preset database respectively. If each first preset database exists in the second preset database respectively, the data to be verified is synchronized to the target database; the design module 300 is used to design a verification strategy for each data to be verified based on the data to be verified, and when the verification strategy of each data to be verified meets the preset verification conditions, extract at least one target data from the multiple data to be verified, and perform data preprocessing on the target data based on the verification rules of each data to be verified to obtain the verification result corresponding to each target business need.

[0121] It should be noted that the aforementioned explanation of the data verification method embodiment based on business-driven engine class modeling is also applicable to the data verification device based on business-driven engine class modeling of this embodiment, and will not be repeated here.

[0122] According to the data verification device based on business-driven engine class modeling proposed in the embodiment of the present application, the corresponding data to be verified is determined based on multiple target business needs, verification rules for each data to be verified are generated according to the data to be verified, and the data is synchronized to the corresponding database according to the data type of the data to be verified. A verification strategy for each data to be verified is designed according to the data to be verified, so that different data to be verified are verified using corresponding verification strategies to obtain verification results corresponding to each target business need. The device can adapt to the elasticity of business evolution, respond quickly to changes in demand, realize consistency verification of multi-source heterogeneous data and large amounts of data, improve data quality, and improve data processing efficiency.

[0123] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0124] Memory 401 , processor 402 , and a computer program stored in the memory 401 and executable on the processor 402 .

[0125] When the processor 402 executes the program, the data verification method based on the business-driven engine class modeling provided in the above embodiment is implemented.

[0126] Furthermore, the electronic device further comprises:

[0127] The communication interface 403 is used for communication between the memory 401 and the processor 402 .

[0128] The memory 401 is used to store computer programs that can be executed on the processor 402 .

[0129] The memory 401 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.

[0130] If the memory 401, the processor 402 and the communication interface 403 are implemented independently, the communication interface 403, the memory 401 and the processor 402 can be connected to each other through a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0131] Optionally, in a specific implementation, if the memory 401, the processor 402 and the communication interface 403 are integrated on a chip, the memory 401, the processor 402 and the communication interface 403 can communicate with each other through an internal interface.

[0132] The processor 402 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application.

[0133] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned data verification method based on business-driven engine class modeling.

[0134] An embodiment of the present application also provides a computer program product, including a computer program or instructions, characterized in that when the computer program or instructions are executed, the data verification method based on business-driven engine class modeling as described in the above embodiment is implemented.

[0135] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0136] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0137] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0138] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one or a combination of multiple of the following technologies known in the art: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array, a field programmable gate array, etc.

[0139] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0140] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A data verification method based on business-driven engine class modeling, characterized in that: The following steps are involved: Acquire multiple target business requirements, determine the data to be verified corresponding to each target business requirement, and generate verification rules for each data to be verified according to the data to be verified; Determine a first preset database of the data to be verified, and determine whether each first preset database exists in a second preset database respectively, and if each first preset database exists in the second preset database respectively, synchronize the data to be verified to a target database; A verification strategy for each data to be verified is designed based on the data to be verified, and when the verification strategy for each data to be verified meets a preset verification condition, at least one target data is extracted from the multiple data to be verified, and data preprocessing is performed on the target data based on the verification rules of each data to be verified to obtain a verification result corresponding to each target business requirement.

2. The method according to claim 1, characterized in that The obtaining of the plurality of target business requirements and determining the data to be verified corresponding to each target business requirement includes: Obtaining data indicators corresponding to each target business requirement and calculation logic corresponding to the data indicators, time dimensions corresponding to the data indicators, target organizations corresponding to the data indicators, and statistical dimensions of members corresponding to the data indicators; Based on the data indicators and the calculation logic corresponding to the data indicators, the time dimension corresponding to the data indicators, the target organization corresponding to the data indicators and the statistical dimension of the members corresponding to the data indicators, the data to be verified corresponding to each target business demand is obtained.

3. The method according to claim 1, characterized in that The designing of a verification strategy for each of the data to be verified based on the data to be verified includes: Design a verification script framework for the data to be verified corresponding to each target business requirement; A verification strategy for each of the data to be verified is obtained based on the verification script framework.

4. The method according to claim 3, characterized in that: After designing a verification strategy for each of the data to be verified based on the data to be verified, the method further includes: If the verification strategy for each data to be verified does not meet the preset verification conditions, the verification script framework is expanded to obtain a new verification strategy for each data to be verified based on the expanded verification script framework, and the verification strategy for each new data to be verified is judged until the new verification strategy for each data to be verified meets the preset verification conditions.

5. The method according to claim 1, characterized in that The performing data preprocessing on the target data based on the verification rule of each data to be verified to obtain the verification result corresponding to each target business requirement includes: Extracting data processing information of the target data within a preset time; The data processing information is compared with the data to be verified. If the comparison results are consistent, the data processing information is determined to be successfully verified and a verification success result is obtained. Otherwise, the data processing information is determined to have failed verification and a verification failure result is obtained, and a verification failure report is generated.

6. The method according to claim 5, characterized in that After determining that the data processing information verification fails, obtaining a verification failure result, and generating a verification failure report, the method further includes: The verification failure result is analyzed, and data repair is performed based on the analysis result, and the repaired data processing information is compared with the data to be verified until the comparison result is consistent, and the verification is terminated.

7. A data verification device based on business-driven engine class modeling, characterized in that: include: An acquisition module is used to acquire multiple target business requirements, determine the data to be verified corresponding to each target business requirement, and generate a verification rule for each data to be verified according to the data to be verified; A determination module, used to determine a first preset database of the data to be verified, and to determine whether each first preset database exists in a second preset database respectively, and if each first preset database exists in the second preset database respectively, then synchronizing the data to be verified to a target database; A design module is used to design a verification strategy for each data to be verified based on the data to be verified, and when the verification strategy for each data to be verified meets a preset verification condition, extract at least one target data from multiple data to be verified, and perform data preprocessing on the target data based on the verification rules of each data to be verified to obtain a verification result corresponding to each target business requirement.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the data verification method based on business-driven engine class modeling as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, it is used to implement the data verification method based on business-driven engine class modeling as described in any one of claims 1-6.

10. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed, the data verification method based on business-driven engine class modeling as described in any one of claims 1-6 is implemented.

Images