Data audit method and device based on data cutover

Abstract

The present invention discloses a data audit method and device based on data cutover, which relates to the field of data engineering technology. The main purpose of the present invention is to propose a standardized data audit scheme for data cutover projects, so as to improve the quality of data cutover. The main technical scheme of the present invention is: in the process of cutting over source data to target data, determine the unit data required to perform the data cutover operation, and the unit data is the data with the minimum granularity split based on the association relationship of the source data; detect whether the unit data on which the target data test script depends has completed the data cutover operation; if completed, execute the target data test script; detect whether the data test script on which the target functional module test script depends has passed the test; if the test has passed, execute the target functional module test script; determine the completion quality of the data cutover operation according to the execution result of the functional module test script.

Description

Technical Field

[0001] The present invention relates to the technical field of data engineering, and in particular to a data audit method and device based on data cutover. Background Art

[0002] With the popularity of cloud computing, more and more customers are building their own systems on the cloud. And with the popularity of users, more and more users are gradually needing to migrate their core systems from the cloud to the cloud. Unlike the migration of new systems to the cloud, the migration of core systems often requires compatibility with the original data. In order for the new system to accurately apply the data of the old system, it is necessary to convert the data of the old system according to preset rules and migrate it to the new system when switching between the new and old systems. The act of converting and migrating data is called data cutover. The consistency, accuracy, completeness and availability of the cutover data are important evaluation criteria for the quality of data cutover.

[0003] At present, the industry has no systematic standard solution for the data audit process after data cutover. Most of them are physical-level data audits, such as simple audits based on the number of data items, content summary, etc. However, when such simple audit data is used in subsequent system function tests, once a problem or failure occurs, the testers often need to spend a lot of energy to analyze whether the cause of the failure is a data problem or a functional problem, which affects the execution progress of the overall project. Summary of the invention

[0004] In view of the above problems, the present invention proposes a data audit method and device based on data cutover, the main purpose of which is to propose a standardized data audit scheme for data cutover projects, so as to improve the quality of data cutover.

[0005] In order to achieve the above object, the present invention mainly provides the following technical solutions:

[0006] In a first aspect, the present invention provides a data audit method based on data cutover, specifically comprising:

[0007] In the process of cutting over the source data into the target data, determining the unit data required for performing the data cutting over operation, wherein the unit data is the data of the minimum granularity split based on the association relationship of the source data;

[0008] Check whether the unit data that the target data test script depends on has completed the data cutover operation;

[0009] If completed, the target data test script is executed;

[0010] Check whether the data test script that the target functional module test script depends on has passed the test;

[0011] If the test passes, the target function module test script is executed;

[0012] The completion quality of the data cutover operation is determined according to the execution result of the function module test script.

[0013] In a second aspect, the present invention provides a data audit device based on data cutover, specifically comprising:

[0014] An acquisition unit, used to determine unit data required for performing a data cutover operation in a process of cutting over source data into target data, wherein the unit data is data of the minimum granularity split based on an association relationship of the source data;

[0015] A first detection unit is used to detect whether the unit data on which the target data test script depends has completed the data cutover operation;

[0016] A first testing unit, configured to execute the target data testing script when the first detection unit determines that the unit data completes the data cutover operation;

[0017] A second detection unit is used to detect whether the data test script of the first test unit test, on which the target functional module test script depends, has passed the test;

[0018] A second testing unit, configured to execute the target functional module test script when the second detection unit determines that the data test script has passed the test;

[0019] The determination unit is used to determine the completion quality of the data cutover operation according to the execution result of the function module test script by the second test unit.

[0020] In a third aspect, the present invention provides a data audit method based on data cutover, the method being applied to a data cutover end, the data cutover end being used to cutover source data with a first association relationship acquired by a source end into target data with a second association relationship required by a target end, the method comprising:

[0021] Obtain data test scripts and functional module test scripts based on data audit requests;

[0022] In the process of cutting over the source data into the target data, determining the unit data required for performing the data cutting over operation, wherein the unit data is the data of the minimum granularity split based on the association relationship of the source data;

[0023] Check whether the unit data that the data test script depends on has completed the data cutover operation;

[0024] If completed, the data test script is executed;

[0025] Check whether the data test script that the functional module test script depends on has passed the test;

[0026] If the test passes, the functional module test script is executed;

[0027] An audit report corresponding to the data audit request is generated according to the execution result of the function module test script, and the audit report is used to evaluate the completion quality of the data cutover operation.

[0028] On the other hand, the present invention provides a processor, which is used to run a program, wherein the program executes the above-mentioned data audit method based on data cutover when running.

[0029] On the other hand, the present invention provides a computer-readable storage medium, which is used to store a computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned data audit method based on data cutover.

[0030] By means of the above technical scheme, the data audit method and device based on data cutover provided by the present invention is a standardized scheme for data audit performed when heterogeneous data is cutover. In the scheme, the data audit process is divided into two levels. The audit test of the data level is first performed. When the data is correct, the audit test of the system function module level is further performed. After passing the above tests, it can be determined that the heterogeneous data can be correctly identified and applied in the new system after data cutover. And when the scheme is applied to the data cutover process, only the test scripts corresponding to the data level and the function module level need to be manually provided. When the system performs data cutover, it will automatically trigger the running of the test script according to the completion of the cutover task without manual intervention. Compared with the existing simple auditing method, the embodiment of the present invention performs data audit on the data association relationship synchronously during the data cutover process, which improves the consistency, accuracy, completeness and availability of the data after data cutover, and is also convenient for eliminating the faults caused by data cutover errors when a fault problem occurs in the subsequent functional test, improving the troubleshooting efficiency, and thus improving the overall execution efficiency of the project.

[0031] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0033] Figure 1 A schematic diagram showing a data cutover process framework with data audit operations is shown;

[0034] Figure 2 A flow chart of a data audit method based on data cutover proposed in an embodiment of the present invention is shown;

[0035] Figure 3 A flow chart of another data audit method based on data cutover proposed in an embodiment of the present invention is shown;

[0036] Figure 4 A schematic diagram of a test script hierarchy framework in an embodiment of the present invention is shown;

[0037] Figure 5 A block diagram showing a data audit device based on data cutover proposed in an embodiment of the present invention is shown;

[0038] Figure 6 A block diagram showing another data audit device based on data cutover proposed in an embodiment of the present invention;

[0039] Figure 7 A flow chart of another data audit method based on data cutover proposed in an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0040] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.

[0041] The data audit method proposed in the present invention is a verification method for data that has completed the cutover operation during the data cutover process. Therefore, before describing the specific embodiments of the present invention, the overall process of data cutover in the present invention is briefly introduced. The data cutover solution in the present invention is a universal and standardized data migration solution proposed to meet the needs of users to migrate their core systems under the cloud to the cloud. In this solution, the data migration and transformation is mainly performed by the data cutover end, and the data cutover end can adopt data cutover rules of corresponding formats according to the computing medium used by the user. The computing medium includes but is not limited to databases, application systems, big data platforms, etc. Here, the computing medium is taken as an example of a database, that is, the data cutover end is applied to the database, and the data cutover process is as follows: Figure 1As shown in the figure, the source end is used to store the data of the core system under the cloud, while the target end is used to store the data of the system on the cloud. The cutover transit database in the figure is used as the computing medium used by the customer to perform data cutover operations to realize the conversion and migration of the source data of the source end to the target end. The specific process is as follows: the user triggers a data cutover request at the data cutover end. The data cutover request contains information about obtaining the specified source data from the specified source end. The data cutover end calls the data transmission service DTS (Data Transmission Service) in order to obtain the specified source data from the specified source end, and store the obtained source data in the corresponding table (source library mirror) in the database; afterwards, the data cutover end will determine the first association relationship of the source data based on the obtained source data through the old system (core system under the cloud) data model, and at the same time, obtain the cutover rule applicable to the database, which is a pre-set method for converting the first association relationship of the data into a second association relationship applicable to the new system (core system on the cloud); in this way, the data cutover end can convert the obtained source data according to the cutover rule, convert the source data with the first association relationship into the target data with the second association relationship, and save the target data in the corresponding position (target library mirror) in the database. It should be noted that when applying the cutover rule, the source data needs to be split into gear data according to the first association relationship, and then the association relationship between the unit data is reconstructed according to the cutover rule to obtain the target data with the second association relationship. At the same time, the data audit proposed in the present invention is to verify the cutover operation performed in the process of cutting over the source data to the target data, mainly through the pre-acquired test script to automatically test the target data, and the difference between the data audit in the present invention and the existing method is that the applied test script is a comprehensive test based on the data level and the functional module level, so as to ensure the correct conversion of heterogeneous data; finally, the data cutover end transmits the converted target data to the target end through the data transmission tool (the data synchronization tool DataX shown in the figure). Based on the description of the above process, the data cutover process can also be reused in other computing media, and the difference in its reuse process is only that it is necessary to obtain the cutover rules applicable to the corresponding computing medium.

[0042] Based on the above overall introduction to the data cutover process, a data audit method based on data cutover provided by an embodiment of the present invention is further explained. This method is mainly aimed at verifying the data after data cutover of heterogeneous data. Compared with homogeneous data, after data cutover, due to the change in the data association relationship of heterogeneous data, the data audit needs to verify the correctness of the conversion of its association relationship. Therefore, the audit method of homogeneous data is not suitable for heterogeneous data. In addition, since the industry does not have a unified standardized solution for the execution process of data cutover projects, it is mainly customized development for projects. Therefore, there is no standardized verification method for the audit of cutover data, which makes the data audit operation and subsequent functional testing performed simultaneously, which increases the complexity and difficulty of the test process and seriously reduces the progress of project execution. To this end, the present invention proposes a standardized audit solution for data cutover of heterogeneous data, which can be applied to various heterogeneous data cutover solutions. The specific steps of this method are as follows: Figure 2 As shown, the method includes:

[0043] Step 101: In the process of cutting over source data to target data, determine the unit data required to perform the data cutting over operation.

[0044] The unit data refers to the data of the smallest granularity split based on the association relationship of the source data. Figure 1 It can be seen from the data cutover process shown that when converting source data with a first association relationship into target data with a second association relationship, the source data needs to be split, and the split data is the unit data, which can be reconstructed into the target data with the second association relationship according to the cutover rules. In addition, the source data and the target data in this embodiment are heterogeneous data, so the first association relationship and the second association relationship are different association relationships. For example, when cutting over heterogeneous relational databases, that is, when cutting over data for relational databases in the old and new systems, the data in the relational database in the old system and the data in the relational database in the new system are heterogeneous data, and the unit data is the data content of a relational data table.

[0045] Step 102: Check whether the unit data on which the target data test script depends has completed the data cutover operation.

[0046] In an ideal state, after a conversion operation is performed on a unit data during the data cutover process, there is a corresponding data test script for the unit data to verify whether the cutover operation performed on the unit data is correct. To this end, this step is to detect whether the unit data on which each acquired data test script depends has completed the data cutover operation. When it is detected that the unit data on which the target data test script depends has completed the data cutover operation, step 102 is executed. On the contrary, when the unit data on which the target data test script depends has not completed the data cutover operation, the unfinished unit data is continuously tested until it is completed and step 102 is executed.

[0047] It should be noted that the data test script is a verification and audit performed on the data content after cutover and the relationship between the data.

[0048] Step 103: Execute the target data test script.

[0049] The target data test script in this step is not limited in its specific execution mode when it is executed. Since the embodiment of the present invention can be embedded in the data cutover process, when there are multiple executable target data test scripts, their specific execution order can rely on the task scheduling tool in the data cutover to manage the execution order of the target data test scripts.

[0050] The execution results of the target data test script are divided into test pass and test fail. A test pass indicates that there is no problem with the cutover operation performed by the unit data on which it depends, while a test failure indicates that there is a problem with the cutover operation. Furthermore, when the test fails, the problematic data content in the unit data can be recorded according to the problems tested, and corresponding error information can be generated based on the problematic data.

[0051] The above two steps are the execution steps for the data test script. In actual applications, the number of data test scripts is related to the number of unit data, and the cutover operation of unit data is not completely executed concurrently, but has a certain execution order. Therefore, by detecting the unit data that has completed the cutover operation and sequentially executing the corresponding data test scripts, the unit data can be quickly verified and audited at the data level to improve the audit efficiency.

[0052] Step 104: Check whether the data test script that the target functional module test script depends on has passed the test.

[0053] In this step, the function module test script is a test script set for the specific application scenario of the system and is used to verify whether the cutover data has relevant functions in the specified application scenario.

[0054] The execution of the functional module test script needs to rely on the execution result of the data test script in the previous step. Generally, according to the division of application scenarios, a functional module test script depends on at least multiple unit data according to its functional requirements. Therefore, the functional module test script can only be executed after the unit data it depends on completes the data cutover operation and the corresponding data test script passes, that is, execute step 105.

[0055] It can be seen that in this embodiment, the audit process of heterogeneous data is divided into at least two layers of verification, one is the data layer: it is a verification of data content and association relationships, and the other is the functional module layer: it is a functional verification corresponding to the application scenario, that is, to verify the functions of the system data. The verification and audit of the functional module layer also needs to be performed after the data layer verification is passed, that is, when it is determined that there is no problem with the cutover of the unit data, the accuracy of the cutover of the data at the application level is further audited.

[0056] Step 105: Execute the target function module test script.

[0057] The specific execution process of the target functional module test script is similar to the execution process of the data test script in step 102, and its specific execution order can also be executed by the task scheduling tool in the data cutover solution. In this regard, this step will not be repeated.

[0058] Step 106: Determine the completion quality of the data cutover operation according to the execution result of the functional module test script.

[0059] Data audit is to verify the quality of data cutover operation and find problematic data. This step is to evaluate the quality of data cutover performed by system data migration based on the execution results of the above-mentioned functional module test scripts and data test scripts. In this regard, the evaluation results in this step can be output and displayed in the form of an evaluation report, in which the execution results of each test script are recorded. Evaluation indicators can also be customized for the above-mentioned test schemes, and the values ​​of the customized evaluation indicators are counted to evaluate the verification effect of the above-mentioned audit process on all cutover data.

[0060] Through the description of the above embodiments, it can be known that the data audit method based on data cutover provided by the embodiment of the present invention is applied in the process of cutover of heterogeneous data, and the data test script that depends on the unit data of the completed cutover operation is executed, and after the data test script passes the test, the corresponding functional module test script is further executed, so as to realize the implementation scheme of multi-level audit of the cutover data. In addition, the execution of the data audit scheme is automatically triggered based on the execution status of the unit data cutover operation in the data cutover process, which can be applied to various existing data cutover schemes. At the same time, under the premise that the provided test script is comprehensive, the data that has completed the cutover can be comprehensively audited and verified to ensure the consistency, accuracy, integrity and availability of the data, thereby improving the execution efficiency and accuracy of the data cutover process.

[0061] Further, for Figure 2 The data audit method based on data cutover shown in FIG. 1 is described in more detail in the following embodiment. In this embodiment, the data test script and the function module test script are further divided. The specific audit process is as follows: Figure 3 As shown, including:

[0062] Step 201: Obtain data test scripts and function module test scripts.

[0063] Among them, data test scripts and functional module test scripts are scripts edited by data engineers according to data cutover rules and verification requirements. Their respective functions have been Figure 2 In practical applications, this step can be a test script uploaded by a data engineer through a visual interface, or a test script edited online by a data engineer in a computing medium for performing data auditing according to its execution environment, and the computing medium is not limited to a database, application, or big data computing platform.

[0064] In this embodiment, the data test script and the functional module test script are further refined, wherein the data test script can be specifically divided into a unit test script and an integration test script, wherein a unit test script refers to a script for performing data testing on the content of unit data, and generally has a one-to-one correspondence with the unit data. Of course, when the unit data structure is the same and the required test content is consistent, a unit test script may also correspond to multiple unit data; and an integration test script is a script for performing data testing based on multiple unit data corresponding to system functions, that is, the integration test script generally corresponds to multiple unit data, and accordingly, an integration test script will have a correspondence with multiple unit test scripts.

[0065] The functional module test script in this embodiment can be further refined into test scripts edited based on different preset test types, wherein the preset test type includes at least one of white box testing, black box testing, and core indicator re-inspection testing.

[0066] According to the above division, the structure of the data audit solution implemented in this embodiment can be expressed as follows: Figure 4 In the architecture shown, the most basic layer is the unit test script, on the basis of which the integration test script is executed. These two layers are based on data audit verification, and on the basis of data audit verification, the functional module test script is further executed.

[0067] Specifically, the unit test script is mainly based on the data cutover rules, such as the total number of data items, the total number of item categories, the old and new comparison of the total amount of each item, the verification of non-empty state constraints, enumeration constraints, etc.; the integration test script is mainly based on the verification of data associations in the new system, such as the foreign key association between the main table and the sub-table, the existence of sufficient condition records, the existence of necessary condition records, the existence of unique records, etc. It can also include consistency verification of upstream and downstream redundancy in the system itself, and inspection and verification of whether the performance of the system in the state machine flow meets expectations.

[0068] As for the functional module test scripts, the white box test can also be divided into test scripts for data cutover solutions. For example, for the data cutover solution with cold and hot data separation, it is necessary to build test cases for the cold and hot boundary points; for the full + incremental data cutover solution, it is necessary to design test cases for the full data and incremental data respectively. And test scripts for data cutover rules for implementing process editing of different functions. For example: in the e-commerce scenario, different product channels have different product model cutover rules in the old and new systems, and different test cases need to be formulated for this. For another example, in the government scenario, different attributes of the person handling the document have different model design and conversion rules in the old and new systems, and different test cases need to be formulated for this.

[0069] For black box testing, it can be divided into test scripts for functional entities, test scripts for functional implementation processes, and test scripts for grayscale traffic. Among them, the inspection of functional entities often determines whether the data seen at the functional module level is normal. Since in the process of data cutover, a complex system can be considered a chaos project, the comparison of the new and old systems of functional entities obtained by simulating user requests outside the project is often more reliable. For example, in some client systems with browser and server structures, the client's restful API is reused to complete the comparison of the new and old functional entities and perform traversal testing. For another example, in some client systems with microservice architectures, the service layer interface is reused to complete the comparison of the new and old service functional entities and perform traversal testing. In addition, the function implementation process test is the key to a system being able to run smoothly and steadily during the cutover and cutover re-security period. Since there is a large amount of intermediate functional data in the data cutover process that cannot be pushed to the final state, a complete key function combing is performed, so it is even more necessary to design a one-to-one cutover test case. For example, the government affairs handling system tests all major types of handling that rely on the cutover data. For another example, the e-commerce system tests all major transaction processes that rely on the cutover data. The above tests on functional entities and functional implementation processes represent the basic test requirements for system function cutover and data cutover, but it is not easy to easily enumerate all the functions of a complex system. At this time, introducing real functional data traffic and performing double-write verification of grayscale traffic will be a more feasible approach. For example, in a partial cutover solution that can be double-written, the accuracy of the cutover data can be verified by collecting grayscale traffic of functional data for a longer period of time. For another example, in a partial quasi-real-time cutover solution, the accuracy of the cutover data can be verified by collecting double-writes of functional data consistency for a longer period of time.

[0070] For the core indicator re-inspection test, it is a test to re-inspect the functions of the core indicators of the actual operation of different customers. For example, the trading system often focuses on the assessment of funds and inventory. Therefore, the customer's inventory counting function code can be reused to compare the new and old systems before and after the cutover, and the fund settlement code can be used to compare the new and old systems before and after the cutover.

[0071] The above detailed description of the specific division and content of data test scripts and functional module test scripts is given, and according to Figure 4 As shown in the architecture diagram, the data audit performed by the embodiment of the present invention is based on the premise that the test scripts involved in each layer of the framework can be obtained. Therefore, when this step is executed, in addition to obtaining the test script, it is also necessary to identify the type of the obtained test script to ensure that the received test script can be formed as follows: Figure 4The audit framework shown in the figure, therefore, in the process of executing this step, it is also necessary to execute the step of determining whether the test script can be used to perform data audit operations, the specific operations are as follows:

[0072] Determine whether the acquired data test script includes a unit test script and an integration test script.

[0073] If the data test script includes these two test scripts, further determine whether the received functional module test script includes a test script of a preset test type, wherein the preset test type includes at least one of a white box test, a black box test, and a core indicator recheck test;

[0074] If it contains at least one functional module test script, it is determined that the obtained test script can perform subsequent data audit operations. On the contrary, if the above judgment on the data test script and the functional module test script is negative, a prompt message needs to be generated to inform the data engineering personnel that the corresponding test script needs to be supplemented. It should be noted that there is no logical order relationship between the above judgment steps for the data test script and the functional module test script. The purpose is to determine that the types of the obtained test scripts are comprehensive and sufficient to perform a comprehensive verification and audit of the cutover data.

[0075] Step 202: In the process of cutting over the source data into the target data, determine the unit data required to perform the data cutting over operation.

[0076] Step 203: Check whether the unit data that the unit test script depends on has completed the data cutover operation.

[0077] Step 204: When the unit data on which the unit test script depends completes the data cutover operation, the unit test script is executed.

[0078] Step 205: Determine whether the unit test script has passed the test based on the correspondence between the target integration test script and the unit test script.

[0079] This step is based on the execution result of the unit test script to detect whether the unit test script that the integrated test script depends on has passed the test. If it has passed, step 205 is executed. If it has not passed, an error message is generated to record the data content corresponding to the error.

[0080] Step 206: Execute the target integration test script.

[0081] Step 207: Check whether the data test script that the target functional module test script depends on has passed the test.

[0082] The data test script in this step is specifically an integration test script. Since the integration test script has a functional domain to which it belongs, the associated integration test script can be determined according to the functional domain to which the target functional module test script belongs, and then it can be detected whether these integration test scripts have passed the test. If the test passes, step 207 can be further executed. Otherwise, if there is an integration test script that has not passed the test, an error message will be generated accordingly, and the unit data and data content corresponding to the error will be recorded.

[0083] Step 208: Execute the target functional module test script.

[0084] In another preferred embodiment of the present invention, since different functional module test scripts are involved in different specific functional fields, the corresponding integrated test scripts are also not fixed. Therefore, it is difficult to quickly obtain the dependency relationship between the functional module test script and the integrated test script. For this reason, the timing of executing the functional module test script in this preferred embodiment can be after all data cutover operations belonging to the system are completed, and when the integrated test script is tested and passed, the functional module test script is uniformly executed. That is to say, in this preferred embodiment, the dependency relationship between the functional module test script and the data test script is no longer concerned, and the execution order is to execute the data test script first, that is, execute the unit test script and the integrated test script. After completing all the integrated test scripts, if it is determined that the data test scripts are all passed, the functional module test script is further executed. When the number of functional module test scripts is multiple, it is further necessary to schedule the execution order of the functional module test script based on the task scheduling tool.

[0085] Step 209: Determine the completion quality of the data cutover operation according to the execution result of the functional module test script.

[0086] The specific execution content of this step includes: recording the test results of the target data test script and the target functional module test script. The test results are recorded in the above steps. Here, only the test results need to be obtained; at the same time, the indicator data used to evaluate the data audit effect is counted according to the preset rules; finally, the audit report is generated according to the test results and indicator data. The audit report can be used to display and evaluate the completion quality of the data cutover operation.

[0087] The test results in the audit report mainly reflect the evaluation of the audit process. Data engineers can use this test result to understand which tests were conducted during the data cutover process and what the test results were. The indicator data mainly reflects the scope of the audit of the cutover data. Through this indicator data, you can understand which cutover data has been audited.

[0088] For indicator data, the embodiment of the present invention exemplarily proposes three indicators, namely, the number of rules per unit data, the coverage rate of unit data, and the field coverage rate, wherein the field refers to the field involved in the data content in the unit data.

[0089] The number of unit data rules refers to the number of scripts that are used to test the unit data. The scripts include unit test scripts, integration test scripts, and functional module test scripts. This indicator can provide overall feedback on the audit quality of the target unit data. That is, the larger the number, the higher the number and dimension of the unit data being verified, and the higher the corresponding data cutover quality.

[0090] The unit data coverage rate refers to the ratio of the number of unit data under test to the total number of unit data, where the unit data under test refers to the number of rules whose index value is non-zero. This coverage rate is the most basic quality assurance indicator for cutover work. Once it is non-100%, it means that the cutover data corresponding to some cutover rules are completely not covered by the test, resulting in missed tests.

[0091] Field coverage is the ratio of the number of fields tested in the unit data to the total number of fields in the unit data. The level of this indicator represents the details of the cutover audit work in the field dimension. Since the cutover audit work is designed based on the field content, this field is often low in real project practice and difficult to count due to the different project investment and field-level audit scheme design difficulties. However, it is also the most realistic feedback on the implementation of project data audit work. That is, the field coverage is an audit indicator obtained by further refining the unit data coverage.

[0092] Through the above Figure 3 The various detailed steps of the embodiment shown, and Figure 4 As can be seen from the test architecture shown, the data audit solution proposed in the present invention sets up multi-level test scripts to verify the cutover data during the cutover of heterogeneous data. At the same time, it specifically defines the specific content included in each layer of test scripts and the logical relationship between the execution of test scripts between layers.

[0093] Due to the complexity of data cutover rules, a reasonable data audit sequence can greatly reduce the troubleshooting time caused by data cutover. In this embodiment, unit testing of unit data is performed first, which can quickly discover problems at the unit data level, such as data loss, enumeration value errors, default value errors, etc. The subsequent integration test based on system functions can quickly discover functional data problems, such as primary and foreign key correspondence errors, missing necessary condition records, etc. And based on the completed unit test in the previous step, it can be immediately located that there is a problem with the association relationship between unit data, thereby improving the efficiency of problem location. The subsequent functional module test can eliminate a large number of problems on the data side, and the main direction of problem troubleshooting can be turned to the problem of matching the functions in the new system with the data of the old system. Once the above-mentioned data audit test is skipped, the location of the problem will need to be checked one by one from top to bottom, which often involves functional module groups in several fields in actual applications, resulting in extremely low troubleshooting efficiency. It can be seen that the data audit solution using this architecture can improve the troubleshooting efficiency of data verification problems, thereby improving the consistency, accuracy, integrity and availability of the delivered cutover data.

[0094] Furthermore, as a response to the above Figure 2 , 3 The implementation of the data audit method based on data cutover is shown in the figure. The embodiment of the present invention provides a data audit device based on data cutover, which is used to audit and verify the cutover operation of heterogeneous data to improve the quality of data cutover. For the sake of ease of reading, this device embodiment will no longer repeat the details of the above method embodiments one by one, but it should be clear that the device in this embodiment can correspond to all the contents of the above method embodiments. The device is as follows Figure 5 As shown, specifically including:

[0095] An acquisition unit 31 is used to determine unit data required for performing a data cutover operation in a process of cutting over source data into target data, wherein the unit data is data of the minimum granularity split based on the association relationship of the source data;

[0096] A first detection unit 32 is used to detect whether the unit data obtained by the acquisition unit 31 on which the target data test script depends has completed the data cutover operation;

[0097] A first testing unit 33, configured to execute the target data testing script when the first detecting unit 32 determines that the unit data completes the data cutover operation;

[0098] A second detection unit 34 is used to detect whether the data test script tested by the first test unit 33, on which the target functional module test script depends, has passed the test;

[0099] A second testing unit 35, configured to execute the target functional module test script when the second detecting unit 34 determines that the data test script test passes;

[0100] The determination unit 36 ​​is used to determine the completion quality of the data cutover operation according to the execution result of the function module test script by the second test unit 35.

[0101] Further, such as Figure 6 As shown, the device also includes:

[0102] A receiving unit 37, used to obtain a data test script and a function module test script;

[0103] The judging unit 38 is used to determine, based on the contents of the data test script and the functional module test script acquired by the receiving unit 37, whether the test script can be used to perform the data audit operation.

[0104] Further, such as Figure 6 As shown, the judging unit 38 includes:

[0105] A first determination module 381 is used to determine whether the data test script includes a unit test script and an integrated test script, wherein the unit test script is a script for performing data test on corresponding unit data, and the integrated test script is a script for performing data test on multiple unit data corresponding to a functional module;

[0106] A second judgment module 382 is used to further judge whether the functional module test script includes a test script of a preset test type when the first judgment module 381 determines that the unit test script and the integration test script are included, and the preset test type includes at least one of a white box test, a black box test, and a core indicator recheck test;

[0107] The determination module 383 is further configured to, when the second determination module 382 determines that there is a test script of a preset test type, determine whether the acquired test script can be used to perform a data audit operation.

[0108] Further, such as Figure 6 As shown, the first testing unit 33 includes:

[0109] A first execution module 331, configured to execute the unit test script when the unit data on which the unit test script depends completes the data cutover operation;

[0110] A judging module 332, configured to judge whether the unit test script executed by the first executing module 331 has passed the test according to the correspondence between the target integration test script and the unit test script;

[0111] The second execution module 333 is used to execute the target integration test script after the judgment module 332 determines that the unit test script has passed the test.

[0112] Further, such as Figure 6 As shown, the second detection unit 34 includes:

[0113] A determination module 341, configured to determine an associated integrated test script according to the function module to which the target function module test script belongs;

[0114] The detection module 342 is used to detect whether the integrated test script determined by the determination module 341 has passed the test.

[0115] Further, such as Figure 6 As shown, the second testing unit 35 includes:

[0116] A judging module 351 is used to judge whether the data cutover operation corresponding to the source data is completed;

[0117] The execution module 352 is used to execute the target function module test script when the judgment module 351 determines that the data cutover operation is completed and the data test script is passed.

[0118] Further, such as Figure 6 As shown, the determining unit 36 ​​includes:

[0119] A recording module 361, used to record the test results of the target data test script and the target function module test script;

[0120] Statistics module 362, used to collect statistics of indicator data for evaluating data auditing effect according to preset rules;

[0121] The generating module 363 is used to generate an audit report according to the test results obtained by the recording module 361 and the indicator data obtained by the statistical module 362, and the audit report is used to evaluate the completion quality of the data cutover operation.

[0122] Furthermore, for the above Figure 2 , 3 The data audit method shown in the embodiment can be implemented as a function in the data cutover service, or can be applied as a separate application or software to the data cutover terminal, which is used to cutover the source data with a first association relationship obtained by the source terminal to the target data with a second association relationship required by the target terminal. The data audit method proposed in the present invention will be executed based on the data audit request triggered during the data cutover process, that is, when the present invention appears as an independent application or service, it is necessary to trigger the data audit process based on the audit request. In this regard, the specific steps of this embodiment are as follows: Figure 7As shown, including:

[0123] Step 301: Obtain data test scripts and functional module test scripts according to data audit requests.

[0124] The data audit request may be triggered by a data cutover service, or may be triggered by a temporary setting by a user at the data cutover end according to application requirements.

[0125] It should be noted that, based on the data audit request, what is triggered is the data audit process, not just obtaining the test script.

[0126] Step 302: In the process of cutting over the source data into the target data, determine the unit data required to perform the data cutting over operation.

[0127] Among them, unit data is the smallest granularity data split based on the association relationship of source data. For detailed description, please refer to Figure 2 The description of step 101 in the illustrated embodiment will not be repeated here.

[0128] Step 303: Check whether the unit data on which the data test script depends has completed the data cutover operation. If completed, execute the data test script.

[0129] Step 304: Check whether the data test script that the function module test script depends on has passed the test. If the test has passed, execute the function module test script.

[0130] Step 305: Generate an audit report corresponding to the data audit request according to the execution result of the functional module test script.

[0131] The above steps 303-305 are the data audit process based on the unit data during the data cutover process. For details, please refer to Figure 2 , 3 The corresponding steps in will not be repeated here.

[0132] In addition, an embodiment of the present invention further provides a processor, the processor is used to run a program, wherein the program executes the above Figure 2 , 3 , A data audit method based on data cutover is provided by any one of the embodiments shown in 7.

[0133] In addition, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the above Figure 2 , 3 , the data audit method based on data cutover as described in any one of the embodiments shown in 7.

[0134] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0135] It is understandable that the related features in the above methods and devices can be referenced to each other. In addition, the "first", "second" and the like in the above embodiments are used to distinguish the embodiments, but do not represent the advantages and disadvantages of the embodiments.

[0136] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0137] The algorithm and display provided herein are not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious to construct the structure required for this type of system. In addition, the present invention is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the description of the above specific language is for disclosing the preferred embodiment of the present invention.

[0138] In addition, the memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0139] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0140] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data cutover-based data auditing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data cutover-based data auditing device generate instructions for implementing the process Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0141] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data audit device based on data cutover to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0142] These computer program instructions may also be loaded onto a computer or other programmable data audit device based on data cutover, so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0143] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0144] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0145] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0146] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0147] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0148] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A data audit method based on data cutover, the method comprising: In the process of cutting over the source data into the target data, determining the unit data required for performing the data cutting over operation, wherein the unit data is the data of the minimum granularity split based on the association relationship of the source data; Detect whether the unit data on which the target data test script depends has completed the data cutover operation, wherein the target data detection script includes: a unit test script and an integration test script, wherein the unit test script is a script for performing data testing on the corresponding unit data, and the integration test script is a script for performing data testing based on multiple unit data corresponding to the functional module; If completed, the target data test script is executed; Check whether the data test script that the target functional module test script depends on has passed the test; If the test passes, the target function module test script is executed; The completion quality of the data cutover operation is determined according to the execution result of the function module test script.

2. The method according to claim 1, characterized in that The method further comprises: Obtain data test scripts and functional module test scripts; Determine, based on the contents of the acquired data test script and functional module test script, whether the test script can be used to perform data audit operations.

3. The method according to claim 2, characterized in that Determine the test scripts that can be used to perform data audit operations based on the contents of the acquired data test scripts and functional module test scripts, including: Determine whether the data test script includes the unit test script and the integration test script; If yes, determine whether the functional module test script includes a test script of a preset test type, wherein the preset test type includes at least one of a white box test, a black box test, and a core indicator recheck test; If so, it is determined that the acquired test script can be used to perform the data audit operation.

4. The method according to claim 3, characterized in that The executing the target data test script includes: When the unit data on which the unit test script depends completes the data cutover operation, executing the unit test script; According to the correspondence between the target integration test script and the unit test script, determine whether the unit test script has passed the test; If passed, the target integration test script is executed.

5. The method according to claim 3, characterized in that: Check whether the data test script that the target functional module test script depends on has passed the test, including: Determine the associated integration test script according to the function module to which the target function module test script belongs; Check whether the integration test script passes the test.

6. The method according to claim 5, characterized in that Executing the target function module test script includes: Determining whether the data cutover operation corresponding to the source data is completed; If completed and the data test script passes the test, execute the target function module test script.

7. The method according to any one of claims 1 to 6, characterized in that Determining the completion quality of the data cutover operation according to the execution result of the functional module test script includes: Record the test results of the target data test script and the target function module test script; Collect statistics of indicators used to evaluate the effectiveness of data audits according to preset rules; An audit report is generated according to the test results and the indicator data, and the audit report is used to evaluate the completion quality of the data cutover operation.

8. The method according to claim 7, characterized in that The indicator data includes the number of rules per unit data, unit data coverage, and field coverage; The number of rules of the unit data is the number of scripts that apply the unit data for testing; The unit data coverage is the ratio of the number of unit data tested to the total number of unit data; The field coverage is the ratio of the number of tested fields in the unit data to the total number of fields in the unit data.

9. A data audit device based on data cutover, the device comprising: An acquisition unit, used to determine unit data required for performing a data cutover operation in a process of cutting over source data into target data, wherein the unit data is data of the minimum granularity split based on an association relationship of the source data; A first detection unit is used to detect whether the unit data on which the target data test script depends has completed the data cutover operation, wherein the target data detection script includes: a unit test script and an integration test script, wherein the unit test script is a script for performing data testing on the corresponding unit data, and the integration test script is a script for performing data testing based on multiple unit data corresponding to the functional module; A first testing unit, configured to execute the target data testing script when the first detection unit determines that the unit data completes the data cutover operation; A second detection unit is used to detect whether the data test script of the first test unit test, on which the target functional module test script depends, has passed the test; A second testing unit, configured to execute the target functional module test script when the second detection unit determines that the data test script has passed the test; The determination unit is used to determine the completion quality of the data cutover operation according to the execution result of the function module test script by the second test unit.

10. A data audit method based on data cutover, the method is applied to a data cutover terminal, the data cutover terminal is used to cutover source data with a first association relationship obtained by a source terminal into target data with a second association relationship required by a target terminal, the method comprising: Obtain data test scripts and functional module test scripts based on data audit requests; In the process of cutting over the source data into the target data, determining the unit data required for performing the data cutting over operation, wherein the unit data is the data of the minimum granularity split based on the association relationship of the source data; Check whether the unit data that the data test script depends on has completed the data cutover operation; If completed, the data test script is executed; Check whether the data test script that the functional module test script depends on has passed the test; If the test passes, the functional module test script is executed; An audit report corresponding to the data audit request is generated according to the execution result of the function module test script, and the audit report is used to evaluate the completion quality of the data cutover operation.

11. A processor, characterized in that: The processor is used to run a program, wherein the program, when running, executes the data audit method based on data cutover according to any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, wherein when the computer program is running, it controls the device where the computer-readable storage medium is located to execute the data audit method based on data cutover according to any one of claims 1 to 8.

Images