A data processing task verification method, device, equipment and storage medium

Abstract

The present invention discloses a verification method, device, equipment and storage medium for data processing tasks, the method comprising: obtaining source full data of a source server and source incremental data generated thereon; performing a data processing task on the source full data and the source incremental data according to the source data processing logic of the source server to obtain source result data; performing a merge data processing task on the source full data and the source incremental data according to the target data processing logic in the target server to obtain target result data; and verifying whether the two result data are consistent. In the present invention, by making the basic data and change data of the target server and the source server consistent, the data results are calculated and verified respectively, which refines the verification method, reduces the possibility of data errors, and makes the verification method applicable to more scenarios.

Description

Technical Field

[0001] The present invention relates to the field of computer information technology, and in particular to a verification method, device, equipment and storage medium for a data processing task. Background Art

[0002] With the development of computer technology and the widespread use of office software, document databases are collections of data stored in a specific manner, shared with multiple users, and independent of the application. Typically, users can choose a database that suits their business needs. As their business expands and upgrades, they may choose to store data in other databases. For example, many companies have used ORACLE for years to create reports, but are now choosing HIVE databases. Because ORACLE and HIVE databases have significant differences in their storage methods, the two databases have different data processing logic, and the underlying data they synchronize will differ due to differences in synchronization time.

[0003] To prevent the data in the new database from impacting business operations, data in the target database (such as the HIVE database) must be verified to ensure consistency across different databases. Existing technologies verify the target database's data by directly comparing the resulting data from the two databases. This crude verification method cannot meet the more detailed verification requirements in various scenarios. Summary of the Invention

[0004] The present invention provides a verification method, device, equipment and storage medium for data processing tasks to solve the problem that the data verification methods in the prior art are rough and cannot meet the more detailed data verification requirements in various scenarios.

[0005] A verification method for a data processing task, comprising:

[0006] Obtaining the first-layer source full data of the source server and the first-layer source incremental data generated based on the first-layer source full data;

[0007] After performing data processing tasks on the first-layer source full data and the first-layer source incremental data using the first-layer source data processing logic of the source server, first-layer result data is obtained;

[0008] Performing a data processing task of merging the first-layer source full data and the first-layer source incremental data using the first-layer target data processing logic of the target server to obtain the first-layer target result data;

[0009] Verify whether the source result data of the first layer is consistent with the target result data of the first layer.

[0010] A verification device for a data processing task, comprising:

[0011] an acquisition module for acquiring the first-layer source full data of a source server and the first-layer source incremental data generated on the basis of the first-layer source full data, as well as acquiring the first-layer source result data and the first-layer target result data, wherein the first-layer source result data is obtained by executing a data processing task on the first-layer source full data and the first-layer source incremental data using the first-layer source data processing logic of the source server; and the first-layer target result data is obtained by executing a data processing task on the first-layer source full data and the first-layer incremental data using the first-layer target data processing logic of the target server;

[0012] An execution module, configured to execute the first layer target data processing logic of the source server and obtain the first layer source result data; and configured to execute the first layer target data processing logic of the target server and obtain the first layer target result data;

[0013] The verification module is used to verify whether the source result data of the first layer is consistent with the target result data of the first layer.

[0014] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the verification method for the above-mentioned data processing task when executing the computer program.

[0015] A computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the verification method for the above-mentioned data processing task.

[0016] The verification method, device, computer equipment and storage medium for the above-mentioned data processing tasks, by having the target server and the source server both use the source full data and source incremental data, try to make the basic data and change data of the target server and the source server completely consistent, and then calculate the data results separately and verify them, further refine the verification method, reduce the possibility of data errors, and facilitate the subsequent quick determination of the accuracy of the target server data processing logic, thereby making the verification method applicable to more scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0018] Figure 1 This is a schematic diagram of an application environment of a verification method for a data processing task in one embodiment of the present invention;

[0019] Figure 2 is a flow chart of a verification method for a data processing task in one embodiment of the present invention;

[0020] Figure 3 4 is a flowchart of the implementation of step S41 of the verification method for a data processing task in one embodiment of the present invention;

[0021] Figure 4 4 is a flowchart of the implementation of step S46 of the verification method for a data processing task in one embodiment of the present invention;

[0022] Figure 5 1 is a flowchart of the implementation of step S20 of the verification method for a data processing task in one embodiment of the present invention;

[0023] Figure 6 is a schematic diagram of a verification device for a data processing task in one embodiment of the present invention;

[0024] Figure 7 FIG. 1 is a schematic diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The present invention provides a verification method for data processing tasks, which is applied to a data verification system. The data verification system may include a source server and a target server. The execution subject of the verification method may be a database management server independent of the source server and the target server, and may be a source server or a target server.

[0026] In an embodiment of the present invention, the execution subject can be illustrated by taking the target server as an example. In the first possible application scenario: the data processing logic of the source server is simulated in the target server, and the data processing logic of the simulated source server is loaded into the target server, that is, the target server can execute both the data processing logic in the source server and the data processing logic in the target server. In a second possible application scenario, the target server and the source server are two independent devices, and the target server receives the data obtained by the source server according to its own data processing logic. In this embodiment, the execution subject of the verification method takes the target server as an example, and is illustrated by taking the above-mentioned second possible application scenario as an example.

[0027] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0028] The data processing task verification method provided by the embodiment of the present invention can be applied in the following situations: Figure 1 In an application environment, a target server communicates with a source server through a network, and operations such as acquiring, processing, verifying, and storing data are performed by the target server and the source server.

[0029] In one embodiment, if Figure 2 As shown, a verification method for a data processing task is provided, comprising:

[0030] S10: Obtain the first-layer source full data of the source server and the first-layer source incremental data generated based on the first-layer source full data.

[0031] The target server receives the first layer of source data synchronized by the source server, ensuring that the lowest level of data in the target server is consistent with the lowest level of data in the source server. This is to ensure consistency of the basic data for data processing.

[0032] It should be noted that the full data in this embodiment refers to the basic data at the bottom layer of the database's data processing logic. The source full data can be all the data in the source database, or part of the data in the source database, or a data table. The first-level source incremental data refers to the changed data generated based on the first-level source full data.

[0033] For example, the source incremental data of the first layer is a data table, which includes vertical columns and horizontal rows. For example, the vertical columns include four attribute parameters, which are: name, address, telephone number, and age. The source full data of the first layer can be shown in Table 1 below:

[0034] Name address Telephone age Zhang San AAAAA 15012345678 31 Li Si BBBBB 15112345678 32 Zhao Wu CCCCC 15212345678 33 Wang Er DDDDD 15312345678 34

[0035] Table 1

[0036] It should be noted that Table 1 above is merely an example for convenience of explanation and does not constitute a limitative description of this application. The source incremental data can be the change data corresponding to at least one attribute parameter. For example, in the source full data, Zhang San's phone number is "15012345678", and the incremental data is: "18012345678"; or, in another possible scenario, the source incremental data is another data table, for example, the source incremental data is shown in Table 2 below:

[0037] Name address Telephone age Yang Liu EEEEE 15412345678 35

[0038] Table 2

[0039] It should be noted that the above Table 2 is only an exemplary description of the incremental data, and the exemplary description of the incremental data does not constitute a limitative description of the present application. Depending on different services, the first-layer source incremental data is different.

[0040] S20: After performing data processing tasks on the first-layer source full data and the first-layer source incremental data using the first-layer source data processing logic of the source server, the first-layer source result data is obtained.

[0041] It should be noted that the database processes data through data processing logic to obtain result data. In the embodiment of this proposal, each layer in the source server has corresponding data processing logic for this execution layer, and the target server also has corresponding data processing logic for this execution layer. Each layer in the source server (or target server) refers to a sub-node of the processing logic from basic data to result data. For example, if the source server has N layers of data, the first layer of the source server has the first layer of source data processing logic, the first layer of the source server has the first layer of source data processing logic, and the Nth layer (N is greater than or equal to 1) of the source server has the Nth layer of source data processing logic.

[0042] The source server uses the first-layer source full data as the basic data for the first-layer data processing tasks. The source server receives write, modify, delete, and other operations on the first-layer source full data, and obtains the first-layer source incremental data. Then, the source server processes the source full data and source incremental data using the first-layer data processing logic, executes the first-layer data processing tasks, and obtains the first-layer source data results. For example, the source server performs data processing tasks on the source full data shown in Table 1 and the source incremental data shown in Table 2 according to the first-layer data processing logic, and obtains the result data shown in Table 3.

[0043] Name address Telephone age Zhang San AAAAA 15012345678 31 Li Si BBBBB 15112345678 32 Zhao Wu CCCCC 15212345678 33 Wang Er DDDDD 15312345678 34 Yang Liu EEEEE 15412345678 35

[0044] Table 3

[0045] The source data processing logic of the first layer can be: matching the attribute parameters of the source full data and the source incremental data, for example, matching the "name", "address", "telephone", and "age" in the source full data with the "name", "address", "telephone", and "age" in the source incremental data, merging the data corresponding to the matching attribute parameters to obtain the source result data of the first layer.

[0046] S30: Obtain the target data processing logic of the first layer in the target server to execute a data processing task of merging the source full data of the first layer and the source incremental data of the first layer to obtain the target result data of the first layer.

[0047] The target server also uses the first-layer source full data (for example, as shown in Table 1) as the basic data for data processing tasks to ensure that the underlying data for data processing by the target server and the source server are consistent, and uses the first-layer source incremental data (for example, as shown in Table 2) as the change data of the source full data, so that the change data of the source full data performing data processing tasks and the change data of the number of people performing data processing on the target server are consistent.

[0048] The target server performs data processing tasks on the source full data and source incremental data using the first-level target data processing logic to obtain the first-level target result data.

[0049] For example, the target data processing logic of the first layer can be: match each first attribute parameter in the source incremental data with the attribute parameter set in the source full data to obtain the target attribute parameter that matches the first attribute parameter, and then insert the parameter value corresponding to the first attribute parameter into the data corresponding to the target attribute parameter in the source full data. For example, in the source incremental data, match the "name" in the source incremental data with the attribute parameter in the source full data, then determine that the "name" in the source full data is the target attribute parameter, and then insert the parameter value "Yang Liu" corresponding to the "name" column in the source incremental data into the column corresponding to "name" in the source full data.

[0050] S40: Verify whether the source result data of the first layer is consistent with the target result data of the first layer.

[0051] Both the source result data and the target result data can be table data, with rows and columns as coordinates, and the coordinate value of each coordinate is traversed. For example, the data in the first row and first column of the source result data is "name", and the data in the first row and first column of the target result data is "name"; the data in the second row and first column of the source result data is "Zhang San", and the data in the second row and first column of the target result data is "Zhang San", and so on, compare whether the source result data of the first layer and the target result data of the first layer are consistent.

[0052] If the source result data of the first layer is consistent with the target result data of the first layer, S41 is executed; if the source result data of the first layer is inconsistent with the target result data of the first layer, S46 is executed.

[0053] In this embodiment, by having both the target server and the source server use the source full data and source incremental data, the basic data and change data of the target server and the source server are made completely consistent as much as possible, and then the data results are calculated and verified separately, the verification method is refined, and the possibility of data errors is reduced, so that the accuracy of the target server data processing logic can be quickly determined subsequently, making the verification method applicable to more scenarios.

[0054] In one embodiment, when it is necessary to determine the next operation based on the verification result, such as Figure 2 As shown, in step S40, the source result data of the first layer is checked to see if it is consistent with the target result data of the first layer. The following operations are performed according to the result, including:

[0055] S41: If the source result data of the first layer is consistent with the target result data of the first layer, it is determined that the data processing task of the first layer of the target server is executed accurately, and the execution accuracy of the data processing task of the second layer is verified.

[0056] S46: If the source result data of the first layer is inconsistent with the target result data of the first layer, a prompt message is generated.

[0057] This prompt message is used to indicate the cause of the query problem.

[0058] Since the underlying data for executing data processing tasks in the target server is consistent with the underlying data in the source server, and the change data for executing data processing tasks is also consistent with the change data in the source server, if the source result data of the first layer is consistent with the target result data of the first layer, then it can be determined that the data processing tasks of the first layer of the target server are executed accurately, that is, the data processing logic of the first layer in the target server is accurate.

[0059] It should be noted that the target data processing logic of the first layer may be the same as the source data processing logic of the first layer, or it may be different from the source data processing logic of the first layer. If the source data processing logic of the first layer is the same as the target data processing logic of the first layer, then the source result data of the first layer and the target result data of the first layer must be the same; when the source data processing logic of the first layer and the target data processing logic of the first layer are different, if the source result data and the target result data are the same, it means that although the data processing logic of the first layer in the target server is different from the source processing logic of the first layer, the target data processing logic of the first layer performs the data processing task accurately, which is equivalent to different processing methods can obtain the same result, and this situation is acceptable.

[0060] In the embodiment of this case, the first layer is the bottom layer of the data processing logic, and verification is performed layer by layer from the first layer to the upper layer. In this embodiment, only two layers are used as an example for illustration. In actual applications, it may include three layers, four layers, and so on, until the accuracy of the top-level data processing logic is verified.

[0061] Furthermore, in the embodiment of the present invention, the prompt information generated includes: extracting a difference value between the source result data of the first layer and the target result data of the first layer; and generating the prompt information according to the difference value.

[0062] This prompt information is used to indicate that the first-layer data processing logic in the target server is inaccurate, prompting the staff to further find the cause of the problem. Since only the first-layer data processing logic is currently involved, it can be clearly determined that the problem occurs in the first-layer data processing logic, saving time and effort in finding the problem.

[0063] In one embodiment, if the source result data of the first layer and the target result data of the first layer are verified to be consistent, such as Figure 3 As shown, the accuracy of the data processing task execution on the second layer is verified, which specifically includes the following steps:

[0064] S42: Acquire the second-layer source incremental data, where the second-layer source incremental data is incremental data generated based on the first-layer source result data.

[0065] After the source server generates the second-layer source incremental data, the second-layer source incremental data is synchronized to the target server. The target server uses the first-layer target result data as the basic data for the second-layer data processing and the second-layer source incremental data as the change data, thereby ensuring that when verifying the second-layer target data processing logic, the second-layer basic data in the target server is consistent with the second-layer basic data in the source server, and the second-layer change data in the target server is consistent with the second-layer change data in the source server.

[0066] S43: Using the second-layer source data processing logic of the source server to perform data processing tasks on the first-layer source result data and the second-layer source incremental data, to obtain the second-layer source result data.

[0067] The source server uses the second-layer source data processing logic to perform a merge processing task on the first-layer source result data and the second-layer source incremental data, obtaining the second-layer source result data. This second-layer source result data is then synchronized to the target server, which receives the second-layer source result data sent by the source server. Specifically, the principle by which the source server performs this merge processing task is similar to that of the first-layer data processing task and will not be elaborated here.

[0068] S44: The target result data of the first layer and the source full data of the second layer are merged and processed by the second layer target data processing logic of the target server to obtain the target result data of the second layer.

[0069] The target server uses the target result data of the first layer as the basic data of the second layer data processing task, and uses the received source incremental data of the second layer as the change data of the second layer, and then uses the second layer data processing logic to perform the merging processing task of the target result data of the first layer and the source incremental data of the second layer to obtain the target result data of the second layer.

[0070] S45: Verify whether the source result data of the second layer is consistent with the target result data of the second layer.

[0071] Further, if the source result data of the second layer is consistent with the target result data of the second layer, then S451 is executed; if the source result data of the second layer is inconsistent with the target result data of the second layer, then S452 is executed. Figure 3 As shown:

[0072] S451: If the source result data of the second layer is consistent with the target result data of the second layer, it is determined that the data processing task of the second layer of the target server is executed accurately, so as to verify the target result data of the next layer (third layer) with the source data result of the next layer.

[0073] S452: If the source result data of the second layer is inconsistent with the target result data of the second layer, a prompt message is generated.

[0074] If the verification results are consistent, it can be easily determined that the second-layer data execution logic of the target server is accurate, and the accuracy of the execution of the next-layer data processing tasks can be verified. If the verification results are inconsistent, the second-layer data execution logic error can be quickly known for subsequent correction.

[0075] Furthermore, if Figure 4 As shown, the generated prompt information includes:

[0076] 401: Extract the difference between the source result data of the second layer and the target result data of the second layer;

[0077] 402: Generate prompt information based on the difference value.

[0078] This prompt information is used to indicate that the second-layer data processing logic in the target server is inaccurate, prompting the staff to further find the cause of the problem. Since only the second-layer data processing logic is currently involved, it can be clearly determined that the problem occurs in the second-layer data processing logic, saving time and effort in finding the problem.

[0079] In one embodiment, in step S30, the first-layer target data processing logic of the target server is used to perform a merge data processing operation on the first-layer source full data and the first-layer incremental data to obtain the first-layer target result data, which specifically includes the following steps:

[0080] S311: Identify a first attribute parameter set in the source full data of the first layer, where each first attribute parameter in the first attribute parameter set corresponds to the first data, and a second attribute parameter set in the incremental data of the first layer, where each second attribute parameter in the second attribute parameter set corresponds to the second data.

[0081] For example, the full source data of the first layer is shown in Table 1. The first attribute parameters included in the first attribute parameter set are: name, address, telephone number and age. The first data corresponding to the name are "Zhang San", "Li Si", "Zhao Wu", etc.; the second attribute parameter in the incremental data of the first layer is "address", and the second data corresponding to the "address" is AAAAA, ABCDE, CCCCC, DDDDD, EEEEE.

[0082] S321: If the first attribute parameter set includes the second attribute parameter set, determine the target attribute parameter included in the intersection of the first attribute parameter set and the second attribute parameter set.

[0083] The first attribute parameter set includes: name, address, telephone number and age. The second attribute parameter set is "address". The first attribute parameter set includes the second attribute parameter set. The target attribute parameter included in the intersection of the first attribute parameter set and the second attribute parameter set is determined to be "address".

[0084] S331: Determine the first data to be executed of the data processing task of the first layer, the first data to be executed includes the second data corresponding to the target attribute parameter, and the first data corresponding to the first attribute parameter in the third attribute parameter set; the third attribute parameter set is the attribute parameter set in the first attribute parameter set except the target attribute parameter.

[0085] For example, the first attribute parameter set includes name, address, phone number, and age; the target attribute parameter set includes address; and the third attribute parameter set includes name, phone number, and age. Determine the first data corresponding to the attribute parameter "name," the first data corresponding to the attribute parameter "phone number," the first data corresponding to the attribute parameter "age," and the second data corresponding to the attribute parameter "address."

[0086] S341: Execute the first-layer data processing logic on the first to-be-executed data according to the first-layer data processing task to obtain the first-layer target result data.

[0087] The first layer data processing logic is executed according to the first layer data processing task. For example, the data processing logic is: merging the first attribute parameter in the third attribute parameter set and its corresponding first data, and the second data corresponding to the target attribute parameter.

[0088] For example, the target result data of the first layer is shown in Table 4 below:

[0089] Name address Telephone age Zhang San AAAAA 15012345678 31 Li Si ABCDE 15112345678 32 Zhao Wu CCCCC 15212345678 33 Wang Er DDDDD 15312345678 34 Yang Liu EEEEE 15412345678 35

[0090] Table 4

[0091] Optionally, in step S30, the first-layer target data processing logic of the target server is used to perform a merge data processing operation on the first-layer source full data and the first-layer incremental data to obtain the first-layer target result data, which specifically includes the following steps:

[0092] S312: Identify a first attribute parameter set in the source full data of the first layer, where each first attribute parameter in the first attribute parameter set corresponds to the first data, and a second attribute parameter set in the incremental data of the first layer, where each second attribute parameter in the second attribute parameter set corresponds to the second data.

[0093] S322: If the first attribute parameter set does not include any second attribute parameter in the second attribute parameter set, determining second to-be-executed data for the data processing task of the first layer, where the second to-be-executed data includes first data corresponding to each first attribute parameter in the first attribute parameter set and second data corresponding to each second attribute parameter in the second attribute parameter set.

[0094] S332: Execute the data processing logic of the first layer on the second to-be-executed data according to the data processing task of the first layer to obtain target result data of the first layer.

[0095] Optionally, if the first attribute parameter set includes some of the second attribute parameters in the second attribute parameter set, first determine the target data parameter set (such as name and address) in the intersection of the first attribute parameter set (such as name, address, telephone number) and the second attribute parameter set (such as name, address, age).

[0096] Determining target attribute parameters and corresponding second data;

[0097] Determine a fourth attribute parameter set (such as telephone number) other than the target attribute parameter set in the first attribute parameter set and the corresponding first data;

[0098] A fifth attribute parameter set (such as age) other than the target attribute parameter set in the second attribute parameter set and corresponding second data are determined.

[0099] The target attribute parameter and the corresponding second data, the fourth attribute parameter set and the corresponding first data, the second attribute parameter and the corresponding second data are merged to obtain the target result data of the first layer.

[0100] In this embodiment, full data and incremental data are defined and classified according to different attribute parameters, and these classified data are merged and processed, which increases the diversity of target result data and further refines the data and data processing steps.

[0101] In one embodiment, if Figure 5 As shown, in step S20, before obtaining the source result data of the first layer, the following steps are also included:

[0102] S501: The source server generates the first layer of source incremental data, synchronizes the first layer of source incremental data to the target server, and the source server records the time when the source incremental data is generated.

[0103] Specifically, when the source server generates source incremental data, a condition is triggered to synchronize the incremental data to the target server, and the source server records the time when the source incremental data is generated.

[0104] S502: The target server receives the source incremental data of the first layer, and records the start time of receiving the source incremental data of the first layer and the end time of receiving the source incremental data of the first layer.

[0105] For example, the target server starts receiving the incremental data at 12:10:00 and ends receiving the incremental data of one layer at 12:15:00. It should be noted that the start and end times are examples. In actual applications, the durations indicated by the start and end times are related to the incremental data and the transmission rate.

[0106] S503: The target server transmits the time information to the source server, where the time information includes the start time and the end time.

[0107] S504: The source server generates the first-layer source result data based on the source incremental data generated between the start time and the end time and the first-layer source full data.

[0108] S505: The source server receives the time information and verifies, based on the time information, whether the first-layer source incremental data sent to the target server is consistent with the source incremental data used to obtain the first-layer source result data.

[0109] Specifically, the source server also generates source incremental data after the end time (such as 12:15:10). The target server sends the time information to the source server, notifying the source server that the source incremental data for calculating the first layer of source result data is the data of changes that occurred between the start time and the end time, so as to avoid inconsistency in the incremental data used by the source server and the target server to calculate the result data.

[0110] In an embodiment of the present application, the target server obtains the source full data and source incremental data of the current execution layer of the source server, then obtains the source result data and target result data of the current execution, and then verifies whether the source result data and the result data are consistent. If the results are consistent, it is determined that the first layer of data processing tasks of the target server is executed accurately, so as to verify the accuracy of the data processing tasks of the next layer; if they are inconsistent, a prompt message is generated, which is used to indicate the cause of the query problem. In the verification process of each layer, if it is found that the data processing logic in the current execution layer is inaccurate, the cause of the problem can be found in time, which saves time and effort, and thus ensures that the same result data can be obtained by executing the data processing tasks with the data processing logic of the two databases.

[0111] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0112] In one embodiment, a verification device for a data processing task is provided, and the verification device for the data processing task corresponds one-to-one to a verification method for a data processing task in the above embodiment. Figure 6 As shown, the verification device for the data processing task includes an acquisition module, an execution module, and a verification module. The detailed description of each functional module is as follows:

[0113] Acquisition module 601 is used to acquire the first-layer source full data of the source server and the first-layer source incremental data generated based on the first-layer source full data, as well as to acquire the first-layer source result data and the first-layer target result data. The first-layer source result data is the result data obtained after the first-layer source data processing logic of the source server performs a data processing task on the first-layer source full data and the first-layer source incremental data; the first-layer target result data is the first-layer target result data obtained after the first-layer target data processing logic of the target server performs a data processing task on the first-layer source full data and the first-layer incremental data.

[0114] The execution module 602 is used to execute the first layer target data processing logic of the source server and obtain the first layer source result data; and is used to execute the first layer target data processing logic of the target server and obtain the first layer target result data.

[0115] The verification module 603 is used to verify whether the source result data of the first layer is consistent with the target result data of the first layer.

[0116] The specific definition of the verification device for data processing tasks can be found in the definition of the verification method for data processing tasks above, and will not be repeated here. The various modules in the verification device for data processing tasks can be implemented in whole or in part through software, hardware, or a combination thereof. The above modules can be embedded in or independent of the processor in the server in hardware form, or can be stored in the memory of the server in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0117] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7As shown. The computer device includes a processor, memory, and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements a verification method for a data processing task.

[0118] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the verification method for the data processing task in the above-mentioned embodiment are implemented, or when the processor executes the computer program, the functions of the modules of the verification device for the data processing task in the above-mentioned embodiment are implemented. To avoid repetition, they are not described here.

[0119] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the verification method for the data processing task in the above embodiment are implemented, or when the computer program is executed by a processor, the functions of each module in the verification device for the data processing task in the above embodiment are implemented. To avoid repetition, they are not described here.

[0120] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0121] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0122] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A verification method for a data processing task, characterized in that: include: Obtaining the first-layer source full data of the source server and the first-layer source incremental data generated based on the first-layer source full data; After performing data processing tasks on the first-layer source full data and the first-layer source incremental data using the first-layer source data processing logic of the source server, the first-layer source result data is obtained; Performing a data processing task of merging the first-layer source full data and the first-layer source incremental data using the first-layer target data processing logic of the target server to obtain the first-layer target result data; Verifying whether the source result data of the first layer is consistent with the target result data of the first layer; Before obtaining the source result data of the first layer, the following steps are included: The source server generates source incremental data of the first layer, synchronizes the source incremental data of the first layer to the target server, and the source server records the time when the incremental data of the first layer is generated; The target server receives the source incremental data of the first layer, and records the start time of receiving the source incremental data of the first layer and the end time of receiving the source incremental data of the first layer; The target server transmits time information to the source server, where the time information includes the start time and the end time; The source server generates the source result data of the first layer according to the source incremental data generated between the start time and the end time and the source full data of the first layer; The source server receives the time information and verifies, based on the time information, whether the source incremental data sent to the target server is consistent with the source incremental data used to obtain the source result data of the first layer; The step of verifying whether the source result data of the first layer is consistent with the target result data of the first layer includes: If the source result data of the first layer is consistent with the target result data of the first layer, it is determined that the data processing task of the first layer of the target server is executed accurately, and the accuracy of the execution of the data processing task of the second layer is verified; If the source result data of the first layer is inconsistent with the target result data of the first layer, a prompt message is generated.

2. The data processing task verification method according to claim 1, characterized in that: Verifying the accuracy of the execution of the second-layer data processing tasks includes: Acquire source incremental data of the second layer, where the source incremental data of the second layer is incremental data generated based on the source result data of the first layer; Performing data processing tasks on the source result data of the first layer and the source incremental data of the second layer using the second layer source data processing logic of the source server to obtain the source result data of the second layer; Merging the target result data of the first layer and the source incremental data of the second layer with the target data processing logic of the second layer of the target server to obtain the target result data of the second layer; Verify whether the source result data of the second layer is consistent with the target result data of the second layer.

3. The data processing task verification method according to claim 2, characterized in that: The verifying whether the source result data of the second layer is consistent with the target result data of the second layer includes: If the source result data of the second layer is consistent with the target result data of the second layer, it is determined that the data processing task of the second layer of the target server is executed accurately, and the accuracy of the execution of the data processing task of the next layer is verified; If the source result data of the second layer is inconsistent with the target result data of the second layer, the prompt information is generated.

4. The data processing task verification method according to claim 1 or 3, characterized in that: The generating prompt information includes: Extracting the difference between the source result data and the target result data of the current execution layer; The prompt information is generated according to the difference value.

5. The data processing task verification method according to any one of claims 1 to 3, characterized in that: The first-layer target data processing logic of the target server is used to execute a data processing task of merging the first-layer source full data and the first-layer source incremental data to obtain the first-layer target result data, including: Identifying a first attribute parameter set in the source full data of the first layer, where each first attribute parameter in the first attribute parameter set corresponds to first data, and a second attribute parameter set in the incremental data of the first layer, where each second attribute parameter in the second attribute parameter set corresponds to second data; If the first attribute parameter set includes the second attribute parameter set, determining the target attribute parameter included in the intersection of the first attribute parameter set and the second attribute parameter set; Determining first data to be executed for the data processing task of the first layer, where the first data to be executed includes second data corresponding to the target attribute parameter and first data corresponding to a first attribute parameter in a third attribute parameter set; the third attribute parameter set is a set of attribute parameters in the first attribute parameter set excluding the target attribute parameter; The first layer of data processing logic is executed on the first to-be-executed data according to the data processing task of the first layer to obtain the target result data of the first layer.

6. A verification device for a data processing task, characterized in that: include: an acquisition module, configured to acquire the first-layer source full data of a source server and the first-layer source incremental data generated based on the first-layer source full data, as well as the first-layer source result data and the first-layer target result data, wherein the first-layer source result data is the first-layer source result data obtained after performing a data processing task on the first-layer source full data and the first-layer source incremental data using the first-layer source data processing logic of the source server; The target result data of the first layer is obtained by performing a data processing task of merging the source full data of the first layer and the incremental data of the first layer using the first layer target data processing logic of the target server; An execution module, configured to execute the first-layer target data processing logic of the source server and obtain the first-layer source result data; Used to execute the first layer target data processing logic of the target server and obtain the first layer target result data; A verification module, configured to verify whether the source result data of the first layer and the target result data of the first layer are consistent; Before obtaining the source result data of the first layer, the following steps are included: The source server generates source incremental data of the first layer, synchronizes the source incremental data of the first layer to the target server, and the source server records the time when the incremental data of the first layer is generated; The target server receives the source incremental data of the first layer, and records the start time of receiving the source incremental data of the first layer and the end time of receiving the source incremental data of the first layer; The target server transmits time information to the source server, where the time information includes the start time and the end time; The source server generates the source result data of the first layer according to the source incremental data generated between the start time and the end time and the source full data of the first layer; The source server receives the time information and verifies, based on the time information, whether the source incremental data sent to the target server is consistent with the source incremental data used to obtain the source result data of the first layer; The step of verifying whether the source result data of the first layer is consistent with the target result data of the first layer includes: If the source result data of the first layer is consistent with the target result data of the first layer, it is determined that the data processing task of the first layer of the target server is executed accurately, and the accuracy of the execution of the data processing task of the second layer is verified; If the source result data of the first layer is inconsistent with the target result data of the first layer, a prompt message is generated.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the verification method for the data processing task according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the verification method for the data processing task as claimed in any one of claims 1 to 5 are implemented.

Images