Big data cluster task transformation verification method, device and system and electronic equipment

Abstract

The invention discloses a big data cluster task transformation verification method, device and system and electronic equipment. The method comprises the following steps of: storing a data source address of each task in a process of running each task in a first computing cluster; wherein the data source address comprises input data of a task and a data source address of first output data corresponding to the input data; based on the data source address of the input data of each task, obtaining second output data of each task obtained by running each task in a second computing cluster; and on the basis of the data source address of the first output data of each task and the second output data, verifying whether task transformation for migrating each task from the first computing cluster to the second computing cluster is successful or not. According to the embodiment of the invention, the technical problem that the data verification efficiency is low after the big data computing cluster is migrated in the related art can be solved.

Description

Technical Field

[0001] The present application relates to the field of cloud technology, and in particular to a method, device, system and electronic device for verifying the transformation of a big data cluster task. Background Art

[0002] During the migration of a big data computing cluster, it is usually necessary to reconstruct the computing task code between the new and old big data computing clusters. After the transformation is completed, the accuracy of the task logic needs to be verified in the new cluster, that is, the input and output data of the same task in the new and old big data computing clusters must be consistent.

[0003] In the related technology, the correctness of the computing task transformation is verified by manually comparing the task calculation results of the new and old clusters before and after the migration. However, this method is limited by manual technical experience and cost, and it is difficult to maintain the consistency of data input content when calculating the new and old cluster tasks. Therefore, the efficiency of data verification of the big data computing cluster after migration is low, and it is difficult to improve the accuracy of the computing task transformation. Summary of the invention

[0004] In view of the above problems, the present application provides a big data cluster task transformation verification method, device, system and electronic device to at least solve the technical problems in the related technology that the efficiency of data verification of the big data computing cluster after migration is low and it is difficult to improve the accuracy of computing task transformation.

[0005] According to a first aspect of an embodiment of the present application, a method for verifying task transformation of a big data cluster is provided, comprising: storing a data source address of each task in the process of running each task in a first computing cluster; wherein the data source address includes a data source address of input data of the task, and a data source address of first output data corresponding to the input data; based on the input data of each task, obtaining second output data of each task obtained by running each task in a second computing cluster; based on the first output data and the second output data of each task, verifying whether the task transformation of migrating each task from the first computing cluster to the second computing cluster is successful.

[0006] According to a second aspect of an embodiment of the present application, a big data cluster task transformation verification device is provided, and the above-mentioned information retrieval optimization device includes: a storage unit, which is used to store the data source address of each task in the process of running each task in the first computing cluster; wherein the data source address includes the input data of the task and the first output data corresponding to the input data; an acquisition unit, which is used to acquire the second output data of each task obtained by running each task in the second computing cluster based on the input data of each task; and a verification unit, which is used to verify whether the task transformation of migrating each task from the first computing cluster to the second computing cluster is successful based on the first output data and the second output data of each task.

[0007] According to a third aspect of an embodiment of the present application, a big data cluster task transformation verification system is provided, comprising a first computing cluster, a second computing cluster, and a big data cluster task transformation verification device; the first computing cluster is used to send the data source address of each task to the big data cluster task transformation verification device during the execution of each task; the second computing cluster is used to execute each task based on the data source address of the input data of each task to obtain the second output data of each task; the big data cluster task transformation verification device is used to store the data source address of each task during the execution of each task in the first computing cluster; wherein the data source address includes the data source address of the input data of the task, and the data source address of the first output data corresponding to the input data; based on the data source address of the input data of each task, the second output data of each task obtained by executing each task in the second computing cluster is obtained; based on the data source address of the first output data of each task and the second output data, verify whether the task transformation of migrating each task from the first computing cluster to the second computing cluster is successful.

[0008] According to a fourth aspect of an embodiment of the present application, there is also provided an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the big data cluster task transformation verification method of the first aspect through the computer program.

[0009] According to the fifth aspect of the embodiment of the present application, a computer-readable storage medium is also provided, in which a computer program is stored, wherein the computer program is configured to execute the big data cluster task transformation verification method of the first aspect mentioned above when running.

[0010] In an embodiment of the present application, a data source address of each task stored in a first computing cluster during the execution of each task is adopted; wherein the data source address includes input data of the task and first output data corresponding to the input data; based on the input data of each task, the second output data of each task obtained by executing each task in a second computing cluster is obtained; based on the first output data and the second output data of each task, a method is used to verify whether the task transformation of migrating each task from the first computing cluster to the second computing cluster is successful; since the present application obtains the data source address of each task in the original computing cluster during the cluster migration process, that is, the data environment of each task in the original computing cluster during execution can be accurately obtained, and the data environment can be used to accurately verify whether the task transformation of each task from the original computing cluster to the target computing cluster is successful; therefore, the present application can not only improve the efficiency of data verification of the big data computing cluster after migration, but also improve the accuracy of computing task transformation, thereby solving the technical problem that the efficiency of data verification of the big data computing cluster after migration is low and it is difficult to improve the accuracy of computing task transformation in the related technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Moreover, the same reference numerals are used throughout the drawings to represent the same components. In the drawings:

[0012] Figure 1 It is a schematic diagram of an application environment of an optional big data cluster task transformation verification method according to an embodiment of the present application;

[0013] Figure 2 is a schematic diagram of an application environment of another optional big data cluster task transformation verification method according to an embodiment of the present application;

[0014] Figure 3 It is a flowchart of an optional big data cluster task transformation verification method according to an embodiment of the present application;

[0015] Figure 4 It is a flowchart of another optional big data cluster task transformation verification method according to an embodiment of the present application;

[0016] Figure 5 It is a flowchart of another optional big data cluster task transformation verification method according to an embodiment of the present application;

[0017] Figure 6It is a flow chart of the data protection phase of an optional big data cluster task transformation verification method according to an embodiment of the present application;

[0018] Figure 7 It is a flowchart of a verification phase of an optional big data cluster task transformation verification method according to an embodiment of the present application;

[0019] Figure 8 is a schematic diagram showing an optional data storage method according to an embodiment of the present application;

[0020] Fig. 9 It is a structural diagram of a big data cluster task transformation verification device provided in an embodiment of the present application;

[0021] Fig.10 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only embodiments of a part of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0024] As an optional implementation, the above data processing method can be applied to, but is not limited to, Figure 1In the application environment shown. The application environment may include but is not limited to: a task transformation verification device 102 for human-computer interaction with a user, a first computing cluster 110, and a second computing cluster 116. An application client with big data cluster task transformation verification is running in the task transformation verification device 102. The above-mentioned task transformation verification device 102 includes a display 108, a processor 106, and a memory 104. The display 108 is used to present the verification result of the task transformation. The processor 106 is used to store the data source address of each task in the process of running each task in the first computing cluster; wherein the data source address includes the data source address of the input data of the task, and the data source address of the first output data corresponding to the input data; based on the data source address of the input data of each task, the second output data of each task obtained by running the task in the second computing cluster is obtained; based on the data source address of the first output data of each task and the second output data, verify whether the task transformation of migrating the tasks from the first computing cluster to the second computing cluster is successful. In addition, the first computing cluster 110 includes a database 112 and a processing engine 114, and the database 112 is used to store the data source address of each task in the process of running each task. The processing engine 114 is used to send the data source address of each task during the execution of each task to the task transformation verification device 102. The server 116 includes a database 118 and a processing engine 120. The database 118 is used to store the data source address of the input data of each task. The processing engine 120 is used to execute each task to obtain the second output data of each task.

[0025] In one or more embodiments, the big data cluster task transformation verification method of the present application can be applied to Figure 2 In the application environment shown. Figure 2 As shown, human-computer interaction can be performed between the user 202 and the user device 204. The user device 204 includes a memory 206 and a processor 208. In this embodiment, the user device 204 can refer to but is not limited to the operation performed by the above-mentioned task transformation verification device 102 to verify whether the task transformation of migrating the tasks from the first computing cluster to the second computing cluster is successful.

[0026] Optionally, the task transformation verification device 102 and the user device 204 include but are not limited to mobile phones, laptops, tablet computers, PDAs, MIDs (Mobile Internet Devices), desktop computers, smart TVs, etc. The application client with the big data cluster task transformation verification function in the embodiment of the present application includes but is not limited to a video client, an instant messaging client, a browser client, etc. The above network may include but is not limited to: a wired network, a wireless network, wherein the wired network includes: a local area network, a metropolitan area network and a wide area network, and the wireless network includes: Bluetooth, WIFI and other networks that realize wireless communication. The first computing cluster 110 and the second computing cluster 116 may be a single server, or a server cluster composed of multiple servers, or a cloud server, and the above cloud server includes but is not limited to a private cloud server or a public cloud server. The above is only an example, and no limitation is made to this in this embodiment.

[0027] In the related technology, the correctness of the computing task transformation is verified by manually comparing the task calculation results of the new and old clusters before and after the migration. However, this method is limited by manual technical experience and cost, and it is difficult to maintain the consistency of data input content when calculating the new and old cluster tasks. Therefore, the efficiency of data verification of the big data computing cluster after migration is low, and it is difficult to improve the accuracy of the computing task transformation.

[0028] In order to solve the above technical problems, as an optional implementation method, Figure 3 As shown, the embodiment of the present application provides a big data cluster task transformation verification method, including the following steps:

[0029] S302, storing a data source address of each task in the process of running each task in the first computing cluster; wherein the data source address includes a data source address of input data of the task and a data source address of first output data corresponding to the input data.

[0030] Specifically, in an embodiment of the present application, the data source address of each task in the process of running each task in the first computing cluster is stored in a preset storage space, and the preset storage space can be a storage space in the target cluster or other storage clusters. Compared with the related art by directly copying data to a preset storage space, the present application adopts the storage data source address, which can occupy as few storage resources as possible under the premise of achieving accurate verification. It should be noted that in order to ensure the accuracy of task transformation verification, the stored data source address of the present application includes the data source address of the input data of the task, and the data source address of the first output data corresponding to the input data; it should be noted that the data corresponding to the above data source address can include the input data and output data of each task before the data update occurs. Therefore, it can be ensured that in the verification stage after the cluster task is transformed, the above input data and output data are used as verification data to accurately verify whether the task transformation from the first computing cluster to the second computing cluster is successful.

[0031] S304, based on the data source address of the input data of each task, obtaining the second output data of each task obtained by running each task in the second computing cluster;

[0032] Specifically, the data source address of the input data of each task is obtained from the preset storage space, and the input data corresponding to each task is obtained. After each task is run in the second computing cluster according to the input data, the second output data of each task can be obtained. The second output data is the output data obtained after the task in the first computing cluster is run in the second computing cluster. It should be noted that the first computing cluster is the original computing cluster, and the second computing cluster is the target cluster for migrating the tasks of the first cluster.

[0033] S306: Verify whether task transformation of migrating each task from the first computing cluster to the second computing cluster is successful based on the data source address of the first output data of each task and the second output data.

[0034] Specifically, the first output data can be obtained according to the data source address of the first output data of each task. In one example, for example, by verifying whether the first output data and the second output data are the same, it is determined whether the task transformation of each task from the first computing cluster to the second computing cluster is successful. It should be noted that the embodiment of the present application can also verify whether the first output data and the second output data are the same data by summing, calculating the total, calculating the average, median, etc., of the data contained in the first output data and the second output data, so as to further determine whether the task transformation of each task from the first computing cluster to the second computing cluster is successful.

[0035] In an embodiment of the present application, a data source address of each task stored in a first computing cluster during the execution of each task is adopted; wherein the data source address includes input data of the task and first output data corresponding to the input data; based on the input data of each task, the second output data of each task obtained by executing each task in a second computing cluster is obtained; based on the first output data and the second output data of each task, a method is used to verify whether the task transformation of migrating each task from the first computing cluster to the second computing cluster is successful; since the present application obtains the data source address of each task in the original computing cluster during the cluster migration process, that is, the data environment of each task in the original computing cluster during execution can be accurately obtained, and the data environment can be used to accurately verify whether the task transformation of each task from the original computing cluster to the target computing cluster is successful; therefore, the present application can not only improve the efficiency of data verification of the big data computing cluster after migration, but also improve the accuracy of computing task transformation, thereby solving the technical problem that the efficiency of data verification of the big data computing cluster after migration is low and it is difficult to improve the accuracy of computing task transformation in the related technology.

[0036] In one or more embodiments, Figure 4 As shown, the embodiment of the present application also provides a big data cluster task transformation verification method, comprising the following steps:

[0037] S402, receiving task data of a first task uploaded by a first computing cluster within a preset data protection period, where the first task is any task among the tasks;

[0038] S404, analyzing whether the first task is to update associated data of the first task based on the task data of the first task;

[0039] S406: Obtain the data source address of the first task based on the parsing result.

[0040] Specifically, in an embodiment of the present application, the above-mentioned task data includes operation data corresponding to the task, and operation instructions corresponding to the operation data, and the operation instructions include read operation instructions or update operation instructions. The associated data of the task includes input data and output data corresponding to the task. Within the preset data protection period, according to the operation instructions of the task uploaded by the first computing cluster, the operation instructions are parsed to see whether they include update operation instructions, and it is determined whether the first task is going to update the associated data of the first task. If update operation instructions are included, the data to be updated is stored in the data table of the preset storage space, and the data tables of the storage space can be accessed based on the data source address of the first task.

[0041] S408 : Based on the data source address of the input data of each task, obtain the second output data of each task obtained by running each task in the second computing cluster.

[0042] S410 , based on the data source address of the first output data of each task and the second output data, verify whether task transformation for migrating each task from the first computing cluster to the second computing cluster is successful.

[0043] The above steps S408-S410 have been clearly explained in the previous text and will not be repeated here.

[0044] In one or more embodiments, Figure 5 As shown, the embodiment of the present application also provides a big data cluster task transformation verification method, comprising the following steps:

[0045] S502, receiving task data of a first task uploaded by a first computing cluster within a preset data protection period, where the first task is any task among the tasks;

[0046] S504: Based on the task data of the first task, analyze whether the first task is to update the associated data of the first task.

[0047] S506: When the parsing result indicates that the first task will not update its associated data, obtain the data source address corresponding to the first task.

[0048] S508: Generate a foreign table in a preset storage space, and store the data source address corresponding to the first task in the foreign table.

[0049] Specifically, the present application embodiment takes the preset storage space as the data protection space as an example, such as Figure 8 As shown, the associated data (input data and output data) of an example task in the original storage space involves 3 tables (database external tables), Table 1, Table 2 and Table 3, each table has 5 partitions, and a total of 15 partitions. The example task corresponds to the same number of external tables in the data protection space, each table has 5 partitions, and a total of 15 partitions. Stored in each partition is the data source address corresponding to the first task, that is, the address of the original storage space. Through the above-mentioned technical means, the data redundancy of the preset storage space can be reduced, and the storage resources can be occupied as little as possible while achieving accurate verification.

[0050] S510 : Based on the data source address of the input data of each task, obtain second output data of each task obtained by running each task in a second computing cluster.

[0051] S512: Verify whether task transformation for migrating each task from the first computing cluster to the second computing cluster is successful based on the data source address of the first output data of each task and the second output data.

[0052] The above steps S502-S504, S510-S512 have been clearly explained in the previous text and will not be repeated here.

[0053] In one or more embodiments, obtaining the data source address of the first task before data update based on the parsing result includes:

[0054] When the parsing result indicates that the first task will update its associated data, obtaining a data source address corresponding to the first task;

[0055] A physical table corresponding to the first task is generated in the preset storage space, associated data of the first task is copied according to the data source address, and the copied data is stored in the physical table.

[0056] Specifically, Figure 8 As shown, the associated data of an example task in the original storage space involves 3 tables, Table 1, Table 2 and Table 3, each table has 5 partitions, a total of 15 partitions. The example task corresponds to the same number of external tables in the preset storage space (data protection space), each table has 5 partitions, a total of 15 partitions. In the case that the example task will update its own associated data, that is, for the data in partition 3 of Table 1 corresponding to the data update operation, the data to be updated in partition 3 is copied to the data protection space, and the corresponding physical table is established in partition 3 of Table 1 of the data protection space. Figure 8 The physical table of partition 3 of the data protection space in the example stores the associated data corresponding to the example task.

[0057] In one or more embodiments, before generating the physical table corresponding to the first task in the preset storage space, the method further includes:

[0058] If a table exists in the preset storage space, and the data source address corresponding to the first task is stored in the table, the data source address corresponding to the first task in the table is deleted.

[0059] like Figure 8 As shown, in the case where the example task will update its own associated data, such as partition 3 in Table 1 to be updated, the data to be updated in partition 3 is copied to the data protection space, and a corresponding physical table is established in partition 3 in the data protection space table 1. If there is an external table in partition 3 in the data protection space table 1, and the external table stores the data source address corresponding to the example task in the original storage space, the data source address corresponding to the example task in the external table is deleted.

[0060] In one or more embodiments, obtaining, based on the input data of each task, second output data of each task obtained by running each task in the second computing cluster includes:

[0061] Receive a running request of a first task uploaded by a second computing cluster, and obtain a data source address of input data of the first task from a preset storage space;

[0062] Sending a data source address of input data of the first task to the second computing cluster, so that the second computing cluster runs the first task based on the data source address of the input data of the first task;

[0063] Acquire, from the second computing cluster, second output data obtained by the second computing cluster running the first task.

[0064] Specifically, in an embodiment of the present application, for example, the big data cluster task transformation verification device receives a request to run task A uploaded by the second computing cluster, obtains the data source address of the input data of task A from the preset storage space, and the second computing cluster obtains the input data of task A according to the data source address of the input data. Task A obtains second output data after running in the second computing cluster according to the input data, and the big data cluster task transformation verification device can obtain the second output data from the second computing cluster.

[0065] In one or more embodiments, obtaining the data source address of the input data of the first task from the preset storage space includes:

[0066] If the external table in the preset storage space contains the data source address of the first task, obtaining the data source address of the input data of the first task from the external table;

[0067] If the external table in the preset storage space does not include the data source address of the first task, the address of the physical table in the preset storage space is determined as the data source address of the input data of the first task.

[0068] Specifically, in the embodiment of the present application, for example, if the external table in the preset storage space contains the data source address of Task A, Task A, then the data source address of the input data of Task A can be obtained from the external table, and the input data of Task A can be obtained according to the data source address obtained from the external table. When the external table does not contain the data source address of Task A, in the embodiment of the present application, the address of the physical table in the preset storage space is determined as the data source address of the input data of Task A, the physical table is accessed from the preset storage space according to the address of the physical table, and the input data of Task A is obtained from the physical table.

[0069] In one or more embodiments, verifying whether task transformation of migrating each task from the first computing cluster to the second computing cluster is successful based on the first output data and the second output data of each task includes:

[0070] Read the first output data of each task from the data source address of the first output data of each task respectively;

[0071] Calculate a first inspection index of the first output data and a second inspection index of the second output data of each task respectively;

[0072] When the first inspection index and the second inspection index of each task are the same, determining that the task transformation of each task from the first computing cluster to the second computing cluster is successful;

[0073] In the case where there are tasks whose first inspection index is different from the second inspection index, it is determined that task transformation of each task from the first computing cluster to the second computing cluster is unsuccessful.

[0074] Specifically, in the embodiment of the present application, the first inspection index of the first output data and the second inspection index of the second output data include, but are not limited to, data obtained by summing, totaling, averaging, median, etc. the first output data and the second output data. In the embodiment of the present application, it is not necessary to compare each data in the first output data and the second output data one by one to accurately verify whether the data contained in the two are exactly the same.

[0075] In one example, for example, the first inspection indicator of the current task is the first sum value obtained by summing up each data in the first output data, and the second inspection indicator is the second sum value obtained by summing up each data in the first output data. When the first sum value and the second sum value are the same, it is determined that the task transformation of the current task from the first computing cluster to the second computing cluster is successful; in the case of tasks where the first sum value and the second sum value are different, it is determined that the task transformation of the current task from the first computing cluster to the second computing cluster is unsuccessful. In this way, by verifying the first inspection indicator and the second inspection indicator of each task, it is accurately determined that the task transformation of each task from the first computing cluster to the second computing cluster is unsuccessful.

[0076] Based on the above embodiments, in an application embodiment, the present application embodiment also provides a big data cluster task transformation verification method, which includes a data protection stage and an accuracy verification stage after the cluster task is transformed.

[0077] like Figure 6As shown, the above data protection stage is the data environment for computing and running all tasks to be transformed within the data protection period, so that the cluster can obtain accurate input data and output data to be compared in the verification stage after migration. The data protection stage includes the following steps:

[0078] 1. Set a period of time as the data protection period, receive submission requests from each task in the original computing cluster during this period, and detect all computing tasks.

[0079] 2. Analyze each task code, parse out the input data source and output data source information in the task from the partition of the data table of each task, and record the task ID, input data source and storage metadata information of input data, output data table and storage metadata information of output data, and insert the above data into the global detection queue.

[0080] 3. Create a data protection space in the storage space corresponding to the new cluster, and create a foreign table corresponding to each task in the data protection space. The foreign table stores the input and output data sources of the task.

[0081] 4. If within the data protection period, it is determined that a task will update the data source in the global monitoring queue, the task will be blocked first.

[0082] 5. Perform write-time copy protection on the data source to be updated, that is, synchronously copy the partition data of the table where the data to be updated is located from the original storage cluster to the data protection space, delete the original external table reference of the data to be updated, and create a corresponding physical table for it.

[0083] 6. Resume the blocked task and continue executing the task.

[0084] like Figure 7 As shown in the figure, in the accuracy verification phase after the big data cluster task is transformed, it is necessary to run the transformed task in the target cluster and obtain the running results, and compare the running results with the calculation results of the original cluster. Since the input data of the two are exactly the same, a full equality comparison with low complexity can be performed. Specifically, the accuracy verification phase includes:

[0085] 1. Assuming that the task transformation has been completed and the conditions for running in the target cluster are met, submit the task in the target cluster;

[0086] 2. Based on the blood relationship between the task data, determine the input data source and output data source corresponding to the task.

[0087] Automatically schedule the calculation for each task according to the scheduling relationship; automatically replace and splice the input data to obtain the input data source required by each task, so that the input data is completely consistent with the task when it is running in the original computing cluster. There are two situations here: if the input data source of the task has not changed during the data protection period, when the target computing cluster is running, the input data refers to the data table or partition in the original cluster; conversely, if the input data source of the task changes during the data protection period, since the write-time copy mechanism is set, when the task is run in the new computing cluster, the input data of the task is obtained from the physical table of the data protection space.

[0088] 3. Replace the input data source of the task with the data source in the data protection space.

[0089] 4. Execute each task in the target computing cluster.

[0090] 5. Obtain the input data and output data of each task in the original computing cluster.

[0091] 6. Save the second output result of each task in the target computing cluster, compare it with the first output result of the task in the original computing cluster, and verify whether the task transformation from the original computing cluster to the target computing cluster is successful based on the comparison result.

[0092] There are also two situations here. If the first output data of the task in the original computing cluster has not changed during the data protection period, the second output result will be compared with the corresponding table or partition in the original computing cluster. Otherwise, the second output result will be compared with the data of the corresponding table or partition in the data protection space.

[0093] The embodiment of the present application can determine the comparison index by performing algorithms such as summing, totaling, averaging, and median on the data in the data tables of the first output result and the second output result, and automatically complete the result inspection of the task according to whether the comparison index is the same to verify the correctness of the task transformation. The above algorithm has low computing power requirements during the verification process, so it can greatly improve the efficiency of the task transformation correctness verification.

[0094] The above-mentioned technical solution of the present application also has the following beneficial technical effects: by protecting the data environment when the task is running, it ensures that the task can have exactly the same data environment when running before and after the transformation, thereby greatly improving the accuracy of the correctness verification of subsequent task transformations.

[0095] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0096] According to another aspect of the embodiment of the present application, a big data cluster task transformation verification device of a big data cluster task transformation verification method is also provided. Fig. 9 As shown, the device comprises:

[0097] The storage unit 902 is used to store the data source address of each task in the process of running each task in the first computing cluster; wherein the data source address includes the input data of the task and the data source address of the first output data corresponding to the input data;

[0098] An acquiring unit 904 is configured to acquire, based on a data source address of input data of each task, second output data of each task obtained by running each task in a second computing cluster;

[0099] The verification unit 906 is used to verify whether the task transformation of migrating each task from the first computing cluster to the second computing cluster is successful based on the data source address of the first output data of each task and the second output data.

[0100] In an embodiment of the present application, a data source address of each task stored in a first computing cluster during the execution of each task is adopted; wherein the data source address includes input data of the task and first output data corresponding to the input data; based on the input data of each task, the second output data of each task obtained by executing each task in a second computing cluster is obtained; based on the first output data and the second output data of each task, a method is used to verify whether the task transformation of migrating each task from the first computing cluster to the second computing cluster is successful; since the present application obtains the data source address of each task in the original computing cluster during the cluster migration process, that is, the data environment of each task in the original computing cluster during execution can be accurately obtained, and the data environment can be used to accurately verify whether the task transformation of each task from the original computing cluster to the target computing cluster is successful; therefore, the present application can not only improve the efficiency of data verification of the big data computing cluster after migration, but also improve the accuracy of computing task transformation, thereby solving the technical problem that the efficiency of data verification of the big data computing cluster after migration is low and it is difficult to improve the accuracy of computing task transformation in the related technology.

[0101] In one or more embodiments, the storage unit 902 includes:

[0102] A first receiving module, configured to receive task data of a first task uploaded by a first computing cluster within a preset data protection period, wherein the first task is any one of the tasks;

[0103] A parsing module, configured to parse, based on the task data of the first task, whether the first task is to update the associated data of the first task;

[0104] The first acquisition module is used to acquire the data source address of the first task based on the parsing result.

[0105] In one or more embodiments, the first acquisition module includes:

[0106] A first acquisition subunit is used to acquire the data source address corresponding to the first task when the parsing result indicates that the first task will not update its associated data;

[0107] The first storage subunit is used to generate a foreign table in a preset storage space, and store the data source address corresponding to the first task in the foreign table.

[0108] In one or more embodiments, the first acquisition module includes:

[0109] A second acquisition subunit is used to acquire a data source address corresponding to the first task when the parsing result indicates that the first task will update its associated data;

[0110] The second storage subunit is used to generate a physical table corresponding to the first task in the preset storage space, copy the associated data of the first task according to the data source address, and store the copied data in the physical table.

[0111] In one or more embodiments, the big data cluster task transformation verification device further includes:

[0112] A deleting unit is configured to delete the data source address corresponding to the first task in the outer table if there is an outer table in the preset storage space and the data source address corresponding to the first task is stored in the outer table.

[0113] In one or more embodiments, the acquisition unit 904 includes:

[0114] A second receiving module is used to receive a running request of the first task uploaded by the second computing cluster, and obtain a data source address of input data of the first task from a preset storage space;

[0115] a sending module, configured to send a data source address of input data of the first task to the second computing cluster, so that the second computing cluster runs the first task based on the data source address of the input data of the first task;

[0116] A second acquisition module is configured to acquire, from the second computing cluster, second output data obtained by the second computing cluster running the first task.

[0117] In one or more embodiments, the second receiving module includes:

[0118] a third acquisition subunit, configured to acquire the data source address of the input data of the first task from the external table in the preset storage space based on the external table containing the data source address of the first task;

[0119] The determination subunit is used to determine the address of the physical table in the preset storage space as the data source address of the input data of the first task if the outer table in the preset storage space does not contain the data source address of the first task.

[0120] In one or more embodiments, the verification unit 906 includes:

[0121] A reading module, used for reading the first output data of each task from the data source address of the first output data of each task;

[0122] A calculation module, used for respectively calculating a first inspection index of the first output data and a second inspection index of the second output data of each task;

[0123] A first determination module, configured to determine that the task transformation of each task from the first computing cluster to the second computing cluster is successful when the first inspection index of each task is the same as the second inspection index;

[0124] The second determination module is configured to determine that task transformation of each task from the first computing cluster to the second computing cluster is unsuccessful when there are tasks with different first inspection indicators and second inspection indicators.

[0125] According to another aspect of the embodiment of the present application, a big data cluster task transformation verification system is also provided, the system comprising a first computing cluster, a second computing cluster, and a big data cluster task transformation verification device;

[0126] The first computing cluster is used to send the data source address of each task to the big data cluster task transformation verification device during the execution of each task;

[0127] The second computing cluster is used to run each task based on a data source address of input data of each task to obtain second output data of each task;

[0128] The big data cluster task transformation verification device is used to store the data source address of each task in the process of running each task in the first computing cluster; wherein the data source address includes the data source address of the input data of the task and the data source address of the first output data corresponding to the input data;

[0129] Based on the data source address of the input data of each task, obtaining the second output data of each task obtained by running each task in the second computing cluster;

[0130] Based on the data source address of the first output data of each task and the second output data, verify whether task transformation for migrating each task from the first computing cluster to the second computing cluster is successful.

[0131] According to another aspect of the embodiment of the present application, an electronic device for implementing the above-mentioned big data cluster task transformation verification method is also provided. The electronic device can be Figure 1 The task modification verification device shown in the figure. This embodiment takes the electronic device as the task modification verification device as an example for explanation. Fig.10 As shown, the electronic device includes a memory 1002 and a processor 1004. The memory 1002 stores a computer program, and the processor 1004 is configured to execute the steps in any of the above method embodiments through the computer program.

[0132] Optionally, in this embodiment, the electronic device may be at least one network device among a plurality of network devices of a computer network.

[0133] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0134] S1, storing the data source address of each task in the process of running each task in the first computing cluster; wherein the data source address includes the data source address of the input data of the task and the data source address of the first output data corresponding to the input data;

[0135] S2, based on the input data of each task, obtaining second output data of each task obtained by running each task in a second computing cluster;

[0136] S3: Based on the first output data and the second output data of each task, verify whether task transformation of migrating each task from the first computing cluster to the second computing cluster is successful.

[0137] Among them, the memory 1002 can be used to store software programs and modules, such as the program instructions / modules corresponding to the big data cluster task transformation verification method and device in the embodiment of the present application. The processor 1004 executes various functional applications and data processing by running the software programs and modules stored in the memory 1002, that is, the above-mentioned big data cluster task transformation verification method is realized. The memory 1002 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1002 may further include a memory remotely arranged relative to the processor 1004, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. Among them, the memory 1002 can be specifically used, but not limited to, for storing task transformation verification results.

[0138] As an example, Fig.10 As shown, the memory 1002 may include, but is not limited to, the storage unit 902, the acquisition unit 904, and the verification unit 906 in the big data cluster task transformation verification device. In addition, it may also include, but is not limited to, other module units in the big data cluster task transformation verification device, which will not be repeated in this example.

[0139] Optionally, the transmission device 1006 is used to receive or send data via a network. Specific examples of the above-mentioned network may include a wired network and a wireless network. In one example, the transmission device 1006 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers via a network cable so as to communicate with the Internet or a local area network. In one example, the transmission device 1006 is a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0140] In addition, the electronic device mentioned above further includes: a connection bus 1008, which is used to connect various module components in the electronic device mentioned above.

[0141] In other embodiments, the electronic device may be a node in a distributed system, wherein the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting the multiple nodes in the form of network communication. Among them, the nodes may form a peer-to-peer (P2P) network, and any form of computing device, such as a server, terminal, or other electronic device, may become a node in the blockchain system by joining the peer-to-peer network.

[0142] In one or more embodiments, the present application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned serialization method. Wherein, the computer program is configured to execute the steps of any of the above-mentioned method embodiments when running.

[0143] Optionally, in this embodiment, the computer-readable storage medium may be configured to store a computer program for performing the following steps:

[0144] S1, storing the data source address of each task in the process of running each task in the first computing cluster; wherein the data source address includes the data source address of the input data of the task and the data source address of the first output data corresponding to the input data;

[0145] S2, based on the input data of each task, obtaining second output data of each task obtained by running each task in a second computing cluster;

[0146] S3: Based on the first output data and the second output data of each task, verify whether task transformation of migrating each task from the first computing cluster to the second computing cluster is successful.

[0147] Optionally, in this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0148] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0149] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on such understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which can be personal computers, servers or network devices, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention.

[0150] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0151] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0152] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0153] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0154] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

[0155] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

Claims

1. A big data cluster task transformation verification method, characterized in that: The method comprises: storing a data source address of each task in the process of running each task in the first computing cluster; wherein the data source address includes a data source address of input data of the task and a data source address of first output data corresponding to the input data; Based on the data source address of the input data of each task, obtaining the second output data of each task obtained by running each task in the second computing cluster; Based on the data source address of the first output data of each task and the second output data, verify whether task transformation for migrating each task from the first computing cluster to the second computing cluster is successful.

2. The method according to claim 1, characterized in that The data source address of each task stored in the first computing cluster during the execution of each task includes: Within a preset data protection period, receiving task data of a first task uploaded by a first computing cluster, where the first task is any one of the tasks; Based on the task data of the first task, analyzing whether the first task is to update the associated data of the first task; The data source address of the first task is obtained based on the parsing result.

3. The method according to claim 2, characterized in that The acquiring the data source address of the first task based on the parsing result includes: When the parsing result indicates that the first task will not update its associated data, obtaining the data source address corresponding to the first task; A table is generated in a preset storage space, and the data source address corresponding to the first task is stored in the table.

4. The method according to claim 2 or 3, characterized in that: The acquiring the data source address of the first task before data update based on the parsing result includes: When the parsing result indicates that the first task will update its associated data, obtaining a data source address corresponding to the first task; A physical table corresponding to the first task is generated in the preset storage space, associated data of the first task is copied according to the data source address, and the copied data is stored in the physical table.

5. The method according to claim 4, characterized in that Before generating the physical table corresponding to the first task in the preset storage space, the method further includes: If a table exists in the preset storage space, and the data source address corresponding to the first task is stored in the table, the data source address corresponding to the first task in the table is deleted.

6. The method according to claim 1, characterized in that Based on the input data of each task, obtaining second output data of each task obtained by running each task in the second computing cluster includes: Receive a running request of a first task uploaded by a second computing cluster, and obtain a data source address of input data of the first task from a preset storage space; Sending a data source address of input data of the first task to the second computing cluster, so that the second computing cluster runs the first task based on the data source address of the input data of the first task; Acquire, from the second computing cluster, second output data obtained by the second computing cluster running the first task.

7. The method according to claim 6, characterized in that The acquiring the data source address of the input data of the first task from the preset storage space includes: If the external table in the preset storage space contains the data source address of the first task, obtaining the data source address of the input data of the first task from the external table; If the external table in the preset storage space does not include the data source address of the first task, the address of the physical table in the preset storage space is determined as the data source address of the input data of the first task.

8. The method according to claim 1, characterized in that Verifying whether task transformation of migrating each task from the first computing cluster to the second computing cluster is successful based on the first output data and the second output data of each task, including: Read the first output data of each task from the data source address of the first output data of each task respectively; Calculate a first inspection index of the first output data and a second inspection index of the second output data of each task respectively; When the first inspection index and the second inspection index of each task are the same, determining that the task transformation of each task from the first computing cluster to the second computing cluster is successful; In the case where there are tasks whose first inspection index is different from the second inspection index, it is determined that task transformation of each task from the first computing cluster to the second computing cluster is unsuccessful.

9. A big data cluster task transformation verification device, characterized in that: The device comprises: A storage unit, used to store the data source address of each task in the process of running each task in the first computing cluster; wherein the data source address includes the input data of the task and the data source address of the first output data corresponding to the input data; an acquiring unit, configured to acquire, based on a data source address of input data of each task, second output data of each task obtained by running each task in a second computing cluster; A verification unit is used to verify whether the task transformation of migrating each task from the first computing cluster to the second computing cluster is successful based on the data source address of the first output data of each task and the second output data.

10. A big data cluster task transformation verification system, characterized in that: The system includes a first computing cluster, a second computing cluster, and a big data cluster task transformation verification device; The first computing cluster is used to send the data source address of each task to the big data cluster task transformation verification device during the execution of each task; The second computing cluster is used to run each task based on a data source address of input data of each task to obtain second output data of each task; The big data cluster task transformation verification device is used to store the data source address of each task in the process of running each task in the first computing cluster; wherein the data source address includes the data source address of the input data of the task, and the data source address of the first output data corresponding to the input data; based on the data source address of the input data of each task, obtain the second output data of each task obtained by running each task in the second computing cluster; Based on the data source address of the first output data of each task and the second output data, verify whether task transformation for migrating each task from the first computing cluster to the second computing cluster is successful.

11. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method according to any one of claims 1 to 8.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the method according to any one of claims 1 to 8.

Images