A new and old task checking method and device

Abstract

The present disclosure provides a new and old task verification method and device, belonging to the field of big data offline computing. The method comprises: obtaining initial data of a target task to be verified in a target business; cleaning the initial data of the target task according to a target data cleaning rule of the target business to obtain first data, wherein the target data cleaning rule is used to adapt the first data to a new and old task verification rule; calculating the md5 value corresponding to the target task from the first data; according to the new and old task verification rule, verifying the md5 value corresponding to the target task with the md5 values corresponding to a plurality of historical tasks stored in a reference library to determine the new and old task verification result of the target task. The present disclosure can improve verification accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of big data offline computing, and in particular to a new and old task verification method and device. BACKGROUND

[0002] In the practical application of big data computing engines, there are usually multiple tasks that need to be executed, analyzed and processed, among which the tasks that are routinely executed and processed are called old tasks, and the tasks that are executed and processed according to new requirements are called new tasks. Since the old tasks have been executed multiple times for analysis and processing, the related specifications are relatively mature, and the runtime is relatively stable; while the new tasks have not been verified, there is a risk of causing instability to the offline big data, so different specifications and higher risk management requirements need to be set. Before executing the analysis and processing of the tasks, the verification of the new and old tasks needs to be performed to distinguish the specifications and risk management and improve the stability of the offline big data operation.

[0003] Currently, there are the following methods to distinguish new and old tasks:

[0004] 1. Based on query language: The current solution is to determine the new and old of the task based on the query language submitted by the user. However, this method can only determine the new and old of the task at the time of query submission, and cannot dynamically determine the new and old of the task during the execution of the task. If the task changes or is resubmitted during the execution, it is impossible to accurately distinguish between new and old tasks. Moreover, the user's routine tasks generally only replace table partitions and should not be identified as new tasks.

[0005] 2. Query identifier: Using task identifiers to distinguish the new and old of the task is a common method. However, this method requires maintenance and management of task identifiers, which may face identifier conflicts or management complexity problems in large-scale and high-concurrency environments. Moreover, it requires large specification modifications to user tasks, which is costly and time-consuming.

[0006] 3. Query plan: Comparing query plans to distinguish new and old tasks is an effective method. However, this method may be affected by query optimization or caching mechanisms. If the query plan is cached or reused, it is impossible to accurately determine the new and old of the task.

[0007] 4. Task status: Task status is used to track the execution progress and state changes of the task. However, the task status may be affected by various factors such as network delay, resource limitation, etc. In a distributed environment, the accuracy and real-time performance of the task status may be delayed or inaccurate. Moreover, it is impossible to distinguish between user routine tasks and new tasks.

[0008] 5. Scheduling tool: It is a common method to use a scheduling tool to manage the execution of tasks. However, this approach requires reliance on an additional scheduling system and can introduce additional complexity and management overhead. At the same time, the configuration and maintenance of the scheduling tool also require certain technical and resource investment.

[0009] Taking the first method as an example, each time the execution of the routine task is performed, only the table partition is replaced, and the rest is the same. If the user-submitted query language is used as the judgment standard, it is possible to identify as a new task, which is actually the same old task.

[0010] In summary, the existing offline big data engine may have some shortcomings in distinguishing new tasks and old tasks, and the accuracy of verification is not high. In practical applications, these methods are not suitable for production environments. SUMMARY

[0011] Therefore, the embodiments of the present disclosure provide a new and old task verification method and device to improve the verification accuracy.

[0012] According to an aspect of the present disclosure, a new and old task verification method is provided, the method comprising:

[0013] obtaining initial data of a target task to be verified in a target service;

[0014] cleaning the initial data of the target task according to a target data cleaning rule of the target service to obtain first data, wherein the target data cleaning rule is used to adapt the first data to a new and old task verification rule;

[0015] calculating an md5 value corresponding to the target task from the first data;

[0016] According to the new and old task verification rule, the md5 value corresponding to the target task is verified with the md5 values corresponding to a plurality of historical tasks stored in a reference library, and the new and old task verification result of the target task is determined.

[0017] According to another aspect of the present disclosure, a new and old task verification device is provided, the device comprising:

[0018] The obtaining module is configured to obtain initial data of a target task to be verified in a target service;

[0019] The cleaning module is configured to clean the initial data of the target task according to a target data cleaning rule of the target service to obtain first data, wherein the target data cleaning rule is used to adapt the first data to a new and old task verification rule;

[0020] The calculation module is configured to calculate an md5 value corresponding to the target task from the first data;

[0021] a checking module configured to check, according to the new-old task checking rule, the md5 value corresponding to the target task against md5 values corresponding to a plurality of historical tasks stored in the benchmark library, and determine a new-old task checking result of the target task.

[0022] According to another aspect of the present disclosure, an electronic device is provided, comprising:

[0023] a processor; and

[0024] a memory storing a program,

[0025] wherein the program comprises instructions that, when executed by the processor, cause the processor to perform the new-old task checking method.

[0026] According to another aspect of the present disclosure, a non-transitory computer readable storage medium storing computer instructions is provided, wherein the computer instructions are configured to cause a computer to perform the new-old task checking method.

[0027] In the present disclosure, whenever new-old task checking needs to be performed on a target task, initial data of the target task to be checked in a target service is obtained, the initial data of the target task is cleaned according to a target data cleaning rule of the target service to obtain first data, an md5 value corresponding to the target task is calculated from the first data, the md5 value corresponding to the target task is checked against md5 values corresponding to a plurality of historical tasks stored in a benchmark library according to a new-old task checking rule, and a new-old task checking result of the target task is determined. Since the target data cleaning rule can make the first data adapt to the new-old task checking rule, and the target data cleaning rule is adapted to the target service, the accuracy of the new-old task checking is improved, and the new-old task checking can be applied to a production environment. BRIEF DESCRIPTION OF DRAWINGS

[0028] More details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0029] Figure 1 a flowchart of a new-old task checking method is shown according to an exemplary embodiment of the present disclosure;

[0030] Figure 2 a schematic diagram of a Hive engine is shown according to an exemplary embodiment of the present disclosure;

[0031] Figure 3 a schematic diagram of a Presto engine is shown according to an exemplary embodiment of the present disclosure;

[0032] Figure 4A schematic diagram of a Spark engine is shown according to an example embodiment of the present disclosure;

[0033] Figure 5 A schematic block diagram of a new-old task verification apparatus is shown according to an example embodiment of the present disclosure;

[0034] Figure 6 A structural block diagram of an example electronic device that can be used to implement embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0035] Embodiments of the present disclosure will be described in more detail with reference to the drawings. While certain embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be interpreted as being limited to the embodiments set forth herein; rather, these embodiments are provided so that the present disclosure can be more thoroughly and completely understood. It should be understood that the drawings of the present disclosure are only for illustrative purposes and should not be used to limit the scope of protection of the present disclosure.

[0036] It should be understood that each step recited in the method embodiments of the present disclosure can be executed in different orders and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the execution of the steps shown. The scope of the present disclosure is not limited in this respect.

[0037] The term "comprising" and variations thereof as used herein are used inclusively, i.e., "comprising, but not limited to." The term "based on" is "based, at least in part, on." The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments." Related definitions are given below in the description of the various terms. It should be noted that the concepts "first", "second", etc. mentioned in the present disclosure are only used to distinguish different apparatuses, modules or units, and are not used to limit the order or interdependence of the functions performed by these apparatuses, modules or units.

[0038] It should be noted that the modification of "one", "multiple" mentioned in the present disclosure is illustrative rather than limiting, and those skilled in the art should understand that, unless otherwise explicitly indicated in the context, it should be understood as "one or more".

[0039] The names of the messages or information exchanged between the plurality of apparatuses in the embodiments of the present disclosure are only used for illustrative purposes, and are not used to limit the scope of the messages or information.

[0040] In order to solve the technical differences of new and old task processing, risk management, process improvement and other needs of business, at the same time, the infrastructure department needs to set specifications for the tasks submitted by the business side, and needs to distinguish between new and old tasks. At the same time, because the offline tasks run by the business side may only differ in table partition and part of the parameters every day, the rest is the same, so such routine tasks cannot be identified as new tasks.

[0041] In order to meet the above production environment, the embodiment of the application provides a new and old task verification method, which can be applied to a big data offline computing engine, including Hive (Hybrid Integrated and Visual Engine), Presto (Data Query Engine), Spark (Memory Computing Engine) and the like. As shown in the new and old task verification method flow chart, the processing flow of the method can include the following steps 101-104. Figure 1

[0042] Step 101, obtaining the initial data of the target task to be verified in the target business.

[0043] Among them, the initial data can at least include sql data and execution plan data.

[0044] In a possible implementation, the business side can submit tasks in the client of the target engine in the process of implementing the target business. In order to solve the technical differences of new and old task processing, risk management, process improvement and other needs of business, at the same time, the infrastructure department needs to set specifications for the tasks submitted by the business side, and needs to divide the submitted tasks into new and old tasks. On this basis, the task to be verified can be taken as the target task, and the initial data of the target task is obtained.

[0045] Step 102, cleaning the initial data of the target task according to the target data cleaning rule of the target business to obtain the first data.

[0046] Among them, the target data cleaning rule is used to adapt the above first data to the new and old task verification rule.

[0047] In a possible implementation, because the initial data has noise, error, inconsistency and the like, the same task may be judged as different tasks, which affects the accuracy of subsequent analysis, so the initial data can be cleaned to improve the quality and reliability of the data.

[0048] ​And, the data format of each specific business can be different, in order to make the first data after cleaning adapt to the new and old task checking rules, to improve the accuracy of the new and old task checking, each business can have its corresponding data cleaning rule. In the embodiment, whenever the initial data of any business needs to be cleaned, the target cleaning rule corresponding to the business can be obtained, and then the initial data of the corresponding business is cleaned according to the target cleaning rule.

[0049] Optionally, the target data cleaning rule of each business can be pre-set. The target data cleaning rule can be obtained based on the following way:

[0050] The initial data cleaning rule is taken as the first test data cleaning rule, and the following test processing is performed:

[0051] The initial data of the first historical task is cleaned according to the current test data cleaning rule to obtain second data, and the md5 value corresponding to the first historical task is calculated based on the second data, wherein the first historical task refers to the historical task of the target business in a first preset historical time length;

[0052] The initial data of the second historical task is cleaned according to the current test data cleaning rule to obtain third data, and the md5 value corresponding to the second historical task is calculated based on the third data, wherein the second historical task refers to the historical task of the target business in a second preset historical time length;

[0053] According to the new and old task checking rule, the md5 value corresponding to the second historical task and the md5 value corresponding to the first historical task are checked based on the second historical task, to determine the test new and old task checking result of the second historical task relative to the first historical task;

[0054] Based on the accuracy of the test new and old task checking result, the current test data cleaning rule is adjusted to determine the target data cleaning rule, which can be specifically as follows:

[0055] If the accuracy of the test new and old task checking result is less than or equal to a preset threshold, the current test data cleaning rule is adjusted to obtain the next test data cleaning rule, and the above test processing is repeated until the accuracy of the test new and old task checking result is greater than the preset threshold;

[0056] If the accuracy of the test new and old task checking result is greater than the preset threshold, the current test data cleaning rule is taken as the target data cleaning rule.

[0057] As an example, the historical tasks (i.e., first historical tasks) of the target business in the last 3 months (i.e., a first preset historical length) can be taken as the task reference. The initial data of the first historical tasks of the target business in the last 3 months is obtained, and data cleaning is performed according to the initial data cleaning rule to obtain the cleaned data of each first historical task. The task data at this time is referred to as second data in this embodiment. Further, the md5 value of the second data of each first historical task can be calculated. The md5 value is a widely used cryptographic hash function that can generate a 128-bit (16-byte) hash value to ensure the integrity of information transmission. The specific method for calculating the md5 value is not limited in this embodiment.

[0058] The historical tasks (i.e., second historical tasks) of 1 month (i.e., a second preset historical length) are extracted as the tasks to be verified, and the new-old task relationship between the tasks to be verified and the task reference is known, i.e., it is known that the tasks to be verified are new tasks or old tasks relative to the task reference. The initial data of the second historical tasks of 1 month is obtained, and data cleaning is performed according to the initial data cleaning rule to obtain the cleaned data of each second historical task. The task data at this time is referred to as third data in this embodiment. Further, the md5 value of the third data of each second historical task can be calculated.

[0059] According to the pre-set new-old task verification rule, the md5 value corresponding to the second historical task and the md5 value corresponding to the first historical task are verified to determine the test new-old task verification result of the second historical task relative to the first historical task under the current data cleaning rule.

[0060] Since the new-old task relationship of the second historical task relative to the first historical task is known, the test new-old task result of each second historical character can be verified, and the accuracy rate can be calculated. The accuracy rate is compared with a preset threshold (such as 98%, which is not limited). If the accuracy rate is greater than the preset threshold, it can be considered that the current data cleaning rule is adapted to the new-old task verification rule, and the current data cleaning rule is taken as the target data cleaning rule for use in the subsequent new-old task verification process.

[0061] If the accuracy rate is less than or equal to the preset threshold, it can be considered that the current data cleaning rule is not adapted to the new-old task verification rule, and the data cleaning rule can be adjusted, and the above process is repeated based on the adjusted data cleaning rule to re-determine the accuracy rate. Until the accuracy rate is greater than the preset threshold, the current data cleaning rule is taken as the target data cleaning rule for use in the subsequent new-old task verification process.

[0062] Optionally, after the target data cleaning rule is determined, a benchmark library can also be established based on the target data cleaning rule. Specifically, the following processing can be included:

[0063] Obtaining initial data of a plurality of third historical tasks of the target service in a third preset historical time length;

[0064] Cleaning the initial data of the third historical tasks according to the target data cleaning rule to obtain fourth data;

[0065] Calculating the md5 value corresponding to the third historical task from the fourth data;

[0066] Based on the md5 values corresponding to the plurality of third historical tasks, a benchmark library is constructed.

[0067] As an example, the benchmark library can be established based on the historical tasks (i.e., the third historical tasks) of the target service in the last 5 months (i.e., the third preset historical time length). The initial data of the third historical tasks of the target service in the last 5 months is obtained, and the data is cleaned according to the determined target data cleaning rule to obtain the cleaned data of each third historical task. The task data at this time is referred to as the fourth data in this embodiment. Further, the md5 value corresponding to the fourth data of each third historical task can be calculated and stored in the benchmark library to establish the benchmark library used in the new and old task verification process. That is, the historical tasks already stored in the benchmark library are used as the benchmark of the old tasks to determine whether the task being tested is an old task or a new task.

[0068] It should be noted that the first preset historical time length, the second preset historical time length, and the third preset historical time length can be selected as any time length, which is not limited in this embodiment.

[0069] Optionally, in the case where the initial data includes the sql data and the execution plan data of the task:

[0070] The target data cleaning rule corresponding to the sql data can include any one or more of the following: removing empty characters in the sql data, removing numbers in the sql data, removing single quotes and double quotes in the sql data, removing commas in the sql data, and removing task IDs in the sql data.

[0071] The target data cleaning rule corresponding to the execution plan data includes: extracting and parsing the execution plan data; cleaning and processing the execution plan data to remove comments, spaces, and line breaks, extracting specific execution plan nodes and / or key information; and performing data conversion and filtering on the cleaned execution plan data.

[0072] In a possible implementation, removing spaces, carriage returns, tabs and other empty characters in the sql data can solve the problem of invalid character interference. Removing numbers in the sql data can solve the problem of treating routine tasks that only change table partitions as new tasks. Removing single quotes and double quotes in the sql data can solve the problem of splicing sql strings. Removing commas in the sql data can solve the problem of multiple comma-separated numbers in conditions. Removing task IDs in the sql data can solve the problem of distinguishing between new and old tasks due to the resetting of task IDs caused by the restart or reset of the computing engine task queue.

[0073] In the execution plan data, the execution plan is usually presented in the form of text, containing the details of the logical and physical execution plan of the query. The execution plan data can be obtained by executing the EXPLAIN<your_query> command or setting sethive.exec.orc.execution.plan=true in the Hive command line interface. Then the execution plan data is parsed to obtain the text characters contained therein. The obtained text characters are cleaned and processed to remove unnecessary comments, spaces and line breaks, and the specific execution plan nodes and / or key information are extracted. Further, the cleaned data can be converted and filtered to convert the execution plan into a specific data structure (such as a dictionary or JSON) for further analysis or visualization.

[0074] On this basis, in the above initial data cleaning rules, the rules corresponding to the sql data can include any one or more of the above, and the corresponding adjustment can mean adding or deleting any rule. For the execution plan data, the initial data cleaning rule can mean extracting the initial execution plan node and / or key information, and the corresponding adjustment can mean adjusting the extracted execution plan node and / or key information.

[0075] Through the above processing, the target data cleaning rule can be adapted to different businesses, improving the quality and reliability of the data.

[0076] Step 103, calculating the md5 value corresponding to the target task from the first data.

[0077] In a possible implementation, in the case where the initial data of the target task includes sql data and execution plan, the md5 value of the sql data and the md5 value of the execution plan data can be calculated from the first data after cleaning.

[0078] Step 104, according to the new and old task verification rule, verifying the md5 value corresponding to the target task with the md5 values corresponding to the plurality of historical tasks stored in the reference library, to determine the new and old task verification result of the target task.

[0079] In a possible implementation, according to a new-old task verification rule, the md5 value corresponding to the target task is compared with the md5 values corresponding to a plurality of historical tasks stored in the benchmark library, to determine whether there is an md5 value that is equal or similar, that is, a historical task that can be regarded as the same task. If there is, the target task is regarded as an old task; if there is not, the target task is regarded as a new task.

[0080] Optionally, the new-old task verification rule can include:

[0081] For sql data, verification is performed according to the similarity between the task sql data of the task to be verified and the sql data of the historical task;

[0082] For execution plan data, verification is performed according to a target parameter in the execution plan data of the task to be verified, the target parameter including one or more of the following: query complexity information, data size information, index and partition information, data skew and optimization information, engine and tool information.

[0083] Correspondingly, the processing of the step 104 can be as follows:

[0084] The similarity between the md5 value of the sql data of the target task and the md5 value of the sql data of each historical task stored in the benchmark library is determined;

[0085] If there is no similarity greater than the similarity threshold, the target task is determined to be a new task;

[0086] If there is a similarity greater than the similarity threshold, the new-old task characteristics of the target parameter are determined according to the md5 value of the execution plan data of the target task; if the target parameter meets the old task characteristics, the target task is determined to be an old task; if the target parameter meets the new task characteristics, the target task is determined to be a new task.

[0087] In a possible implementation, the sql data can be verified first. The md5 value of the sql data of each historical task in the benchmark library is obtained, and the similarity between the md5 value and the md5 value of the sql data of the target task is calculated. It is determined whether the similarity is greater than the similarity threshold. If it is not greater than, it can be considered that there is no historical task in the benchmark library that is the same as the target task, and the target task is regarded as a new task.

[0088] If it is greater than, it can be considered that there is a historical task in the benchmark library that is the same as or similar to the sql data of the target task. In order to improve the accuracy of new-old task verification, the execution plan data can be further verified. The new-old task characteristics of each target parameter will be introduced below.

[0089] For query complexity information, the new task can have more complex query logic and data processing operations, such as multiple joins, subqueries, window functions, etc., containing more operators and stages. Therefore, the md5 value of the target task execution plan data and the query logic and data processing operation part of the md5 value of each historical task execution plan data can be compared. If the target task has more complex query logic data processing operations, it indicates that the target task can be a new task; if not, it indicates that the target task can be an old task.

[0090] For data size information, the data size characteristics of the old task can be counted through the data statistics information and estimation in the execution plan data md5 value of the historical task, and the data size of the target task can be estimated in the same way. Further, it is judged whether the data size of the target task conforms to the data size characteristics of the old task. If yes, it indicates that the target task can be an old task; if not, it indicates that the target task can be a new task.

[0091] For index and partition information, the execution plan data can include index selection and partition pruning optimization strategies, and the old task can have established indexes and partitions, so the corresponding part of the md5 value can be obtained to judge whether the md5 value of the target task is the same as that of the historical task. If the same, it can be considered that the target task uses the existing index and partition information, indicating that the target task can be an old task; if different, it indicates that the target task can be a new task.

[0092] For data skew and optimization information, the new task can face data skew, in which some data partitions or key values can have uneven distribution. In the execution plan, data redistribution, dynamic bucketing and other optimization strategies can be needed to solve the data skew problem. The old task can have no obvious data skew, and the execution plan can be more simplified. Therefore, the corresponding part of the execution plan data md5 value of the target task can be obtained to judge whether it has data skew and optimization characteristics. If yes, it indicates that the target task can be a new task; if not, it indicates that the target task can be an old task.

[0093] For engine and tool information, the new task can need to use a new Hive engine or tool to meet specific needs, such as using Apache Tez or Apache Spark as an execution engine, or using Hive LLAP to accelerate queries, so in the execution plan, the use and configuration of the new engine or tool need to be considered. The old task can continue to use the traditional Hive engine, and the corresponding part of the execution plan can remain unchanged with the traditional Hive engine. Therefore, the corresponding part of the execution plan data md5 value of the target task can be obtained to judge whether a specific engine and tool are used. If yes, it indicates that the target task can be a new task; if not, it indicates that the target task can be an old task.

[0094] In the new and old task verification rule corresponding to the execution plan data, a plurality of target parameters can be used for verification. In this case, the final new and old task verification result can be determined by comprehensively considering the judgment results corresponding to each target parameter, for example, using the voting method, and the specific comprehensive method is not limited in the embodiment.

[0095] In combination with the above data cleaning rule and the new and old task verification rule, the new and old task verification method provided in the embodiment can determine the target task with only part of the parameters changed and / or only the table partition changed as an old task, and can more accurately distinguish between routine tasks and new tasks. Meanwhile, the sql data and the execution plan data can be used for verification in turn, so that the new and old task verification can be completed more efficiently, accurately and reliably, and the calculation resource consumption when verifying through the execution plan data can be reduced as much as possible.

[0096] Optionally, the target engine can include a Hive hybrid integration and visualization engine, a Presto data query engine and / or a Spark in-memory computing engine.

[0097] In order to adapt to the characteristics of different computing engines, the above new and old task verification process can be performed on different processing nodes. Each computing engine will be introduced below.

[0098] 1. Hive engine

[0099] As shown in the Hive engine schematic diagram in Figure 2 , Beeline is a Hive client; HiveServer2 is a server on a cluster; intelligent-schedule (intelligent scheduling) is a server set between each computing engine, which can be used to perform new and old task verification; and YARN is a cluster resource manager system.

[0100] The user can submit Hive-SQL data of any task to HiveServer2 through Beeline. In HiveServer2, the data is converted into an abstract syntax tree through a Parser module, converted into a query block through a Semantic Analyzer module, converted into a query plan through a Plan module, and submitted to YARN through an Execute module. Before being submitted to YARN, the data of the task can be uploaded to intelligent-schedule for new and old task verification, and after receiving the new and old task verification result fed back by intelligent-schedule, the new and old task verification result is reported to YARN together with the data of the task.

[0101] Optionally, considering that tasks may be lost during SQL transmission, the above steps 101 and 102, namely data cleaning and MD5 value calculation, can be performed in HiveServer2, while the MD5 value of the task is transmitted to intelligent-schedule for verification between new and old tasks.

[0102] 2. Presto engine

[0103] like Figure 3 The diagram shown is a Presto engine illustration. Presto-Cli is the Presto engine client, and Presto-Server is the Presto engine server.

[0104] Users can set task parameters in Presto-Cli and submit them to Presto-Server. In Presto-Server, the task is created using CreateQuery, the task data is converted into an abstract syntax tree using the Parser module, and the task parameters in the abstract syntax tree are retrieved using the getSession module. At this point, the retrieved task parameters can be uploaded to intelligent-schedule for new and old task verification. After receiving the verification results, the query plan is compiled in the Plan module based on the verification results and task parameters. Finally, the query plan is executed through the Execute module.

[0105] 3. Spark Engine

[0106] like Figure 4 The diagram illustrates the Spark engine, where Spark represents the Spark engine client. After a user submits a task on the client, the processing within the client includes the SparkSubmit phase, the SparkContext phase, and the Dataset phase. In the Dataset phase, the task data is uploaded to intelligent-schedule for verification between new and old tasks. Upon receiving the verification results, both the verification results and the task data are submitted to YARN.

[0107] In the embodiment, when it is necessary to perform new-old task verification on a target task, initial data of the target task to be verified in a target service is acquired, the initial data of the target task is cleaned according to a target data cleaning rule of the target service to obtain first data, an md5 value corresponding to the target task is calculated from the first data, and the md5 value corresponding to the target task is verified with md5 values corresponding to a plurality of historical tasks stored in a benchmark library according to a new-old task verification rule to determine a new-old task verification result of the target task. Since the target data cleaning rule can adapt the first data to the new-old task verification rule, and the target data cleaning rule is adapted to the target service, the accuracy of the new-old task verification is improved, and the new-old task verification can be applied to a production environment.

[0108] The embodiment of the present disclosure provides a new-old task verification device, which is used to implement the new-old task verification method. Figure 5 As shown in the figure, the new-old task verification device 500 includes an acquisition module 501, a cleaning module 502, a calculation module 503, and a verification module 504.

[0109] The acquisition module 501 is used to acquire initial data of a target task to be verified in a target service.

[0110] The cleaning module 502 is used to clean the initial data of the target task according to a target data cleaning rule of the target service to obtain first data, wherein the target data cleaning rule is used to adapt the first data to a new-old task verification rule.

[0111] The calculation module 503 is used to calculate an md5 value corresponding to the target task from the first data.

[0112] The verification module 504 is used to verify the md5 value corresponding to the target task with md5 values corresponding to a plurality of historical tasks stored in a benchmark library according to the new-old task verification rule to determine a new-old task verification result of the target task.

[0113] Optionally, the cleaning module 502 is further used to:

[0114] The initial data cleaning rule is used as a first test data cleaning rule, and the following test processing is performed:

[0115] The initial data of a first historical task is cleaned according to a current test data cleaning rule to obtain second data, and an md5 value corresponding to the first historical task is calculated from the second data, wherein the first historical task refers to a historical task of the target service in a first preset historical time length.

[0116] clean the initial data of the second historical task according to the current test data cleaning rule to obtain third data, and calculate the md5 value corresponding to the second historical task from the third data, wherein the second historical task refers to a historical task of the target business in a second preset historical time period;

[0117] verify the md5 value corresponding to the second historical task and the md5 value corresponding to the first historical task based on the new-old task verification rule to determine a test new-old task verification result of the second historical task relative to the first historical task;

[0118] adjust the current test data cleaning rule based on the accuracy of the test new-old task verification result to determine a target data cleaning rule.

[0119] Optionally, the cleaning module 502 is configured to:

[0120] If the accuracy of the test new-old task verification result is less than or equal to a preset threshold, adjust the current test data cleaning rule to obtain a next test data cleaning rule, and repeat the test processing until the accuracy of the test new-old task verification result is greater than the preset threshold.

[0121] If the accuracy of the test new-old task verification result is greater than the preset threshold, the current test data cleaning rule is used as the target data cleaning rule.

[0122] Optionally, the cleaning module 502 is further configured to:

[0123] obtain initial data of a plurality of third historical tasks of the target business in a third preset historical time period;

[0124] clean the initial data of the third historical task according to the target data cleaning rule to obtain fourth data;

[0125] calculate the md5 value corresponding to the third historical task from the fourth data;

[0126] construct the benchmark library based on the md5 values corresponding to the plurality of third historical tasks.

[0127] Optionally, the initial data includes sql data and execution plan data of a task.

[0128] The target data cleaning rule corresponding to the sql data includes any one or more of the following: removing empty characters in the sql data, removing numbers in the sql data, removing single quotes and double quotes in the sql data, removing commas in the sql data, and removing task IDs in the sql data.

[0129] The target data cleaning rule corresponding to the execution plan data comprises: extracting and parsing the execution plan data; cleaning and processing the execution plan data to remove annotations, spaces and line breaks, and extracting specific execution plan nodes and / or key information; and performing data conversion and filtering on the cleaned execution plan data.

[0130] Optionally, the initial data comprises sql data and execution plan data of the task.

[0131] The new and old task verification rule comprises:

[0132] For the sql data, verification is performed according to the similarity between the task sql data of the task to be verified and the sql data of the historical task.

[0133] For the execution plan data, verification is performed according to the target parameter in the execution plan data of the task to be verified, the target parameter comprising one or more of: query complexity information, data size information, index and partition information, data skew and optimization information, and engine and tool information.

[0134] Optionally, the verification module 504 is configured to:

[0135] determine the similarity between the md5 value of the sql data of the target task and the md5 value of the sql data of each historical task stored in the benchmark library;

[0136] If there is no similarity greater than the similarity threshold, the target task is determined to be a new task.

[0137] If there is a similarity greater than the similarity threshold, the new and old task characteristics of the target parameter are determined according to the md5 value of the execution plan data of the target task; if the target parameter meets the old task characteristics, the target task is determined to be an old task; if the target parameter meets the new task characteristics, the target task is determined to be a new task.

[0138] Optionally, the initial data of the target task is obtained by a target engine, and the target engine comprises:

[0139] a Hive hybrid integration and visualization engine, a Presto data query engine and / or a Spark in-memory computing engine

[0140] In this embodiment, whenever new-old task verification needs to be performed on a target task, initial data of the target task to be verified in the target service is acquired, the initial data of the target task is cleaned according to a target data cleaning rule of the target service to obtain first data, an md5 value corresponding to the target task is calculated from the first data, the md5 value corresponding to the target task is verified with md5 values corresponding to a plurality of historical tasks stored in a reference library according to a new-old task verification rule, and a new-old task verification result of the target task is determined. Since the target data cleaning rule can adapt the first data to the new-old task verification rule, and the target data cleaning rule is adapted to the target service, the accuracy of the new-old task verification is improved, and the new-old task verification can be applied to a production environment.

[0141] The example embodiments of the present disclosure further provide an electronic device, comprising: at least one processor; and a memory connected with the at least one processor in communication. The memory stores a computer program capable of being executed by the at least one processor, and the computer program, when executed by the at least one processor, is configured to cause the electronic device to perform the method according to the embodiments of the present disclosure.

[0142] The example embodiments of the present disclosure further provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is configured to cause the computer to perform the method according to the embodiments of the present disclosure.

[0143] The example embodiments of the present disclosure further provide a computer program product comprising a computer program, wherein the computer program, when executed by a processor of a computer, is configured to cause the computer to perform the method according to the embodiments of the present disclosure.

[0144] Reference Figure 6 The structure block diagram of the electronic device 600 which can be a server or a client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent a variety of forms of digital electronic computing devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent a variety of forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections, and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0145] As Figure 6As shown, the electronic device 600 includes a computing unit 601 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 602 or a computer program loaded into a random access memory (RAM) 603 from a storage unit 608. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0146] A plurality of components in the electronic device 600 are connected to the I / O interface 605, including an input unit 606, an output unit 607, a storage unit 608, and a communication unit 609. The input unit 606 can be any type of device that can input information to the electronic device 600, and can receive inputted digital or text information, as well as generate key signal inputs related to user settings and / or function controls of the electronic device. The output unit 607 can be any type of device that can present information, and can include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 608 can include, but is not limited to, a magnetic disk, an optical disk. The communication unit 609 allows the electronic device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0147] The computing unit 601 can be various general and / or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 601 performs various methods and processes described above. For example, in some embodiments, the new-old task verification method can be implemented as a computer software program that is tangibly embodied in a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 600 via the ROM 602 and / or the communication unit 609. In some embodiments, the computing unit 601 can be configured to perform the new-old task verification method by any other appropriate means, such as by means of firmware.

[0148] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or block diagrams. The program code can be embodied entirely on a machine, partially on a machine, fully on a machine, partially on a machine and partially on a remote machine, or fully on a remote machine or server.

[0149] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of a processor, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0150] As used in the present disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus and / or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal that can be used to provide machine instructions and / or data to a programmable processor.

[0151] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0152] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0153] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

Claims

1. A new and old task checking method, characterized in that, The method comprises: obtaining initial data of a target task to be verified in a target service; cleaning the initial data of the target task according to a target data cleaning rule of the target service to obtain first data, wherein the target data cleaning rule is used to adapt the first data to a new-old task verification rule; calculating an md5 value corresponding to the target task from the first data; verifying the md5 value corresponding to the target task with md5 values corresponding to a plurality of historical tasks stored in a reference library according to the new-old task verification rule to determine a new-old task verification result of the target task; wherein the initial data comprises sql data and execution plan data of the task; the new-old task verification rule comprises: for sql data, verifying according to the similarity between the task sql data of the task to be verified and the sql data of the historical task; for execution plan data, verifying according to target parameters in the execution plan data of the task to be verified, the target parameters including one or more of the following: query complexity information, data size information, index and partition information, data skew and optimization information, engine and tool information.

2. The method of claim 1, wherein, The target data cleaning rule is obtained based on the following method: taking the initial data cleaning rule as the first test data cleaning rule, and performing the following test processing: cleaning the initial data of a first historical task according to the current test data cleaning rule to obtain second data, and calculating an md5 value corresponding to the first historical task from the second data, wherein the first historical task refers to a historical task of the target service in a first preset historical time period; cleaning the initial data of a second historical task according to the current test data cleaning rule to obtain third data, and calculating an md5 value corresponding to the second historical task from the third data, wherein the second historical task refers to a historical task of the target service in a second preset historical time period; verifying the md5 value corresponding to the second historical task with the md5 value corresponding to the first historical task according to the new-old task verification rule to determine a test new-old task verification result of the second historical task relative to the first historical task; adjusting the current test data cleaning rule based on the accuracy of the test new-old task verification result to determine the target data cleaning rule.

3. The method of claim 2, wherein, The adjustment of the current test data cleaning rule based on the accuracy of the test new-old task verification result to determine the target data cleaning rule comprises: if the accuracy of the test new-old task verification result is less than or equal to a preset threshold, adjusting the current test data cleaning rule to obtain a next test data cleaning rule, and repeating the test processing until the accuracy of the test new-old task verification result is greater than the preset threshold; if the accuracy of the test new-old task verification result is greater than the preset threshold, taking the current test data cleaning rule as the target data cleaning rule.

4. The method of claim 2, wherein, After determining the target data cleaning rule, the method further comprises: obtaining initial data of a plurality of third historical tasks of the target service in a third preset historical time length; cleaning the initial data of the third historical tasks according to the target data cleaning rule to obtain fourth data; calculating the fourth data to obtain an md5 value corresponding to the third historical task; constructing the benchmark library based on the md5 values corresponding to the plurality of third historical tasks.

5. The method according to any one of claims 1 to 4, characterized in that ； The target data cleaning rule corresponding to the sql data includes any one or more of the following: removing empty characters in the sql data, removing numbers in the sql data, removing single quotes and double quotes in the sql data, removing commas in the sql data, and removing task IDs in the sql data. The target data cleaning rule corresponding to the execution plan data includes: extracting and parsing the execution plan data; cleaning and processing the execution plan data to remove comments, spaces, and line breaks, extracting specific execution plan nodes and / or key information; and performing data conversion and filtering on the cleaned execution plan data.

6. The method of claim 1, wherein, The checking of the md5 value corresponding to the target task and the md5 values corresponding to the plurality of historical tasks stored in the benchmark library according to the new and old task checking rule to determine the new and old task checking result of the target task includes: determining the similarity between the md5 value of the sql data of the target task and the md5 value of the sql data of each historical task stored in the benchmark library; if there is no similarity greater than the similarity threshold, determining that the target task is a new task; if there is a similarity greater than the similarity threshold, determining the new and old task characteristics of the target parameter according to the md5 value of the execution plan data of the target parameter; if the target parameter meets the old task characteristics, determining that the target task is an old task; if the target parameter meets the new task characteristics, determining that the target task is a new task.

7. The method of claim 1, wherein, The initial data of the target task is obtained by a target engine, and the target engine includes: a Hive hybrid integration and visualization engine, a Presto data query engine, and / or a Spark in-memory computing engine.

8. A device for verifying new and old tasks, characterized in that, The device includes: an obtaining module configured to obtain initial data of a target task to be checked in a target service; a cleaning module configured to clean the initial data of the target task according to a target data cleaning rule of the target service to obtain first data, wherein the target data cleaning rule is used to adapt the first data to a new and old task checking rule; a calculating module configured to calculate the first data to obtain an md5 value corresponding to the target task; a checking module configured to check the md5 value corresponding to the target task and md5 values corresponding to a plurality of historical tasks stored in a benchmark library according to a new and old task checking rule to determine a new and old task checking result of the target task; and a determining module configured to determine the new and old task checking result of the target task according to the new and old task checking rule. The initial data includes task SQL data and execution plan data; the new and old task verification rule includes: for the SQL data, verifying according to the similarity between the task SQL data of the task to be verified and the SQL data of the historical task; for the execution plan data, verifying according to the target parameter in the execution plan data of the task to be verified, the target parameter including one or more of the following: query complexity information, data size information, index and partition information, data skew and optimization information, engine and tool information. 9.An electronic device, comprising: a processor; and a memory storing a program, wherein the program includes instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1-7.

10. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7.

Images