Task execution method and device, equipment and storage medium

Abstract

The invention relates to a task execution method and device, equipment and a storage medium, and the method comprises the steps: executing an offline calculation change task in a test environment when an initial offline calculation task is changed; in the execution process of the offline calculation change task, generating test data and determining a target inspection task category corresponding to the test data; querying a target check strategy matched with the target check task category in a preset strategy database; the incidence relation between the preset inspection task category and the preset inspection strategy is stored in the preset strategy database; based on the target check strategy, performing matching check on the test data and the initial data to obtain a target check result; the initial data is execution result data for executing the initial offline calculation task in the application environment; and if the target check result represents that the test data is abnormal, stopping executing the offline calculation change task. The task checking efficiency is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a task execution method, device, equipment and storage medium. Background Art

[0002] Generally, it is necessary to complete the development and release of offline computing tasks in the computing platform. The scheduling system routinely executes computing tasks according to the scheduling strategy configured in the offline computing tasks and stores the output results in the database. An offline computing task contains both code and task configuration information and field information of the final output table. When making iterative changes, it may affect the execution time and resources of the computing task, and may also affect the final output table data, table structure, table primary key and number of table rows. In addition, the various contents affected by the change are in different platforms or systems. Some can be obtained through the function interface query, and some need to be obtained through the query script. It is difficult to obtain comparison results efficiently.

[0003] In the related art, when an offline computing task is changed, the execution time, resource consumption and other information before and after the change are manually queried on the task scheduling platform to check the difference; and the table structure, primary key and other information of the test environment and the online environment are manually queried in the metadata center to check the difference; because the configuration information and table structure checks rely on manual work, the inspection efficiency is low. Summary of the invention

[0004] The present disclosure provides a task execution method, device, equipment and storage medium to at least solve the problem of low efficiency of difference checking before and after changes in offline computing tasks in related technologies. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided a task execution method, including:

[0006] When the initial offline computing task changes, the offline computing change task is executed in the test environment;

[0007] During the execution of the offline calculation change task, generating test data and determining the target inspection task category corresponding to the test data;

[0008] Searching a preset strategy database for a target inspection strategy that matches the target inspection task category; the preset strategy database stores an association relationship between the preset inspection task category and the preset inspection strategy;

[0009] Based on the target inspection strategy, the test data and the initial data are matched and checked to obtain a target inspection result; the initial data is the execution result data of the initial offline computing task executed in the application environment;

[0010] If the target inspection result indicates that the test data is abnormal, the offline calculation change task is stopped.

[0011] In an exemplary embodiment, the method for constructing the preset strategy database includes:

[0012] Get multiple preset inspection task categories;

[0013] Construct a preset inspection strategy corresponding to each preset inspection task category;

[0014] The preset strategy database is constructed according to the association relationship between each preset inspection task category and the preset inspection strategy corresponding to each preset inspection task category.

[0015] In an exemplary embodiment, the plurality of preset inspection task categories include an inspection task category of configuration change, an inspection task category of table information, and an inspection task category of table data, and the step of constructing a preset inspection strategy corresponding to each preset inspection task category includes:

[0016] For the inspection task category of the configuration change, construct a dependency comparison strategy, a task execution time comparison strategy, an execution resource consumption comparison strategy, and a task baseline comparison strategy;

[0017] Generate a preset inspection strategy corresponding to the inspection task category of the configuration change according to the dependency comparison strategy, the task execution time comparison strategy, the execution resource consumption comparison strategy and the task baseline comparison strategy;

[0018] According to the inspection task category of the table information, a field comparison strategy, a partition row number comparison strategy and a primary key comparison strategy are constructed;

[0019] Generate a preset inspection strategy corresponding to the inspection task category of the table information according to the field comparison strategy, partition row number comparison strategy and primary key comparison strategy;

[0020] According to the inspection task category of the table data, a data difference ratio strategy, a data difference detail strategy and a business rule comparison strategy are constructed;

[0021] According to the data difference proportion strategy, the data difference detail strategy and the business rule comparison strategy, a preset inspection strategy corresponding to the inspection task category of the table data is generated.

[0022] In an exemplary embodiment, the method further comprises:

[0023] Parsing the test data and the initial data to obtain dependency comparison data, task execution time comparison data, execution resource consumption comparison data, and task baseline comparison data;

[0024] Based on an unstructured query language, storing the dependency comparison data, the task execution time comparison data, the execution resource consumption comparison data, and the task baseline comparison data in a first database;

[0025] Parse the test data and the initial data to obtain field comparison data, partition row number comparison data, and primary key comparison data;

[0026] Based on the unstructured query language, the field comparison data, the partition row number comparison data and the primary key comparison data are stored in a second database;

[0027] Parse the test data and the initial data to obtain data difference percentage data, data difference detail data and business rule comparison data;

[0028] Based on the structured query language, the data difference proportion data, the data difference detail data and the business rule comparison data are stored in the third database.

[0029] In an exemplary embodiment, if the target inspection task category is the inspection task category of the configuration change, the matching inspection of the test data and the initial data based on the target inspection strategy to obtain the target inspection result includes:

[0030] According to the target inspection strategy corresponding to the inspection task category of the configuration change, calling the preset interface protocol according to the unstructured query language, querying the dependency comparison data corresponding to the dependency comparison strategy in the first database;

[0031] Query the task execution duration comparison data corresponding to the task execution duration comparison strategy; query the execution resource consumption comparison data corresponding to the execution resource consumption comparison strategy; and query the task baseline comparison data corresponding to the task baseline comparison strategy;

[0032] The target inspection result is determined according to the dependency comparison data, the task execution time comparison data, the execution resource consumption comparison data and the task baseline comparison data.

[0033] In an exemplary embodiment, determining the target inspection result according to the dependency comparison data, the task execution time comparison data, the execution resource consumption comparison data, and the task baseline comparison data includes:

[0034] Perform a matching check on the dependency comparison data based on the dependency comparison strategy to obtain a dependency check result;

[0035] Performing a matching check on the task execution time comparison data based on the task execution time comparison strategy to obtain an execution time check result;

[0036] Performing a match check on the execution resource consumption comparison data based on the execution resource consumption comparison strategy to obtain a resource consumption check result;

[0037] Performing a matching check on the task baseline comparison data based on the task baseline comparison strategy to obtain a task baseline check result;

[0038] The target inspection result is determined according to the dependency inspection result, the execution duration inspection result, the resource consumption inspection result and the task baseline inspection result.

[0039] In an exemplary embodiment, if the target inspection task category is the inspection task category of the table information, the matching inspection of the test data and the initial data is performed based on the target inspection strategy to obtain the target inspection result, including:

[0040] According to the target inspection strategy corresponding to the inspection task category of the table information, calling the software development kit to query the field comparison data corresponding to the field comparison strategy in the second database according to the unstructured query language;

[0041] Query the partition row number comparison data corresponding to the partition row number comparison strategy; and query the primary key comparison data corresponding to the primary key comparison strategy;

[0042] The target inspection result is determined according to the field comparison data, the partition row number comparison data and the primary key comparison data.

[0043] In an exemplary implementation, determining the target inspection result according to the field comparison data, the partition row number comparison data, and the primary key comparison data includes:

[0044] Perform a matching check on the field comparison data based on the field comparison strategy to obtain a field check result;

[0045] Perform a matching check on the partition row number comparison data based on the partition row number comparison strategy to obtain a partition row number check result;

[0046] Perform a matching check on the primary key comparison data based on the primary key comparison strategy to obtain a primary key check result;

[0047] The target check result is determined according to the field check result, the partition row number check result and the primary key check result.

[0048] In an exemplary embodiment, if the target inspection task category is the inspection task category of the table data, the matching inspection of the test data and the initial data is performed based on the target inspection strategy to obtain the target inspection result, including:

[0049] According to the target inspection strategy corresponding to the inspection task category of the table data, and according to the structured query language, querying the data difference proportion data corresponding to the data difference proportion strategy in the third database;

[0050] Query the data difference detail data corresponding to the data difference detail strategy;

[0051] Querying the business rule comparison data corresponding to the business rule comparison strategy;

[0052] The target inspection result is determined according to the data difference percentage data, the data difference detail data and the business rule comparison data.

[0053] In an exemplary implementation, determining the target inspection result according to the data difference percentage data, the data difference detail data, and the business rule comparison data includes:

[0054] Perform a matching check on the data difference ratio data based on the data difference ratio strategy to obtain a data difference ratio check result;

[0055] Perform a matching check on the data difference detail data based on the data difference detail strategy to obtain a data difference detail check result;

[0056] Perform a matching check on the business rule comparison data based on the business rule comparison strategy to obtain a business rule comparison check result;

[0057] The target inspection result is determined according to the data difference ratio inspection result, the data difference detail inspection result and the business rule comparison inspection result.

[0058] In an exemplary embodiment, if the target inspection result indicates that the test data is abnormal, the method further includes:

[0059] Adjusting the offline computing change task to obtain an adjusted offline computing change task;

[0060] The adjusted offline computing change task is used again as the offline computing change task, and the process proceeds to the step of executing the offline computing change task in the test environment.

[0061] According to a second aspect of an embodiment of the present disclosure, there is provided a task execution device, including:

[0062] A change task execution module is configured to execute an offline computing change task in a test environment when an initial offline computing task is changed;

[0063] A target inspection category determination module is configured to generate test data and determine a target inspection task category corresponding to the test data during the execution of the offline calculation change task;

[0064] A target strategy query module is configured to query a preset strategy database for a target inspection strategy that matches the target inspection task category; the preset strategy database stores an association relationship between the preset inspection task category and the preset inspection strategy;

[0065] A result determination module is configured to perform a matching check on the test data and initial data based on the target check strategy to obtain a target check result; the initial data is the execution result data of the initial offline computing task executed in the application environment;

[0066] The task stopping module is configured to stop executing the offline calculation change task if the target inspection result indicates that the test data is abnormal.

[0067] In an exemplary embodiment, the device further comprises:

[0068] A preset category acquisition module is configured to execute and acquire a plurality of preset inspection task categories;

[0069] A preset strategy building module is configured to execute and build a preset inspection strategy corresponding to each preset inspection task category;

[0070] The database construction module is configured to construct the preset strategy database according to the association relationship between each preset inspection task category and the preset inspection strategy corresponding to each preset inspection task category.

[0071] In an exemplary embodiment, the plurality of preset inspection task categories include an inspection task category of configuration change, an inspection task category of table information, and an inspection task category of table data, and the preset strategy construction module includes:

[0072] A first comparison strategy construction unit is configured to execute the inspection task category for the configuration change, and to construct a dependency comparison strategy, a task execution time comparison strategy, an execution resource consumption comparison strategy, and a task baseline comparison strategy;

[0073] A first inspection strategy generating unit is configured to execute, according to the dependency comparison strategy, the task execution time comparison strategy, the execution resource consumption comparison strategy and the task baseline comparison strategy, to generate a preset inspection strategy corresponding to the inspection task category of the configuration change;

[0074] A second comparison strategy construction unit is configured to execute a check task category for the table information, and to construct a field comparison strategy, a partition row number comparison strategy, and a primary key comparison strategy;

[0075] A second inspection strategy generating unit is configured to execute a preset inspection strategy corresponding to the inspection task category of the table information according to the field comparison strategy, the partition row number comparison strategy and the primary key comparison strategy;

[0076] A third comparison strategy construction unit is configured to execute a check task category for the table data, and to construct a data difference ratio strategy, a data difference detail strategy, and a business rule comparison strategy;

[0077] The third inspection strategy generating unit is configured to execute a preset inspection strategy corresponding to the inspection task category of the table data according to the data difference proportion strategy, the data difference detail strategy and the business rule comparison strategy.

[0078] In an exemplary embodiment, the device further comprises:

[0079] A first data parsing module is configured to parse the test data and the initial data to obtain dependency comparison data, task execution time comparison data, execution resource consumption comparison data and task baseline comparison data;

[0080] A first data storage module is configured to execute based on an unstructured query language, and store the dependency comparison data, the task execution time comparison data, the execution resource consumption comparison data, and the task baseline comparison data in a first database;

[0081] A second data parsing module is configured to parse the test data and the initial data to obtain field comparison data, partition row number comparison data and primary key comparison data;

[0082] A second data storage module is configured to execute based on an unstructured query language, and store the field comparison data, the partition row number comparison data, and the primary key comparison data in a second database;

[0083] A third data analysis module is configured to analyze the test data and the initial data to obtain data difference percentage data, data difference detail data and business rule comparison data;

[0084] The third data storage module is configured to execute based on the structured query language to store the data difference ratio data, the data difference detail data and the business rule comparison data in the third database.

[0085] In an exemplary embodiment, if the target inspection task category is the configuration change inspection task category, the result determination module includes:

[0086] A dependency comparison data query unit is configured to execute a target inspection strategy corresponding to the inspection task category according to the configuration change, and query the dependency comparison data corresponding to the dependency comparison strategy in the first database according to an unstructured query language by calling a preset interface protocol;

[0087] A baseline comparison data query unit is configured to query the task execution duration comparison data corresponding to the task execution duration comparison strategy; query the execution resource consumption comparison data corresponding to the execution resource consumption comparison strategy; and query the task baseline comparison data corresponding to the task baseline comparison strategy;

[0088] The first target result determination unit is configured to determine the target inspection result according to the dependency comparison data, the task execution time comparison data, the execution resource consumption comparison data and the task baseline comparison data.

[0089] In an exemplary embodiment, the first target result determination unit includes:

[0090] A dependency check result determination subunit is configured to perform a matching check on the dependency comparison data based on the dependency comparison strategy to obtain a dependency check result;

[0091] An execution time check result determination subunit is configured to perform a matching check on the task execution time comparison data based on the task execution time comparison strategy to obtain an execution time check result;

[0092] a resource consumption check result determination subunit, configured to perform a matching check on the execution resource consumption comparison data based on the execution resource consumption comparison strategy to obtain a resource consumption check result;

[0093] A task baseline check result determination subunit is configured to perform a matching check on the task baseline comparison data based on the task baseline comparison strategy to obtain a task baseline check result;

[0094] The first target inspection result determination subunit is configured to determine the target inspection result according to the dependency inspection result, the execution time inspection result, the resource consumption inspection result and the task baseline inspection result.

[0095] In an exemplary embodiment, if the target inspection task category is the inspection task category of the table information, the result determination module includes:

[0096] A field comparison data query unit is configured to execute a target inspection strategy corresponding to the inspection task category according to the table information, and call a software development toolkit to query the field comparison data corresponding to the field comparison strategy in the second database according to an unstructured query language;

[0097] A primary key comparison data query unit is configured to query the partition row number comparison data corresponding to the partition row number comparison strategy; and query the primary key comparison data corresponding to the primary key comparison strategy;

[0098] The second target result determination unit is configured to determine the target inspection result according to the field comparison data, the partition row number comparison data and the primary key comparison data.

[0099] In an exemplary embodiment, the second target result determination unit includes:

[0100] A field check result determination subunit is configured to perform a matching check on the field comparison data based on the field comparison strategy to obtain a field check result;

[0101] A partition row number check result determination subunit is configured to perform a matching check on the partition row number comparison data based on the partition row number comparison strategy to obtain a partition row number check result;

[0102] A primary key check result determination subunit is configured to perform a matching check on the primary key comparison data based on the primary key comparison strategy to obtain a primary key check result;

[0103] The first target check result determination subunit is configured to determine the target check result according to the field check result, the partition row number check result and the primary key check result.

[0104] In an exemplary embodiment, if the target inspection task category is the inspection task category of the table data, the result determination module includes:

[0105] a difference data query unit, configured to execute a target inspection strategy corresponding to the inspection task category of the table data, and query the third database for data difference proportion data corresponding to the data difference proportion strategy according to a structured query language;

[0106] A detail data query unit, configured to query the data difference detail data corresponding to the data difference detail strategy;

[0107] A comparison data query unit, configured to query the business rule comparison data corresponding to the business rule comparison strategy;

[0108] The third target result determination unit is configured to determine the target inspection result according to the data difference proportion data, the data difference detail data and the business rule comparison data.

[0109] In an exemplary embodiment, the third target result determination unit includes:

[0110] A difference result determination subunit is configured to perform a matching check on the data difference ratio data based on the data difference ratio strategy to obtain a data difference ratio check result;

[0111] A detailed inspection result determination subunit is configured to perform a matching check on the data difference detailed data based on the data difference detailed strategy to obtain a data difference detailed inspection result;

[0112] A business check result determination subunit is configured to perform a matching check on the business rule comparison data based on the business rule comparison strategy to obtain a business rule comparison check result;

[0113] The third target result determination subunit is configured to determine the target inspection result according to the data difference ratio inspection result, the data difference detail inspection result and the business rule comparison inspection result.

[0114] In an exemplary embodiment, if the target inspection result indicates that the test data is abnormal, the device further includes:

[0115] An adjustment module is configured to adjust the offline computing change task to obtain an adjusted offline computing change task;

[0116] The jump module is configured to execute the step of re-using the adjusted offline computing change task as the offline computing change task and jump to the step of executing the offline computing change task in the test environment.

[0117] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0118] processor;

[0119] a memory for storing instructions executable by the processor;

[0120] The processor is configured to execute the instructions to implement the task execution method as described above.

[0121] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the task execution method as described above.

[0122] According to a fifth aspect of an embodiment of the present disclosure, there is provided a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the task execution method as described above is implemented.

[0123] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0124] In the present disclosure, when the initial offline computing task is changed, the offline computing change task is executed in a test environment; during the execution of the offline computing change task, test data is generated and the target inspection task category corresponding to the test data is determined; through a pre-constructed preset strategy database, the target inspection strategy matching the target inspection task category is queried; the preset strategy database stores the association between the preset inspection task category and the preset inspection strategy; thereby improving the query efficiency of the target inspection strategy; then, according to the target inspection strategy, the test data and the initial data are matched and checked to obtain the target inspection result; the initial data is the execution result data of the initial offline computing task executed in the application environment; if the target inspection result indicates that the test data is abnormal, the execution of the offline computing change task is stopped. The present disclosure realizes the automated inspection of offline computing change tasks, improves the efficiency of task inspection, and can timely block abnormally executed tasks.

[0125] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0126] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0127] Figure 1 The present invention is an application environment diagram of a task execution method according to an exemplary embodiment.

[0128] Figure 2 The figure is a flowchart of a task execution method according to an exemplary embodiment.

[0129] Figure 3 The present invention is a flowchart of a method for constructing a preset strategy database according to an exemplary embodiment.

[0130] Figure 4The present invention is a flowchart of a method for performing a matching check on the test data and the initial data based on the target check strategy to obtain a target check result according to an exemplary embodiment.

[0131] Figure 5 It is a flowchart of a method for determining the target inspection result according to the dependency comparison data, the task execution time comparison data, the execution resource consumption comparison data and the task baseline comparison data according to an exemplary embodiment.

[0132] Figure 6 It is a structural diagram of a task execution system according to an exemplary embodiment.

[0133] Figure 7 The figure is a schematic diagram of the structure of a big data offline comparison test platform according to an exemplary embodiment.

[0134] Figure 8 The present invention is a flowchart showing the interaction between a comparison test platform and a computing platform according to an exemplary embodiment.

[0135] Fig. 9 The figure is a block diagram of a task execution device according to an exemplary embodiment.

[0136] Fig.10 It is a block diagram of a server according to an exemplary embodiment.

[0137] Fig.11 It is a block diagram of an electronic device for performing a task according to an exemplary embodiment. DETAILED DESCRIPTION

[0138] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0139] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0140] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0141] In the prior art, it is necessary to manually write a check script during testing, and due to differences in table structure and table name, the reusability of the check script written each time is low, the overall efficiency is low, and the comparison rules are not standardized; configuration information and table structure checks rely on manual work, which is inefficient.

[0142] In order to solve the problems of non-standard existing task change comparison rules, the low efficiency of manually writing data comparison scripts and the low efficiency of manual query during the inspection process, the present disclosure provides a task execution method, device, equipment and storage medium to realize automated inspection of offline computing change tasks and improve task inspection efficiency.

[0143] See also Figure 1 , which shows an application environment diagram of a task execution method according to an exemplary embodiment. The application environment may include a server 01 and a client 02.

[0144] Specifically, in the embodiments of this specification, the server 01 may include an independently operated server, or a distributed server, or a server cluster composed of multiple servers, and may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Server 01 may include a network communication unit, a processor, a memory, and the like. Specifically, the server 01 can be used to match and check the test data and initial data generated during the execution of the offline computing change task based on the target inspection strategy, obtain the target inspection result, and send the target inspection result to the client 02.

[0145] Specifically, in the embodiments of this specification, the client 02 may include physical devices such as smart phones, desktop computers, tablet computers, laptop computers, digital assistants, smart wearable devices, and vehicle-mounted terminals, and may also include software running in physical devices, such as web pages provided by some service providers to users, or applications provided by these service providers to users. Specifically, the client 02 may be used to display target inspection results.

[0146] Figure 2 is a flowchart of a task execution method according to an exemplary embodiment. Figure 2As shown, this method can be applied to Figure 1 The server 01 shown includes the following steps.

[0147] In step S201, when the initial offline computing task is changed, the offline computing change task is executed in the test environment.

[0148] In the disclosed embodiment, an initial offline computing task includes both code and configuration information of the task and field information of the final output table, etc. Changes to the initial offline computing task may include but are not limited to tasks such as version upgrades and function improvements of the application corresponding to the task.

[0149] In step S203, during the execution of the offline calculation change task, test data is generated and the target inspection task category corresponding to the test data is determined.

[0150] In the disclosed embodiment, when a task undergoes an iterative change, it may affect the execution time and resources of the computing task, and may also affect the final output table data, table structure, table primary key, and number of table rows, etc. Therefore, it is necessary to measure the impact of this change and find out what is not as expected. One or more preset inspection task categories can be set, and the target inspection task category corresponding to the test data can be determined, where the target inspection task category can be one or more of the multiple preset inspection task categories. During the execution of the offline computing change task, test data can be generated and checked at every preset time interval before the task is executed.

[0151] In step S205, a target inspection strategy matching the target inspection task category is searched in a preset strategy database; the preset strategy database stores the association relationship between the preset inspection task category and the preset inspection strategy.

[0152] In the disclosed embodiment, a preset strategy database storing the association between preset inspection task categories and preset inspection strategies can be pre-built, and then the preset strategy database can be queried for the target inspection strategy matching the target inspection task category, so as to quickly determine the target inspection strategy.

[0153] In step S207, based on the target check strategy, a matching check is performed on the test data and the initial data to obtain a target check result; the initial data is the execution result data of the initial offline computing task executed in the application environment.

[0154] In step S209, if the target inspection result indicates that the test data is abnormal, the offline calculation change task is stopped.

[0155] In the disclosed embodiment, the execution result data of the initial offline computing task executed in the application environment can be obtained to obtain the initial data; then, according to the target inspection strategy, the test data and the initial data are matched and checked to obtain the target inspection result; the target inspection result includes the inspection pass result and the inspection abnormality result; if the target inspection result indicates that the test data is abnormal, the execution of the offline computing change task is stopped.

[0156] In the present disclosure, when the initial offline computing task is changed, the offline computing change task is executed in a test environment; during the execution of the offline computing change task, test data is generated and the target inspection task category corresponding to the test data is determined; through a pre-constructed preset strategy database, the target inspection strategy matching the target inspection task category is queried; the preset strategy database stores the association between the preset inspection task category and the preset inspection strategy; thereby improving the query efficiency of the target inspection strategy; then, according to the target inspection strategy, the test data and the initial data are matched and checked to obtain the target inspection result; the initial data is the execution result data of the initial offline computing task executed in the application environment; if the target inspection result indicates that the test data is abnormal, the execution of the offline computing change task is stopped. The present disclosure realizes the automated inspection of offline computing change tasks, improves the efficiency of task inspection, and can timely block abnormally executed tasks.

[0157] In some embodiments, Figure 3 As shown, Figure 3 A method for constructing a preset strategy database includes:

[0158] S301: Acquire multiple preset inspection task categories;

[0159] S303: Constructing a preset inspection strategy corresponding to each preset inspection task category;

[0160] S305: Constructing the preset strategy database according to the association relationship between each preset inspection task category and the preset inspection strategy corresponding to each preset inspection task category.

[0161] In the disclosed embodiment, a plurality of preset inspection task categories can be pre-constructed according to actual application requirements. The plurality of preset inspection task categories can include but are not limited to inspection task categories for configuration changes, inspection task categories for table information, and inspection task categories for table data. Then, a preset inspection strategy corresponding to each preset inspection task category is constructed; thereby, the preset strategy database is constructed according to the association relationship between each preset inspection task category and the preset inspection strategy corresponding to each preset inspection task category.

[0162] In the disclosed embodiment, the preset strategy database may be constructed in advance based on the association between each preset inspection task category and the preset inspection strategy corresponding to each preset inspection task category, thereby improving the query efficiency of the target inspection strategy.

[0163] In the embodiment of the present disclosure, the plurality of preset inspection task categories include an inspection task category of configuration change, an inspection task category of table information, and an inspection task category of table data. The construction of a preset inspection strategy corresponding to each preset inspection task category includes:

[0164] For the inspection task category of the configuration change, construct a dependency comparison strategy, a task execution time comparison strategy, an execution resource consumption comparison strategy, and a task baseline comparison strategy;

[0165] Generate a preset inspection strategy corresponding to the inspection task category of the configuration change according to the dependency comparison strategy, the task execution time comparison strategy, the execution resource consumption comparison strategy and the task baseline comparison strategy;

[0166] According to the inspection task category of the table information, a field comparison strategy, a partition row number comparison strategy and a primary key comparison strategy are constructed;

[0167] Generate a preset inspection strategy corresponding to the inspection task category of the table information according to the field comparison strategy, partition row number comparison strategy and primary key comparison strategy;

[0168] According to the inspection task category of the table data, a data difference ratio strategy, a data difference detail strategy and a business rule comparison strategy are constructed;

[0169] According to the data difference proportion strategy, the data difference detail strategy and the business rule comparison strategy, a preset inspection strategy corresponding to the inspection task category of the table data is generated.

[0170] In the disclosed embodiment, a plurality of preset inspection task categories may include an inspection task category for configuration changes, an inspection task category for table information, and an inspection task category for table data. For the inspection task category for configuration changes, a dependency comparison strategy, a task execution time comparison strategy, an execution resource consumption comparison strategy, and a task baseline comparison strategy may be constructed; thereby generating a preset inspection strategy corresponding to the inspection task category for configuration changes; for the inspection task category for table information, a field comparison strategy, a partition row number comparison strategy, and a primary key comparison strategy may be constructed; thereby generating a preset inspection strategy corresponding to the inspection task category for table information based on the field comparison strategy, the partition row number comparison strategy, and the primary key comparison strategy; for the inspection task category for table data, a data difference ratio strategy, a data difference detail strategy, and a business rule comparison strategy may be constructed; thereby generating a preset inspection strategy corresponding to the inspection task category for table data. Among them, the data difference ratio refers to the ratio of the difference data before and after the task change in the total data, and the data difference detail refers to the details of the difference data before and after the task change.

[0171] In the disclosed embodiment, preset inspection strategies corresponding to the configuration change inspection task category, table information inspection task category, and table data inspection task category can be constructed respectively, thereby improving the diversity of preset inspection strategies and thus improving the diversity of target inspection strategies.

[0172] In the embodiment of the present disclosure, the method further includes:

[0173] Parsing the test data and the initial data to obtain dependency comparison data, task execution time comparison data, execution resource consumption comparison data, and task baseline comparison data;

[0174] Based on an unstructured query language, storing the dependency comparison data, the task execution time comparison data, the execution resource consumption comparison data, and the task baseline comparison data in a first database;

[0175] Parse the test data and the initial data to obtain field comparison data, partition row number comparison data, and primary key comparison data;

[0176] Based on the unstructured query language, the field comparison data, the partition row number comparison data and the primary key comparison data are stored in a second database;

[0177] Parse the test data and the initial data to obtain data difference percentage data, data difference detail data and business rule comparison data;

[0178] Based on the structured query language, the data difference proportion data, the data difference detail data and the business rule comparison data are stored in the third database.

[0179] In the disclosed embodiments, the database is a collection of large amounts of data that is stored in a computer for a long time, organized, and shareable. The data in the database is organized, described, and stored according to a certain data model, has low redundancy, high data independence, and easy scalability, and can be shared by various users. It is mainly divided into relational databases and non-relational databases. A relational database is a database management system based on a relational model. It uses relationships (tables) to organize data and establishes connections between data through associations between tables. A relational database uses a structured query language to operate and query data. The test data and the initial data can be parsed to obtain dependency comparison data, task execution time comparison data, execution resource consumption comparison data and task baseline comparison data; exemplarily, the dependency comparison data, task execution time comparison data, execution resource consumption comparison data and task baseline comparison data are stored in a first database; the test data and the initial data are parsed to obtain field comparison data, partition row number comparison data and primary key comparison data; based on an unstructured query language, the field comparison data, partition row number comparison data and primary key comparison data are stored in a second database; thereby, a first database and a second database are constructed according to the unstructured query language; the test data and the initial data are then parsed to obtain data difference ratio data, data difference detail data and business rule comparison data; based on a structured query language, the data difference ratio data, data difference detail data and business rule comparison data are stored in a third database to obtain a third database corresponding to the structured query language.

[0180] In the disclosed embodiment, for different data types in the test data and the initial data, different languages ​​can be used to construct databases corresponding to different data types, thereby improving the storage efficiency of the data and facilitating subsequent classification and data query.

[0181] In the embodiment of the present disclosure, if the target inspection task category is the inspection task category of the configuration change, such as Figure 4 As shown, Figure 4 A method for performing a matching check on the test data and the initial data based on the target check strategy to obtain a target check result includes:

[0182] S401: According to the target inspection strategy corresponding to the inspection task category of the configuration change, calling the preset interface protocol according to the unstructured query language, querying the first database for dependency comparison data corresponding to the dependency comparison strategy;

[0183] S403: querying the task execution duration comparison data corresponding to the task execution duration comparison strategy; querying the execution resource consumption comparison data corresponding to the execution resource consumption comparison strategy; and querying the task baseline comparison data corresponding to the task baseline comparison strategy;

[0184] S405: Determine the target inspection result according to the dependency comparison data, the task execution time comparison data, the execution resource consumption comparison data and the task baseline comparison data.

[0185] In the disclosed embodiment, if the target inspection task category is the configuration change inspection task category, multiple sub-strategies corresponding to the category can be obtained, so as to query the comparison data in the corresponding database for data comparison and obtain the target inspection result; the preset interface protocol may include but is not limited to the remote procedure call protocol. For the configuration change inspection task category, the dependency comparison data, the task execution time comparison data, the execution resource consumption comparison data and the task baseline comparison data can be queried in the first database according to the unstructured query language, so as to obtain the target inspection result according to the comparison results of each comparison data.

[0186] In some embodiments, before step S401, it may also include: obtaining query account information, where the query account may be a terminal account that makes a task change request, and then determining whether the query account information has access rights to the first database; if the query account has access rights, executing step S401, based on the target inspection strategy corresponding to the inspection task category of the configuration change, calling a preset interface protocol according to an unstructured query language, to query the first database for dependency comparison data corresponding to the dependency comparison strategy.

[0187] In some embodiments, after determining that the query account has access rights, resource allocation may be further performed to filter out idle resources to perform data matching check tasks.

[0188] In the disclosed embodiment, for the inspection task category of configuration changes, the dependency comparison data, the task execution time comparison data, the execution resource consumption comparison data and the task baseline comparison data can be queried in the first database according to the unstructured query language, thereby improving the data query efficiency and the efficiency of determining the target inspection results.

[0189] In the present disclosure, Figure 5 As shown, Figure 5 A flowchart of a method for determining the target inspection result according to the dependency comparison data, the task execution time comparison data, the execution resource consumption comparison data and the task baseline comparison data, the method comprising:

[0190] S4051: performing a matching check on the dependency comparison data based on the dependency comparison strategy to obtain a dependency check result;

[0191] S4053: performing a matching check on the task execution time comparison data based on the task execution time comparison strategy to obtain an execution time check result;

[0192] S4055: Perform a matching check on the execution resource consumption comparison data based on the execution resource consumption comparison strategy to obtain a resource consumption check result;

[0193] S4057: Perform a matching check on the task baseline comparison data based on the task baseline comparison strategy to obtain a task baseline check result;

[0194] S4059: Determine the target inspection result according to the dependency inspection result, the execution time inspection result, the resource consumption inspection result and the task baseline inspection result.

[0195] In the disclosed embodiment, for dependency comparison data, a dependency comparison strategy can be used to perform a matching check to obtain a dependency check result; for task execution time comparison data, a task execution time comparison strategy can be used to perform a matching check to obtain an execution time check result; for execution resource consumption comparison data, an execution resource consumption comparison strategy can be used to perform a matching check to obtain a resource consumption check result; for task baseline comparison data, a task baseline comparison strategy can be used to perform a matching check to obtain a task baseline check result. The dependency check result, the execution time check result, the resource consumption check result, and the task baseline check result can all include a check pass result and a check abnormality result; if there is at least one check abnormality result among the dependency check result, the execution time check result, the resource consumption check result, and the task baseline check result, the target check result is determined to be a check abnormality result, and the data with abnormalities after the task change can be quickly located; if the dependency check result, the execution time check result, the resource consumption check result, and the task baseline check result are all check pass results, the target check result can be determined to be a check pass result.

[0196] In the disclosed embodiment, if the target inspection task category is the configuration change inspection task category, the target inspection result corresponding to the task category can be quickly determined based on the dependency inspection result, the execution time inspection result, the resource consumption inspection result and the task baseline inspection result.

[0197] In the embodiment of the present disclosure, if the target inspection task category is the inspection task category of the table information, the matching inspection is performed on the test data and the initial data based on the target inspection strategy to obtain the target inspection result, including:

[0198] According to the target inspection strategy corresponding to the inspection task category of the table information, calling the software development kit to query the field comparison data corresponding to the field comparison strategy in the second database according to the unstructured query language;

[0199] Query the partition row number comparison data corresponding to the partition row number comparison strategy; and query the primary key comparison data corresponding to the primary key comparison strategy;

[0200] The target inspection result is determined according to the field comparison data, the partition row number comparison data and the primary key comparison data.

[0201] In an embodiment of the present disclosure, if the target inspection task category is the inspection task category of the table information, multiple strategies corresponding to the inspection task category can be obtained, including a field comparison strategy, a partition row number comparison strategy, and a primary key comparison strategy, and then the corresponding data is queried in the second database according to the multiple strategies to determine the target inspection result.

[0202] In the disclosed embodiment, for the inspection task category of table information, a software development kit can be called to query the field comparison data, the partition row number comparison data and the primary key comparison data in the second database according to an unstructured query language, thereby improving data query efficiency and improving the efficiency of determining target inspection results.

[0203] In some embodiments, before inquiring the field comparison data corresponding to the field comparison strategy in the second database according to the target inspection strategy corresponding to the inspection task category of the table information and calling the software development kit according to the unstructured query language, it can also include: obtaining the query account information, the query account can be the terminal account that makes the task change request, and then judging whether the query account information has the access right to the second database, if the query account has the access right, executing the target inspection strategy corresponding to the inspection task category according to the table information, calling the software development kit according to the unstructured query language, and querying the field comparison data corresponding to the field comparison strategy in the second database. In some embodiments, after determining that the query account has the access right, resource allocation can be further performed to filter out the matching inspection tasks for the idle resource execution data.

[0204] In the embodiment of the present disclosure, determining the target inspection result according to the field comparison data, the partition row number comparison data and the primary key comparison data includes:

[0205] Perform a matching check on the field comparison data based on the field comparison strategy to obtain a field check result;

[0206] Perform a matching check on the partition row number comparison data based on the partition row number comparison strategy to obtain a partition row number check result;

[0207] Perform a matching check on the primary key comparison data based on the primary key comparison strategy to obtain a primary key check result;

[0208] The target check result is determined according to the field check result, the partition row number check result and the primary key check result.

[0209] In the disclosed embodiment, the field comparison data can be matched and checked according to the field comparison strategy to obtain the field check result; the partition row number comparison data can be matched and checked according to the partition row number comparison strategy to obtain the partition row number check result; the primary key comparison data can be matched and checked according to the primary key comparison strategy to obtain the primary key check result; and then the target check result can be determined according to the field check result, the partition row number check result and the primary key check result; if there is at least one abnormal check result among the field check result, the partition row number check result and the primary key check result, the target check result corresponding to the task category is determined to be the abnormal check result; if the field check result, the partition row number check result and the primary key check result are all normal check results, the target check result corresponding to the task category is determined to be the normal check result.

[0210] In the disclosed embodiment, if the target inspection task category is the inspection task category of the table information, the target inspection result corresponding to the task category can be quickly determined based on the field inspection result, the partition row number inspection result and the primary key inspection result.

[0211] In the embodiment of the present disclosure, if the target inspection task category is the inspection task category of the table data, the matching inspection is performed on the test data and the initial data based on the target inspection strategy to obtain the target inspection result, including:

[0212] According to the target inspection strategy corresponding to the inspection task category of the table data, and according to the structured query language, querying the data difference proportion data corresponding to the data difference proportion strategy in the third database;

[0213] Query the data difference detail data corresponding to the data difference detail strategy;

[0214] Querying the business rule comparison data corresponding to the business rule comparison strategy;

[0215] The target inspection result is determined according to the data difference percentage data, the data difference detail data and the business rule comparison data.

[0216] In an embodiment of the present disclosure, if the target inspection task category is the inspection task category of the table data, the data corresponding to each strategy under the category can be obtained according to the third database, and the data difference proportion data corresponding to the data difference proportion strategy can be queried in the third database; the data difference detail data corresponding to the data difference detail strategy can be queried; and the business rule comparison data corresponding to the business rule comparison strategy can be queried, so as to facilitate further determination of the target inspection result corresponding to the inspection task category of the table data.

[0217] In the disclosed embodiment, for the inspection task category of the table data, the data difference percentage data, the data difference detail data and the business rule comparison data can be searched in the third database according to the structured query language to determine the target inspection result, thereby improving the data query efficiency and the efficiency of determining the target inspection result.

[0218] In some embodiments, according to the target inspection strategy corresponding to the inspection task category of the table data, according to the structured query language, before querying the data difference ratio data corresponding to the data difference ratio strategy in the third database, it can also include: obtaining the query account information, the query account can be the terminal account that makes the task change request, and then judging whether the query account information has the access right to the third database, if the query account has the access right, executing the target inspection strategy corresponding to the inspection task category of the table data, and querying the data difference ratio data corresponding to the data difference ratio strategy in the third database according to the structured query language. In some embodiments, after determining that the query account has the access right, resource allocation can be further performed to screen out the matching inspection tasks for the idle resource execution data.

[0219] In the embodiment of the present disclosure, determining the target inspection result according to the data difference proportion data, the data difference detail data and the business rule comparison data includes:

[0220] Perform a matching check on the data difference ratio data based on the data difference ratio strategy to obtain a data difference ratio check result;

[0221] Perform a matching check on the data difference detail data based on the data difference detail strategy to obtain a data difference detail check result;

[0222] Perform a matching check on the business rule comparison data based on the business rule comparison strategy to obtain a business rule comparison check result;

[0223] The target inspection result is determined according to the data difference ratio inspection result, the data difference detail inspection result and the business rule comparison inspection result.

[0224] In the disclosed embodiment, the data difference ratio data can be matched and checked according to the data difference ratio strategy to obtain the data difference ratio check result; then the data difference detail data can be matched and checked according to the data difference detail strategy to obtain the data difference detail check result; and the business rule comparison data can be matched and checked according to the business rule comparison strategy to obtain the business rule comparison check result; thereby, the target inspection result is determined according to the data difference ratio check result, the data difference detail check result and the business rule comparison check result. If there is at least one abnormal inspection result among the data difference ratio check result, the data difference detail check result and the business rule comparison check result, the target inspection result is determined to be the abnormal inspection result. If the data difference ratio check result, the data difference detail check result and the business rule comparison check result are all normal inspection results, the target inspection result is determined to be the normal inspection result.

[0225] In the disclosed embodiment, for the inspection task category of table data, the target inspection result can be quickly determined according to the data difference ratio inspection result, the data difference detail inspection result and the business rule comparison inspection result.

[0226] In some embodiments, the above method further comprises:

[0227] If the target inspection result indicates that the test data is normal, continue to execute the offline calculation change task.

[0228] In the embodiment of the present disclosure, if the target inspection result is a check pass result, it indicates that the test data is normal, and the offline calculation change task continues to be executed.

[0229] In the embodiment of the present disclosure, if the target inspection result indicates that the test data is abnormal, the method further includes:

[0230] Adjusting the offline computing change task to obtain an adjusted offline computing change task;

[0231] The adjusted offline computing change task is used again as the offline computing change task, and the process proceeds to the step of executing the offline computing change task in the test environment.

[0232] In an embodiment of the present disclosure, if the target inspection result characterizes that the test data is abnormal, the abnormal data can be obtained, so that the offline computing change task can be adjusted according to the data category corresponding to the abnormal data, and the adjusted offline computing change task can be re-used as the offline computing change task, and the process can be skipped to the step of executing the offline computing change task in the test environment, and the inspection of the adjusted task can be repeated.

[0233] In the disclosed embodiment, if the target inspection result indicates that the test data is abnormal, the offline computing change task can be adjusted and the adjusted task can be executed, thereby repeatedly inspecting the adjusted task to improve task adjustment efficiency and task execution efficiency.

[0234] In some embodiments, the inspection task category of configuration changes, the inspection task category of table information, and the inspection task category of table data can also be inspected separately to obtain the target inspection result corresponding to the task; when the inspection result of at least one of the inspection task category of configuration changes, the inspection task category of table information, and the inspection task category of table data is an abnormal inspection result, the target inspection result is determined to be an abnormal inspection result; if the inspection results corresponding to the inspection task category of configuration changes, the inspection task category of table information, and the inspection task category of table data are all normal inspection results, the target inspection result is determined to be a normal inspection result.

[0235] In an exemplary embodiment, the present disclosure provides a task execution method, the method comprising:

[0236] When the initial offline computing task changes, the offline computing change task is executed in the test environment;

[0237] During the execution of the offline calculation change task, generating test data and determining the target inspection task category corresponding to the test data;

[0238] Searching a preset strategy database for a target inspection strategy that matches the target inspection task category; the preset strategy database stores an association relationship between the preset inspection task category and the preset inspection strategy;

[0239] According to the target inspection strategy corresponding to the inspection task category of the configuration change, calling the preset interface protocol according to the unstructured query language, querying the dependency comparison data corresponding to the dependency comparison strategy in the first database;

[0240] Query the task execution duration comparison data corresponding to the task execution duration comparison strategy; query the execution resource consumption comparison data corresponding to the execution resource consumption comparison strategy; and query the task baseline comparison data corresponding to the task baseline comparison strategy;

[0241] Perform a matching check on the dependency comparison data based on the dependency comparison strategy to obtain a dependency check result;

[0242] Performing a matching check on the task execution time comparison data based on the task execution time comparison strategy to obtain an execution time check result;

[0243] Performing a match check on the execution resource consumption comparison data based on the execution resource consumption comparison strategy to obtain a resource consumption check result;

[0244] Performing a matching check on the task baseline comparison data based on the task baseline comparison strategy to obtain a task baseline check result;

[0245] The target inspection result is determined according to the dependency inspection result, the execution duration inspection result, the resource consumption inspection result and the task baseline inspection result.

[0246] According to the target inspection strategy corresponding to the inspection task category of the table information, calling the software development kit to query the field comparison data corresponding to the field comparison strategy in the second database according to the unstructured query language;

[0247] Query the partition row number comparison data corresponding to the partition row number comparison strategy; and query the primary key comparison data corresponding to the primary key comparison strategy;

[0248] Perform a matching check on the field comparison data based on the field comparison strategy to obtain a field check result;

[0249] Perform a matching check on the partition row number comparison data based on the partition row number comparison strategy to obtain a partition row number check result;

[0250] Perform a matching check on the primary key comparison data based on the primary key comparison strategy to obtain a primary key check result;

[0251] The target check result is determined according to the field check result, the partition row number check result and the primary key check result.

[0252] According to the target inspection strategy corresponding to the inspection task category of the table data, and according to the structured query language, querying the data difference proportion data corresponding to the data difference proportion strategy in the third database;

[0253] Query the data difference detail data corresponding to the data difference detail strategy;

[0254] Querying the business rule comparison data corresponding to the business rule comparison strategy;

[0255] Perform a matching check on the data difference ratio data based on the data difference ratio strategy to obtain a data difference ratio check result;

[0256] Perform a matching check on the data difference detail data based on the data difference detail strategy to obtain a data difference detail check result;

[0257] Perform a matching check on the business rule comparison data based on the business rule comparison strategy to obtain a business rule comparison check result;

[0258] The target inspection result is determined according to the data difference ratio inspection result, the data difference detail inspection result and the business rule comparison inspection result.

[0259] If at least one target inspection result indicates that the test data is abnormal, the offline calculation change task is stopped.

[0260] like Figure 6 As shown, Figure 6 The present invention is a structural diagram of a task execution system, including an automated data comparison library, a scheduling platform (a first database), a metadata center (a second database), and a data warehouse (a third database); wherein the automated data comparison library is used to store unstructured query language type comparison rules and structured query language type comparison rules, and to query comparison data in the scheduling platform or the metadata center according to the unstructured query language type comparison rules and execute the comparison rules to automatically generate a comparison report; and is also used to query comparison data in the data warehouse according to the structured query language type comparison rules and execute the comparison rules to automatically generate a comparison report.

[0261] like Figure 7 As shown, Figure 7The present invention is a structural diagram of a big data offline comparison test platform; the big data offline comparison test platform includes a comparison rule execution module, a comparison rule management module, a comparison management module and a comparison authority management module; wherein the comparison rule execution module includes a structured query language rule executor and an unstructured query language rule executor; the comparison rule management module includes configuration comparison rules, table information comparison rules and table data comparison rules, and the comparison management module can be used to create comparison tasks, query comparison tasks, generate comparison reports and push comparison results; the comparison authority management module can be used for table authority checking and resource authority management; the comparison rule management module: in the big data offline comparison test platform, the content that needs to be compared and checked is standardized and unified, and defined as different comparison rules. For example, the check items for computing task configuration changes are classified as configuration comparison rules, in which the task dependency differences, the task execution time differences caused by this change, and the computing resource differences caused by the change can be checked. Other types of rules include table information comparison rules, table data comparison rules and custom comparison rules. By managing the comparison rules, the comparison check is more standardized; comparison rule execution module: the comparison rules maintained in the comparison rule management module are general rules, and the output tables of different big data offline computing tasks can be used. The comparison rule execution module automatically executes the comparison rules according to a test table and an online table that need to be compared. During the execution process, the execution strategy is determined according to the type of different rule configurations, and the execution authority check and execution resource allocation are combined with the authority management module; comparison task management module: the comparison check of the execution results of a big data offline computing task can be regarded as a comparison task. In the comparison task management module, you can create a comparison task (multiple comparison rules), query historical comparison tasks, query the comparison report of a comparison, etc. The comparison test platform provides external interfaces, which support comparison task creation, comparison execution status check, comparison execution result confirmation, and comparison report query through these interfaces. Through these external interfaces, the comparison test platform can be easily associated with the computing platform or management platform.

[0262] like Figure 8 As shown, Figure 8 The invention discloses an interactive flow chart of a comparison test platform and a computing platform, wherein the method comprises: the computing platform is developed based on an offline computing task, generates an offline computing task, runs the offline computing task, connects the computing platform to the comparison test platform, and performs a data comparison check before and after the task change by executing the comparison strategy of the comparison test platform to determine whether there is an abnormal difference. If so, a comparison report is pushed to the task change user; the changed user can further determine whether the comparison result is normal. If the result is determined to be normal, the subsequent processes such as testing or publishing are continued; if the result is determined to be abnormal, the table change task is readjusted, and the adjustment task is restarted and checked.

[0263] The present disclosure provides an automatic comparison test method suitable for big data offline computing: when a big data offline computing task changes, the differences in execution time, table information, table data, etc. before and after the change are automatically compared and a visual comparison report is provided. On the one hand, it provides a reference for evaluating the impact of the change; on the other hand, when an abnormal comparison result is found, the offline computing task is readjusted and the next stage of the process is automatically blocked. The standardization of the comparison and inspection content ensures the effective coverage of the comparison and inspection; and there is no need to manually write inspection scripts, and the comparison test is automatically performed, which greatly improves the efficiency of the comparison and inspection; rich external interface support makes it convenient to expand according to actual needs.

[0264] Fig. 9 is a block diagram of a task execution device according to an exemplary embodiment. Fig. 9 , the device comprises:

[0265] The change task execution module 910 is configured to execute the offline computing change task in the test environment when the initial offline computing task is changed;

[0266] The target inspection category determination module 920 is configured to generate test data and determine the target inspection task category corresponding to the test data during the execution of the offline calculation change task;

[0267] The target strategy query module 930 is configured to query a preset strategy database for a target inspection strategy that matches the target inspection task category; the preset strategy database stores an association relationship between the preset inspection task category and the preset inspection strategy;

[0268] The result determination module 940 is configured to perform a matching check on the test data and the initial data based on the target check strategy to obtain a target check result; the initial data is the execution result data of the initial offline computing task executed in the application environment;

[0269] The task stopping module 950 is configured to stop executing the offline calculation change task if the target inspection result indicates that the test data is abnormal.

[0270] In an exemplary embodiment, the device further comprises:

[0271] A preset category acquisition module is configured to execute and acquire a plurality of preset inspection task categories;

[0272] A preset strategy building module is configured to execute and build a preset inspection strategy corresponding to each preset inspection task category;

[0273] The database construction module is configured to construct the preset strategy database according to the association relationship between each preset inspection task category and the preset inspection strategy corresponding to each preset inspection task category.

[0274] In an exemplary embodiment, the plurality of preset inspection task categories include an inspection task category of configuration change, an inspection task category of table information, and an inspection task category of table data, and the preset strategy construction module includes:

[0275] A first comparison strategy construction unit is configured to execute the inspection task category for the configuration change, and to construct a dependency comparison strategy, a task execution time comparison strategy, an execution resource consumption comparison strategy, and a task baseline comparison strategy;

[0276] A first inspection strategy generating unit is configured to execute, according to the dependency comparison strategy, the task execution time comparison strategy, the execution resource consumption comparison strategy and the task baseline comparison strategy, to generate a preset inspection strategy corresponding to the inspection task category of the configuration change;

[0277] A second comparison strategy construction unit is configured to execute a check task category for the table information, and to construct a field comparison strategy, a partition row number comparison strategy, and a primary key comparison strategy;

[0278] A second inspection strategy generating unit is configured to execute a preset inspection strategy corresponding to the inspection task category of the table information according to the field comparison strategy, the partition row number comparison strategy and the primary key comparison strategy;

[0279] A third comparison strategy construction unit is configured to execute a check task category for the table data, and to construct a data difference ratio strategy, a data difference detail strategy, and a business rule comparison strategy;

[0280] The third inspection strategy generating unit is configured to execute a preset inspection strategy corresponding to the inspection task category of the table data according to the data difference proportion strategy, the data difference detail strategy and the business rule comparison strategy.

[0281] In an exemplary embodiment, the device further comprises:

[0282] A first data parsing module is configured to parse the test data and the initial data to obtain dependency comparison data, task execution time comparison data, execution resource consumption comparison data and task baseline comparison data;

[0283] A first data storage module is configured to execute based on an unstructured query language, and store the dependency comparison data, the task execution time comparison data, the execution resource consumption comparison data, and the task baseline comparison data in a first database;

[0284] A second data parsing module is configured to parse the test data and the initial data to obtain field comparison data, partition row number comparison data and primary key comparison data;

[0285] A second data storage module is configured to execute based on an unstructured query language, and store the field comparison data, the partition row number comparison data, and the primary key comparison data in a second database;

[0286] A third data analysis module is configured to analyze the test data and the initial data to obtain data difference percentage data, data difference detail data and business rule comparison data;

[0287] The third data storage module is configured to execute based on the structured query language to store the data difference ratio data, the data difference detail data and the business rule comparison data in the third database.

[0288] In an exemplary embodiment, if the target inspection task category is the configuration change inspection task category, the result determination module includes:

[0289] A dependency comparison data query unit is configured to execute a target inspection strategy corresponding to the inspection task category according to the configuration change, and query the dependency comparison data corresponding to the dependency comparison strategy in the first database according to an unstructured query language by calling a preset interface protocol;

[0290] A baseline comparison data query unit is configured to query the task execution duration comparison data corresponding to the task execution duration comparison strategy; query the execution resource consumption comparison data corresponding to the execution resource consumption comparison strategy; and query the task baseline comparison data corresponding to the task baseline comparison strategy;

[0291] The first target result determination unit is configured to determine the target inspection result according to the dependency comparison data, the task execution time comparison data, the execution resource consumption comparison data and the task baseline comparison data.

[0292] In an exemplary embodiment, the first target result determination unit includes:

[0293] A dependency check result determination subunit is configured to perform a matching check on the dependency comparison data based on the dependency comparison strategy to obtain a dependency check result;

[0294] An execution time check result determination subunit is configured to perform a matching check on the task execution time comparison data based on the task execution time comparison strategy to obtain an execution time check result;

[0295] a resource consumption check result determination subunit, configured to perform a matching check on the execution resource consumption comparison data based on the execution resource consumption comparison strategy to obtain a resource consumption check result;

[0296] A task baseline check result determination subunit is configured to perform a matching check on the task baseline comparison data based on the task baseline comparison strategy to obtain a task baseline check result;

[0297] The first target inspection result determination subunit is configured to determine the target inspection result according to the dependency inspection result, the execution time inspection result, the resource consumption inspection result and the task baseline inspection result.

[0298] In an exemplary embodiment, if the target inspection task category is the inspection task category of the table information, the result determination module includes:

[0299] A field comparison data query unit is configured to execute a target inspection strategy corresponding to the inspection task category according to the table information, and call a software development toolkit to query the field comparison data corresponding to the field comparison strategy in the second database according to an unstructured query language;

[0300] A primary key comparison data query unit is configured to query the partition row number comparison data corresponding to the partition row number comparison strategy; and query the primary key comparison data corresponding to the primary key comparison strategy;

[0301] The second target result determination unit is configured to determine the target inspection result according to the field comparison data, the partition row number comparison data and the primary key comparison data.

[0302] In an exemplary embodiment, the second target result determination unit includes:

[0303] A field check result determination subunit is configured to perform a matching check on the field comparison data based on the field comparison strategy to obtain a field check result;

[0304] A partition row number check result determination subunit is configured to perform a matching check on the partition row number comparison data based on the partition row number comparison strategy to obtain a partition row number check result;

[0305] A primary key check result determination subunit is configured to perform a matching check on the primary key comparison data based on the primary key comparison strategy to obtain a primary key check result;

[0306] The first target check result determination subunit is configured to determine the target check result according to the field check result, the partition row number check result and the primary key check result.

[0307] In an exemplary embodiment, if the target inspection task category is the inspection task category of the table data, the result determination module includes:

[0308] a difference data query unit, configured to execute a target inspection strategy corresponding to the inspection task category of the table data, and query the third database for data difference proportion data corresponding to the data difference proportion strategy according to a structured query language;

[0309] A detail data query unit, configured to query the data difference detail data corresponding to the data difference detail strategy;

[0310] A comparison data query unit, configured to query the business rule comparison data corresponding to the business rule comparison strategy;

[0311] The third target result determination unit is configured to determine the target inspection result according to the data difference proportion data, the data difference detail data and the business rule comparison data.

[0312] In an exemplary embodiment, the third target result determination unit includes:

[0313] A difference result determination subunit is configured to perform a matching check on the data difference ratio data based on the data difference ratio strategy to obtain a data difference ratio check result;

[0314] A detailed inspection result determination subunit is configured to perform a matching check on the data difference detailed data based on the data difference detailed strategy to obtain a data difference detailed inspection result;

[0315] A business check result determination subunit is configured to perform a matching check on the business rule comparison data based on the business rule comparison strategy to obtain a business rule comparison check result;

[0316] The third target result determination subunit is configured to determine the target inspection result according to the data difference ratio inspection result, the data difference detail inspection result and the business rule comparison inspection result.

[0317] In an exemplary embodiment, if the target inspection result indicates that the test data is abnormal, the device further includes:

[0318] An adjustment module is configured to adjust the offline computing change task to obtain an adjusted offline computing change task;

[0319] The jump module is configured to execute the step of re-using the adjusted offline computing change task as the offline computing change task and jump to the step of executing the offline computing change task in the test environment.

[0320] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0321] In an exemplary embodiment, an electronic device is also provided, including a processor; a memory for storing processor executable instructions; wherein, when the processor is configured to execute the instructions stored in the memory, the task execution method provided in any of the above embodiments is implemented.

[0322] The electronic device may be a terminal, a server or a similar computing device. For example, the electronic device is a server. Fig.10 is a block diagram of an electronic device according to an exemplary embodiment. Fig.10 As shown, the server 1000 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1010 (the processor 1010 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 1030 for storing data, and one or more storage media 1020 (such as one or more mass storage devices) for storing application programs 1023 or data 1022. Among them, the memory 1030 and the storage medium 1020 can be short-term storage or permanent storage. The program stored in the storage medium 1020 may include one or more modules, each of which may include a series of instruction operations on the server. Furthermore, the central processing unit 1010 can be configured to communicate with the storage medium 1020 and execute a series of instruction operations in the storage medium 1020 on the server 1000. The server 1000 may also include one or more power supplies 1060, one or more wired or wireless network interfaces 1050, one or more input and output interfaces 1040, and / or one or more operating systems 1021, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0323] The input / output interface 1040 may be used to receive or send data via a network. The specific example of the network may include a wireless network provided by a communication provider of the server 1000. In one example, the input / output interface 1040 includes a network adapter (Network Interface Controller, NIC), which may be connected to other network devices through a base station so as to communicate with the Internet. In one example, the input / output interface 1040 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0324] It can be understood by those skilled in the art that Fig.10 The structure shown is only for illustration and does not limit the structure of the above electronic device. Fig.10 More or fewer components as shown, or with Fig.10 Different configurations are shown.

[0325] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 1030 including instructions, and the instructions can be executed by the processor 1010 of the device 1000 to complete the above method. Alternatively, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0326] In an exemplary embodiment, a computer program product is also provided, including a computer program, and when the computer program is executed by a processor, the task execution method provided in any of the above embodiments is implemented.

[0327] Fig.11 is a block diagram of an electronic device for task execution according to an exemplary embodiment. The electronic device may be a terminal, and its internal structure diagram may be as shown in FIG. Fig.11As shown. The electronic device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a task execution method is implemented. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covering the display screen, or a key, trackball or touchpad provided on the housing of the electronic device, or an external keyboard, touchpad or mouse, etc.

[0328] Those skilled in the art will understand that Fig.11 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present disclosure, and does not constitute a limitation on the electronic device to which the scheme of the present disclosure is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0329] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0330] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0331] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A task execution method, characterized in that: The method comprises: When the initial offline computing task changes, the offline computing change task is executed in the test environment; During the execution of the offline calculation change task, generating test data and determining the target inspection task category corresponding to the test data; Searching a preset strategy database for a target inspection strategy that matches the target inspection task category; the preset strategy database stores an association relationship between the preset inspection task category and the preset inspection strategy; Based on the target inspection strategy, the test data and the initial data are matched and checked to obtain a target inspection result; the initial data is the execution result data of the initial offline computing task executed in the application environment; If the target inspection result indicates that the test data is abnormal, the offline calculation change task is stopped.

2. The method according to claim 1, characterized in that The method for constructing the preset strategy database includes: Get multiple preset inspection task categories; Construct a preset inspection strategy corresponding to each preset inspection task category; The preset strategy database is constructed according to the association relationship between each preset inspection task category and the preset inspection strategy corresponding to each preset inspection task category.

3. The method according to claim 2, characterized in that The plurality of preset inspection task categories include a configuration change inspection task category, a table information inspection task category, and a table data inspection task category. The construction of a preset inspection strategy corresponding to each preset inspection task category includes: For the inspection task category of the configuration change, construct a dependency comparison strategy, a task execution time comparison strategy, an execution resource consumption comparison strategy, and a task baseline comparison strategy; Generate a preset inspection strategy corresponding to the inspection task category of the configuration change according to the dependency comparison strategy, the task execution time comparison strategy, the execution resource consumption comparison strategy and the task baseline comparison strategy; According to the inspection task category of the table information, a field comparison strategy, a partition row number comparison strategy and a primary key comparison strategy are constructed; Generate a preset inspection strategy corresponding to the inspection task category of the table information according to the field comparison strategy, partition row number comparison strategy and primary key comparison strategy; According to the inspection task category of the table data, a data difference ratio strategy, a data difference detail strategy and a business rule comparison strategy are constructed; According to the data difference proportion strategy, the data difference detail strategy and the business rule comparison strategy, a preset inspection strategy corresponding to the inspection task category of the table data is generated.

4. The method according to claim 3, characterized in that The method further comprises: Parsing the test data and the initial data to obtain dependency comparison data, task execution time comparison data, execution resource consumption comparison data, and task baseline comparison data; Based on an unstructured query language, storing the dependency comparison data, the task execution time comparison data, the execution resource consumption comparison data, and the task baseline comparison data in a first database; Parse the test data and the initial data to obtain field comparison data, partition row number comparison data, and primary key comparison data; Based on the unstructured query language, the field comparison data, the partition row number comparison data and the primary key comparison data are stored in a second database; Parse the test data and the initial data to obtain data difference percentage data, data difference detail data and business rule comparison data; Based on the structured query language, the data difference proportion data, the data difference detail data and the business rule comparison data are stored in the third database.

5. The method according to claim 4, characterized in that If the target inspection task category is the inspection task category of the configuration change, performing a matching check on the test data and the initial data based on the target inspection strategy to obtain a target inspection result includes: According to the target inspection strategy corresponding to the inspection task category of the configuration change, calling the preset interface protocol according to the unstructured query language, querying the dependency comparison data corresponding to the dependency comparison strategy in the first database; Query the task execution duration comparison data corresponding to the task execution duration comparison strategy; query the execution resource consumption comparison data corresponding to the execution resource consumption comparison strategy; and query the task baseline comparison data corresponding to the task baseline comparison strategy; The target inspection result is determined according to the dependency comparison data, the task execution time comparison data, the execution resource consumption comparison data and the task baseline comparison data.

6. The method according to claim 5, characterized in that The determining the target inspection result according to the dependency comparison data, the task execution time comparison data, the execution resource consumption comparison data and the task baseline comparison data includes: Perform a matching check on the dependency comparison data based on the dependency comparison strategy to obtain a dependency check result; Performing a matching check on the task execution time comparison data based on the task execution time comparison strategy to obtain an execution time check result; Performing a match check on the execution resource consumption comparison data based on the execution resource consumption comparison strategy to obtain a resource consumption check result; Performing a matching check on the task baseline comparison data based on the task baseline comparison strategy to obtain a task baseline check result; The target inspection result is determined according to the dependency inspection result, the execution duration inspection result, the resource consumption inspection result and the task baseline inspection result.

7. The method according to claim 4, characterized in that If the target inspection task category is the inspection task category of the table information, performing a matching check on the test data and the initial data based on the target inspection strategy to obtain a target inspection result includes: According to the target inspection strategy corresponding to the inspection task category of the table information, calling the software development kit to query the field comparison data corresponding to the field comparison strategy in the second database according to the unstructured query language; Query the partition row number comparison data corresponding to the partition row number comparison strategy; and query the primary key comparison data corresponding to the primary key comparison strategy; The target inspection result is determined according to the field comparison data, the partition row number comparison data and the primary key comparison data.

8. The method according to claim 7, characterized in that The determining the target inspection result according to the field comparison data, the partition row number comparison data and the primary key comparison data includes: Perform a matching check on the field comparison data based on the field comparison strategy to obtain a field check result; Perform a matching check on the partition row number comparison data based on the partition row number comparison strategy to obtain a partition row number check result; Perform a matching check on the primary key comparison data based on the primary key comparison strategy to obtain a primary key check result; The target check result is determined according to the field check result, the partition row number check result and the primary key check result.

9. The method according to claim 4, characterized in that If the target inspection task category is the inspection task category of the table data, performing a matching check on the test data and the initial data based on the target inspection strategy to obtain a target inspection result includes: According to the target inspection strategy corresponding to the inspection task category of the table data, and according to the structured query language, querying the data difference proportion data corresponding to the data difference proportion strategy in the third database; Query the data difference detail data corresponding to the data difference detail strategy; Querying the business rule comparison data corresponding to the business rule comparison strategy; The target inspection result is determined according to the data difference percentage data, the data difference detail data and the business rule comparison data.

10. The method according to claim 9, characterized in that Determining the target inspection result according to the data difference proportion data, the data difference detail data and the business rule comparison data includes: Perform a matching check on the data difference ratio data based on the data difference ratio strategy to obtain a data difference ratio check result; Perform a matching check on the data difference detail data based on the data difference detail strategy to obtain a data difference detail check result; Perform a matching check on the business rule comparison data based on the business rule comparison strategy to obtain a business rule comparison check result; The target inspection result is determined according to the data difference ratio inspection result, the data difference detail inspection result and the business rule comparison inspection result.

11. The method according to claim 1, characterized in that: If the target inspection result indicates that the test data is abnormal, the method further includes: Adjusting the offline computing change task to obtain an adjusted offline computing change task; The adjusted offline computing change task is used again as the offline computing change task, and the process proceeds to the step of executing the offline computing change task in the test environment.

12. A task execution device, characterized in that: include: A change task execution module is configured to execute an offline computing change task in a test environment when an initial offline computing task is changed; A target inspection category determination module is configured to generate test data and determine a target inspection task category corresponding to the test data during the execution of the offline calculation change task; A target strategy query module is configured to query a preset strategy database for a target inspection strategy that matches the target inspection task category; The preset strategy database stores the association relationship between the preset inspection task category and the preset inspection strategy; A result determination module is configured to perform a matching check on the test data and initial data based on the target check strategy to obtain a target check result; the initial data is the execution result data of the initial offline computing task executed in the application environment; The task stopping module is configured to stop executing the offline calculation change task if the target inspection result indicates that the test data is abnormal.

13. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the task execution method as described in any one of claims 1-11.

14. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the task execution method as described in any one of claims 1 to 11.

Images