Task execution methods, devices, equipment and storage media

Abstract

This disclosure relates to a task execution method, apparatus, device, and storage medium. The method includes: executing an offline computing change task in a test environment when an initial offline computing task changes; generating test data and determining the target inspection task category corresponding to the test data during the execution of the offline computing change task; querying a target inspection strategy matching the target inspection task category in a preset strategy database; storing the association between preset inspection task categories and preset inspection strategies in the preset strategy database; performing a matching check on the test data and initial data based on the target inspection strategy to obtain a target inspection result; the initial data is the execution result data of the initial offline computing task executed in the application environment; if the target inspection result indicates that the test data is abnormal, stopping the execution of the offline computing change task. This disclosure improves task inspection efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a task execution method, apparatus, device, and storage medium. Background Technology

[0002] Typically, offline computing tasks need to be developed and deployed within a computing platform. The scheduling system routinely executes these tasks according to the scheduling policies configured within them, storing the results in a database. An offline computing task includes code, configuration information, and the field information of the final output table. Iterative changes may affect the execution time and resources of the computing task, as well as the final output table data, structure, primary key, and number of rows. Furthermore, the affected components exist on different platforms or systems; some can be obtained through user interfaces, while others require query scripts, making efficient comparison of results difficult.

[0003] In related technologies, when an offline computing task changes, the differences are checked by manually querying information such as execution time and resource consumption before and after the change on the task scheduling platform; and by manually querying information such as table structure and primary key of the test environment and the online environment in the metadata center. However, since the configuration information and table structure checks rely on manual methods, the checking efficiency is low. Summary of the Invention

[0004] This disclosure provides a task execution method, apparatus, device, and storage medium to at least solve the problem of low efficiency in checking differences before and after changes to offline computing tasks in related technologies. The technical solution of this disclosure is as follows:

[0005] According to a first aspect of the present disclosure, a task execution method is provided, comprising:

[0006] When the initial offline computing task changes, execute the changed offline computing task in the test environment;

[0007] During the execution of the offline computation change task, test data is generated and the target inspection task category corresponding to the test data is determined;

[0008] The system queries the preset strategy database for target inspection strategies that match the target inspection task category; the preset strategy database stores the association between preset inspection task categories and preset inspection strategies.

[0009] Based on the target inspection strategy, the test data and the initial data are matched and inspected to obtain the target inspection result; the initial data is the execution result data of the initial offline computing task executed in the application environment;

[0010] If the target inspection result indicates that the test data is abnormal, stop executing the offline calculation change task.

[0011] In one exemplary embodiment, the method for constructing the preset policy database includes:

[0012] Retrieve multiple preset inspection task categories;

[0013] Construct a preset inspection strategy for each preset inspection task category;

[0014] The preset strategy database is constructed based on the association between each preset inspection task category and the preset inspection strategy corresponding to each preset inspection task category.

[0015] In one exemplary implementation, the plurality of preset inspection task categories include inspection task categories for configuration changes, inspection task categories for table information, and inspection task categories for table data. The step of constructing a preset inspection strategy corresponding to each preset inspection task category includes:

[0016] For the types of inspection tasks related to the configuration changes, construct dependency comparison strategies, task execution time comparison strategies, execution resource consumption comparison strategies, and task baseline comparison strategies;

[0017] Based on the dependency comparison strategy, task execution time comparison strategy, execution resource consumption comparison strategy, and task baseline comparison strategy, a preset inspection strategy corresponding to the configuration change inspection task category is generated.

[0018] For the types of tasks involving the inspection of the table information, construct field comparison strategies, partition row count comparison strategies, and primary key comparison strategies;

[0019] Based on the field comparison strategy, partition row count comparison strategy, and primary key comparison strategy, a preset inspection strategy corresponding to the inspection task category of the table information is generated.

[0020] For the inspection task categories of the table data, construct data difference percentage strategy, data difference detail strategy, and business rule comparison strategy;

[0021] Based on the data difference percentage strategy, data difference detail strategy, and business rule comparison strategy, a preset inspection strategy corresponding to the inspection task category of the table data is generated.

[0022] In one exemplary embodiment, the method further includes:

[0023] By analyzing the test data and the initial data, we obtain dependency comparison data, task execution time comparison data, execution resource consumption comparison data, and task baseline comparison data.

[0024] Based on an unstructured query language, the dependency comparison data, task execution time comparison data, execution resource consumption comparison data, and task baseline comparison data are stored in the first database;

[0025] The test data and the initial data are analyzed to obtain field comparison data, partition row count comparison data, and primary key comparison data.

[0026] Based on an unstructured query language, the field comparison data, partition row count comparison data, and primary key comparison data are stored in a second database;

[0027] The test data and the initial data are analyzed to obtain data difference percentage data, data difference detail data, and business rule comparison data.

[0028] Based on a structured query language, the data difference percentage data, data difference details data, and business rule comparison data are stored in a third database.

[0029] In an exemplary implementation, if the target inspection task category is the configuration change inspection task category, the step of performing a matching check on the test data and the initial data based on the target inspection strategy to obtain the target inspection result includes:

[0030] According to the target inspection strategy corresponding to the inspection task category of the configuration change, the preset interface protocol is invoked to query the dependency comparison data corresponding to the dependency comparison strategy in the first database using an unstructured query language;

[0031] Query the task execution time comparison data corresponding to the task execution time comparison strategy; query the execution resource consumption comparison data corresponding to the execution resource consumption comparison strategy; and query the task baseline comparison data corresponding to the task baseline comparison strategy.

[0032] The target inspection result is determined based on the dependency comparison data, the task execution time comparison data, the execution resource consumption comparison data, and the task baseline comparison data.

[0033] In one exemplary implementation, determining the target inspection result based on the dependency comparison data, the task execution time comparison data, the execution resource consumption comparison data, and the task baseline comparison data includes:

[0034] Based on the dependency comparison strategy, the dependency comparison data is matched and checked to obtain the dependency check results;

[0035] Based on the task execution time comparison strategy, the task execution time comparison data is matched and checked to obtain the execution time check result;

[0036] Based on the execution resource consumption comparison strategy, the execution resource consumption comparison data is matched and checked to obtain the resource consumption check result;

[0037] Based on the task baseline comparison strategy, the task baseline comparison data is matched and checked to obtain the task baseline check result.

[0038] The target check result is determined based on the dependency check result, the execution time check result, the resource consumption check result, and the task baseline check result.

[0039] In an exemplary implementation, if the target inspection task category is the inspection task category of the table information, the step of performing a matching inspection on the test data and the initial data based on the target inspection strategy to obtain the target inspection result includes:

[0040] Based on the target inspection strategy corresponding to the inspection task category of the table information, the software development kit is invoked to query the field comparison data corresponding to the field comparison strategy in the second database using an unstructured query language;

[0041] Query the partition row count comparison data corresponding to the partition row count comparison strategy; and query the primary key comparison data corresponding to the primary key comparison strategy.

[0042] The target inspection result is determined based on the field comparison data, the partition row count comparison data, and the primary key comparison data.

[0043] In one exemplary implementation, determining the target inspection result based on the field comparison data, the partition row count comparison data, and the primary key comparison data includes:

[0044] Based on the field comparison strategy, the field comparison data is matched and checked to obtain the field check results;

[0045] Based on the partition row count comparison strategy, the partition row count comparison data is matched and checked to obtain the partition row count check result;

[0046] Based on the primary key comparison strategy, the primary key comparison data is matched and checked to obtain the primary key check result;

[0047] The target inspection result is determined based on the field inspection result, the partition row count inspection result, and the primary key inspection result.

[0048] In an exemplary implementation, if the target inspection task category is the inspection task category of the table data, the step of performing a matching inspection on the test data and the initial data based on the target inspection strategy to obtain the target inspection result includes:

[0049] Based on the target inspection strategy corresponding to the inspection task category of the table data, and using structured query language, query the data difference percentage data corresponding to the data difference percentage strategy in the third database;

[0050] Query the data difference details corresponding to the aforementioned data difference details strategy;

[0051] Query the business rule comparison data corresponding to the business rule comparison strategy;

[0052] The target inspection result is determined based on the data difference percentage data, the data difference details data, and the business rule comparison data.

[0053] In one exemplary implementation, determining the target inspection result based on the data difference percentage data, the data difference detail data, and the business rule comparison data includes:

[0054] Based on the data difference ratio strategy, a matching check is performed on the data difference ratio data to obtain the data difference ratio check result.

[0055] Based on the aforementioned data difference detail strategy, a matching check is performed on the data difference detail data to obtain the data difference detail check results;

[0056] Based on the business rule comparison strategy, the business rule comparison data is matched and checked to obtain the business rule comparison check result.

[0057] The target inspection result is determined based on the data difference percentage inspection result, the data difference detail inspection result, and the business rule comparison inspection result.

[0058] In an exemplary embodiment, if the target inspection result indicates that the test data is abnormal, the method further includes:

[0059] The offline computing change task is adjusted to obtain the adjusted offline computing change task;

[0060] The adjusted offline computing change task is re-established as the offline computing change task, and the process jumps to the step of executing the offline computing change task in the test environment.

[0061] According to a second aspect of the present disclosure, a task execution apparatus is provided, comprising:

[0062] The task execution module has been modified to execute an offline computing change task in the test environment when the initial offline computing task changes.

[0063] The target inspection category determination module is configured to generate test data and determine the target inspection task category corresponding to the test data during the execution of the offline computation change task.

[0064] The target strategy query module is configured to query a preset strategy database for target inspection strategies that match the target inspection task category; the preset strategy database stores the association between preset inspection task categories and preset inspection strategies.

[0065] The result determination module is configured to perform a matching check on the test data and the initial data based on the target checking strategy to obtain the target checking result; the initial data is the execution result data of the initial offline computing task executed in the application environment;

[0066] The task stop module is configured to stop executing the offline calculation change task if the target check result indicates that the test data is abnormal.

[0067] In one exemplary embodiment, the apparatus further includes:

[0068] The preset category acquisition module is configured to acquire multiple preset inspection task categories;

[0069] The preset strategy building module is configured to execute the preset inspection strategy corresponding to each preset inspection task category;

[0070] The database construction module is configured to construct the preset strategy database based on the association between each preset inspection task category and the preset inspection strategy corresponding to each preset inspection task category.

[0071] In one exemplary embodiment, the plurality of preset inspection task categories include a configuration change inspection task category, a table information inspection task category, and a table data inspection task category, and the preset strategy construction module includes:

[0072] The first comparison strategy construction unit is configured to perform inspection task categories for the configuration changes and construct a dependency comparison strategy, a task execution time comparison strategy, an execution resource consumption comparison strategy, and a task baseline comparison strategy.

[0073] The first inspection strategy generation unit is configured to generate a preset inspection strategy corresponding to the inspection task category of the configuration change based on the dependency comparison strategy, task execution time comparison strategy, execution resource consumption comparison strategy and task baseline comparison strategy.

[0074] The second comparison strategy construction unit is configured to perform inspection task categories for the table information and construct field comparison strategy, partition row number comparison strategy and primary key comparison strategy.

[0075] The second inspection strategy generation unit is configured to execute a preset inspection strategy corresponding to the inspection task category of the table information based on the field comparison strategy, the partition row number comparison strategy and the primary key comparison strategy.

[0076] The third comparison strategy construction unit is configured to perform inspection task categories for the table data and construct data difference ratio strategy, data difference detail strategy and business rule comparison strategy.

[0077] The third inspection strategy generation unit is configured to execute a preset inspection strategy corresponding to the inspection task category of the table data based on the data difference ratio strategy, data difference details strategy, and business rule comparison strategy.

[0078] In one exemplary embodiment, the apparatus further includes:

[0079] The first data parsing module is configured to parse the test data and the initial data to obtain dependency comparison data, task execution time comparison data, execution resource consumption comparison data and task baseline comparison data.

[0080] The first data storage module is configured to execute an unstructured query language to store the dependency comparison data, task execution time comparison data, execution resource consumption comparison data, and task baseline comparison data into the first database.

[0081] The second data parsing module is configured to parse the test data and the initial data to obtain field comparison data, partition row number comparison data and primary key comparison data.

[0082] The second data storage module is configured to execute an unstructured query language to store the field comparison data, partition row number comparison data, and primary key comparison data into the second database;

[0083] The third data parsing module is configured to parse the test data and the initial data to obtain data difference percentage data, data difference detail data, and business rule comparison data.

[0084] The third data storage module is configured to execute a structured query language to store the data difference percentage data, data difference details data, and business rule comparison data into the third database.

[0085] In an exemplary implementation, if the target inspection task category is the configuration change inspection task category, the result determination module includes:

[0086] The dependency comparison data query unit is configured to execute the target inspection strategy corresponding to the inspection task category changed according to the configuration, and call the preset interface protocol to query the dependency comparison data corresponding to the dependency comparison strategy in the first database according to the unstructured query language.

[0087] The baseline comparison data query unit is configured to query the task execution time comparison data corresponding to the task execution time comparison strategy; query the execution resource consumption comparison data corresponding to the execution resource consumption comparison strategy; and query the task baseline comparison data corresponding to the task baseline comparison strategy.

[0088] The first target result determination unit is configured to determine the target check result based on the dependency comparison data, the task execution time comparison data, the execution resource consumption comparison data, and the task baseline comparison data.

[0089] In one exemplary embodiment, the first target result determination unit includes:

[0090] The dependency check result determination subunit is configured to perform a matching check on the dependency comparison data based on the dependency comparison strategy to obtain the dependency check result;

[0091] The execution time check result determination subunit is configured to perform a matching check on the task execution time comparison data based on the task execution time comparison strategy to obtain the execution time check result;

[0092] The resource consumption check result determination subunit is configured to perform a matching check on the execution resource consumption comparison data based on the execution resource consumption comparison strategy to obtain the resource consumption check result.

[0093] The task baseline check result determination subunit is configured to perform a matching check on the task baseline comparison data based on the task baseline comparison strategy to obtain the task baseline check result.

[0094] The first target check result determination subunit is configured to determine the target check result based on the dependency check result, the execution time check result, the resource consumption check result, and the task baseline check result.

[0095] In an exemplary implementation, if the target inspection task category is the inspection task category of the table information, the result determination module includes:

[0096] The field comparison data query unit is configured to execute the target inspection strategy corresponding to the inspection task category based on the table information, and call the software development kit to query the field comparison data corresponding to the field comparison strategy in the second database according to the unstructured query language;

[0097] The primary key comparison data query unit is configured to perform a query for the partition row count comparison data corresponding to the partition row count comparison strategy; and to query the primary key comparison data corresponding to the primary key comparison strategy.

[0098] The second target result determination unit is configured to determine the target inspection result based on the field comparison data, the partition row number comparison data, and the primary key comparison data.

[0099] In one exemplary embodiment, the second target result determination unit includes:

[0100] The field check result determination subunit is configured to perform a matching check on the field comparison data based on the field comparison strategy to obtain the field check result;

[0101] The partition row count check result determination subunit is configured to perform a matching check on the partition row count comparison data based on the partition row count comparison strategy to obtain the partition row count check result.

[0102] The primary key check result determination subunit is configured to perform a matching check on the primary key comparison data based on the primary key comparison strategy to obtain the primary key check result.

[0103] The first target inspection result determination subunit is configured to determine the target inspection result based on the field inspection result, the partition row count inspection result, and the primary key inspection result.

[0104] In an exemplary implementation, if the target inspection task category is the inspection task category of the table data, the result determination module includes:

[0105] The difference data query unit is configured to execute the target inspection strategy corresponding to the inspection task category of the table data, and query the data difference percentage data corresponding to the data difference percentage strategy in the third database according to the structured query language.

[0106] The detailed data query unit is configured to query the detailed data difference data corresponding to the data difference detailed strategy;

[0107] The comparison data query unit is configured to query the business rule comparison data corresponding to the business rule comparison strategy.

[0108] The third target result determination unit is configured to determine the target inspection result based on the data difference ratio data, the data difference details data, and the business rule comparison data.

[0109] In one exemplary embodiment, the third target result determination unit includes:

[0110] The difference result determination subunit is configured to perform a matching check on the data difference ratio data based on the data difference ratio strategy to obtain the data difference ratio check result.

[0111] The detailed inspection result determination subunit is configured to perform a matching check on the data difference detail data based on the data difference detail strategy to obtain the data difference detail inspection result;

[0112] The business inspection result determination subunit is configured to perform a matching check on the business rule comparison data based on the business rule comparison strategy to obtain the business rule comparison inspection result.

[0113] The third target result determination subunit is configured to determine the target inspection result based on the data difference ratio inspection result, the data difference detail inspection result, and the business rule comparison inspection result.

[0114] In one exemplary embodiment, if the target inspection result indicates that the test data is abnormal, the apparatus further includes:

[0115] The adjustment module is configured to perform adjustments to the offline computing change task to obtain the adjusted offline computing change task;

[0116] The jump module is configured to execute the step of re-establishing the adjusted offline computing change task as the offline computing change task and jumping to the step of executing the offline computing change task in the test environment.

[0117] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0118] processor;

[0119] Memory used to store the processor's executable instructions;

[0120] The processor is configured to execute the instructions to implement the task execution method described above.

[0121] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that, when instructions in the computer-readable storage medium are executed by an electronic device processor, enables the electronic device to perform the task execution method described above.

[0122] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the task execution method described above.

[0123] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0124] In this disclosure, when the initial offline computing task changes, the offline computing change task is executed in a test environment. During the execution of the offline computing change task, test data is generated and the target inspection task category corresponding to the test data is determined. A pre-built preset strategy database is used to query target inspection strategies that match the target inspection task category. The preset strategy database stores the association between preset inspection task categories and preset inspection strategies, thereby improving the query efficiency of target inspection strategies. Then, according to the target inspection strategy, the test data and initial data are matched and checked to obtain the target inspection result. The initial data is the execution result data of the initial offline computing task executed in the application environment. If the target inspection result indicates that the test data is abnormal, the execution of the offline computing change task is stopped. This disclosure achieves automated checking of offline computing change tasks, improves task checking efficiency, and can promptly block abnormally executed tasks.

[0125] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0126] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0127] Figure 1 This is an application environment diagram illustrating a task execution method according to an exemplary embodiment.

[0128] Figure 2 This is a flowchart illustrating a task execution method according to an exemplary embodiment.

[0129] Figure 3 This is a flowchart illustrating a method for constructing a preset strategy database according to an exemplary embodiment.

[0130] Figure 4This is a flowchart illustrating a method for performing a matching check on the test data and initial data based on the target checking strategy, according to an exemplary embodiment, to obtain the target checking result.

[0131] Figure 5 This is a flowchart illustrating a method for determining the target inspection result based on the dependency comparison data, the task execution time comparison data, the execution resource consumption comparison data, and the task baseline comparison data, according to an exemplary embodiment.

[0132] Figure 6 This is a schematic diagram illustrating the structure of a task execution system according to an exemplary embodiment.

[0133] Figure 7 This is a schematic diagram illustrating the structure of a big data offline comparison testing platform according to an exemplary embodiment.

[0134] Figure 8 This is an interactive flowchart illustrating a comparative testing platform and a computing platform according to an exemplary embodiment.

[0135] Figure 9 This is a block diagram illustrating a task execution device according to an exemplary embodiment.

[0136] Figure 10 This is a block diagram illustrating a server according to an exemplary embodiment.

[0137] Figure 11 This is a block diagram illustrating an electronic device for performing a task according to an exemplary embodiment. Detailed Implementation

[0138] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0139] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0140] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0141] In existing technologies, testing requires manually writing inspection scripts. Due to differences in table structure and table names, the reusability of each inspection script is low, resulting in low overall efficiency and non-standard comparison rules. Configuration information and table structure checks rely on manual processes, which is inefficient.

[0142] To address the issues of non-standardized existing task change comparison rules, low efficiency of manually writing data comparison scripts, and low efficiency of manual querying during the inspection process, this disclosure provides a task execution method, apparatus, device, and storage medium to automate the inspection of offline computation change tasks and improve task inspection efficiency.

[0143] Please see Figure 1 The diagram illustrates an application environment for a task execution method according to an exemplary embodiment. The application environment may include a server 01 and a client 02.

[0144] Specifically, in the embodiments of this specification, server 01 may include a standalone server, a distributed server, or a server cluster composed of multiple servers. It may also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Server 01 may include a network communication unit, a processor, and memory, etc. Specifically, server 01 can be used to perform matching checks on the test data and initial data generated during the execution of offline computing change tasks based on a target checking strategy, obtain target checking results, and send the target checking results to client 02.

[0145] Specifically, in this embodiment of the specification, the client 02 may include physical devices such as smartphones, desktop computers, tablets, laptops, digital assistants, smart wearable devices, and in-vehicle terminals, and may also include software running on the physical device, such as web pages provided to users by some service providers, or applications provided to users by such service providers. Specifically, the client 02 can be used to display target inspection results.

[0146] Figure 2 This is a flowchart illustrating a task execution method according to an exemplary embodiment, such as... Figure 2As shown, this method can be applied to Figure 1 The server 01 shown includes the following steps.

[0147] In step S201, when the initial offline computing task changes, the offline computing change task is executed in the test environment.

[0148] In this embodiment of the disclosure, an initial offline computing task includes code, task configuration information, and field information of the final output table. Changes to the initial offline computing task may include, but are not limited to, version upgrades or feature improvements to the application corresponding to the task.

[0149] In step S203, during the execution of the offline calculation change task, test data is generated and the target inspection task category corresponding to the test data is determined.

[0150] In this embodiment of the disclosure, when a task undergoes iterative changes, it may affect the execution time and resources of the computation task, and may also affect the final output table data, table structure, table primary key, and number of rows. Therefore, it is necessary to measure the impact of this change and identify any discrepancies with expectations. One or more preset inspection task categories can be set, and a target inspection task category corresponding to the test data can be determined. The target inspection task category can be one or more of multiple preset inspection task categories. During the execution of the offline computation change task, test data can be generated and checked at preset time intervals before task execution.

[0151] In step S205, a target inspection strategy matching the target inspection task category is queried in the preset strategy database; the preset strategy database stores the association between preset inspection task categories and preset inspection strategies.

[0152] In this embodiment of the disclosure, a preset strategy database that stores the association between preset inspection task categories and preset inspection strategies can be pre-constructed. Then, the target inspection strategy that matches the target inspection task category can be queried in the preset strategy database, thereby quickly determining the target inspection strategy.

[0153] In step S207, based on the target inspection strategy, the test data and the initial data are matched and inspected to obtain the target inspection result; the initial data is the execution result data of the initial offline computing task executed in the application environment.

[0154] In step S209, if the target inspection result indicates that the test data is abnormal, the offline calculation change task is stopped.

[0155] In this embodiment of the disclosure, the execution result data of the initial offline computing task executed in the application environment can be obtained to obtain initial data; then, according to the target checking strategy, the test data and the initial data are matched and checked to obtain the target checking result; the target checking result includes the check pass result and the check anomaly result; if the target checking result indicates that the test data is abnormal, the execution of the offline computing change task is stopped.

[0156] In this disclosure, when the initial offline computing task changes, the offline computing change task is executed in a test environment. During the execution of the offline computing change task, test data is generated and the target inspection task category corresponding to the test data is determined. A pre-built preset strategy database is used to query target inspection strategies that match the target inspection task category. The preset strategy database stores the association between preset inspection task categories and preset inspection strategies, thereby improving the query efficiency of target inspection strategies. Then, according to the target inspection strategy, the test data and initial data are matched and checked to obtain the target inspection result. The initial data is the execution result data of the initial offline computing task executed in the application environment. If the target inspection result indicates that the test data is abnormal, the execution of the offline computing change task is stopped. This disclosure achieves automated checking of offline computing change tasks, improves task checking efficiency, and can promptly block abnormally executed tasks.

[0157] In some embodiments, such as Figure 3 As shown, Figure 3 A method for constructing a pre-defined strategy database includes:

[0158] S301: Obtain multiple preset inspection task categories;

[0159] S303: Construct the preset inspection strategy corresponding to each preset inspection task category;

[0160] S305: Construct the preset strategy database based on the association between each preset inspection task category and the preset inspection strategy corresponding to each preset inspection task category.

[0161] In this embodiment of the disclosure, multiple preset inspection task categories can be pre-constructed according to actual application needs. These multiple preset inspection task categories may include, but are not limited to, inspection task categories for configuration changes, inspection task categories for table information, and inspection task categories for table data. Then, a preset inspection strategy corresponding to each preset inspection task category is constructed. Based on the association between each preset inspection task category and the preset inspection strategy corresponding to each preset inspection task category, the preset strategy database is constructed.

[0162] In this embodiment of the disclosure, the preset strategy database can be constructed in advance based on the association between each preset inspection task category and the preset inspection strategy corresponding to each preset inspection task category, thereby improving the query efficiency of the target inspection strategy.

[0163] In this embodiment of the disclosure, the plurality of preset inspection task categories include configuration change inspection task categories, table information inspection task categories, and table data inspection task categories. The construction of a preset inspection strategy corresponding to each preset inspection task category includes:

[0164] For the types of inspection tasks related to the configuration changes, construct dependency comparison strategies, task execution time comparison strategies, execution resource consumption comparison strategies, and task baseline comparison strategies;

[0165] Based on the dependency comparison strategy, task execution time comparison strategy, execution resource consumption comparison strategy, and task baseline comparison strategy, a preset inspection strategy corresponding to the configuration change inspection task category is generated.

[0166] For the types of tasks involving the inspection of the table information, construct field comparison strategies, partition row count comparison strategies, and primary key comparison strategies;

[0167] Based on the field comparison strategy, partition row count comparison strategy, and primary key comparison strategy, a preset inspection strategy corresponding to the inspection task category of the table information is generated.

[0168] For the inspection task categories of the table data, construct data difference percentage strategy, data difference detail strategy, and business rule comparison strategy;

[0169] Based on the data difference percentage strategy, data difference detail strategy, and business rule comparison strategy, a preset inspection strategy corresponding to the inspection task category of the table data is generated.

[0170] In this embodiment, multiple preset inspection task categories may include configuration change inspection task categories, table information inspection task categories, and table data inspection task categories. For the configuration change inspection task category, a dependency comparison strategy, a task execution time comparison strategy, an execution resource consumption comparison strategy, and a task baseline comparison strategy can be constructed to generate a preset inspection strategy corresponding to the configuration change inspection task category. For the table information inspection task category, a field comparison strategy, a partition row count comparison strategy, and a primary key comparison strategy are constructed to generate a preset inspection strategy corresponding to the table information inspection task category based on these strategies. For the table data inspection task category, a data difference percentage strategy, a data difference detail strategy, and a business rule comparison strategy are constructed to generate a preset inspection strategy corresponding to the table data inspection task category. Here, the data difference percentage refers to the proportion of the difference data appearing before and after the task change in the total data, and the data difference detail refers to the details of the difference data appearing before and after the task change.

[0171] In this embodiment of the disclosure, preset inspection strategies can be constructed for each of the inspection task categories for configuration changes, table information, and table data, thereby increasing the diversity of preset inspection strategies and thus increasing the diversity of target inspection strategies.

[0172] In this embodiment of the disclosure, the method further includes:

[0173] By analyzing the test data and the initial data, we obtain dependency comparison data, task execution time comparison data, execution resource consumption comparison data, and task baseline comparison data.

[0174] Based on an unstructured query language, the dependency comparison data, task execution time comparison data, execution resource consumption comparison data, and task baseline comparison data are stored in the first database;

[0175] The test data and the initial data are analyzed to obtain field comparison data, partition row count comparison data, and primary key comparison data.

[0176] Based on an unstructured query language, the field comparison data, partition row count comparison data, and primary key comparison data are stored in a second database;

[0177] The test data and the initial data are analyzed to obtain data difference percentage data, data difference detail data, and business rule comparison data.

[0178] Based on a structured query language, the data difference percentage data, data difference details data, and business rule comparison data are stored in a third database.

[0179] In this embodiment of the disclosure, a database is a large, organized, and shareable collection of data stored permanently in a computer. The data in the database is organized, described, and stored according to a specific data model, exhibiting low redundancy, high data independence, and easy scalability, and can be shared by various users. It is mainly divided into relational databases and non-relational databases. A relational database is a database management system based on the relational model. It uses relations (tables) to organize data and establishes relationships between data through associations between tables. Relational databases use structured query languages ​​to perform data operations and queries. The test data and the initial data can be parsed to obtain dependency comparison data, task execution time comparison data, execution resource consumption comparison data, and task baseline comparison data. For example, the dependency comparison data, task execution time comparison data, execution resource consumption comparison data, and task baseline comparison data are stored in a first database. The test data and the initial data are parsed to obtain field comparison data, partition row count comparison data, and primary key comparison data. Based on an unstructured query language, the field comparison data, partition row count comparison data, and primary key comparison data are stored in a second database. Thus, the first database and the second database are constructed based on the unstructured query language. The test data and the initial data are then parsed again to obtain data difference percentage data, data difference detail data, and business rule comparison data. Based on a structured query language, the data difference percentage data, data difference detail data, and business rule comparison data are stored in a third database, resulting in a third database corresponding to the structured query language.

[0180] In this embodiment of the disclosure, different languages ​​can be used to construct databases corresponding to different data types in the test data and initial data, which improves the data storage efficiency and facilitates subsequent data retrieval by classification.

[0181] In this embodiment of the disclosure, if the target inspection task category is the inspection task category of the configuration change, such as Figure 4 As shown, Figure 4 A method for performing a matching check on the test data and initial data based on the target checking strategy to obtain the target checking result includes:

[0182] S401: Based on the target inspection strategy corresponding to the inspection task category of the configuration change, call the preset interface protocol to query the dependency comparison data corresponding to the dependency comparison strategy in the first database according to the unstructured query language;

[0183] S403: Query the task execution time comparison data corresponding to the task execution time comparison strategy; query the execution resource consumption comparison data corresponding to the execution resource consumption comparison strategy; and query the task baseline comparison data corresponding to the task baseline comparison strategy.

[0184] S405: Determine the target inspection result based on the dependency comparison data, the task execution time comparison data, the execution resource consumption comparison data, and the task baseline comparison data.

[0185] In this embodiment of the disclosure, if the target inspection task category is the configuration change inspection task category, multiple sub-strategies corresponding to that category can be obtained, and comparison data can be queried in the corresponding database for data comparison to obtain the target inspection result; the preset interface protocol may include, but is not limited to, the remote procedure call protocol. For the configuration change inspection task category, dependency comparison data, task execution time comparison data, execution resource consumption comparison data, and task baseline comparison data can be queried in the first database according to the unstructured query language, so that the target inspection result can be obtained based on the comparison results of each comparison data.

[0186] In some embodiments, before step S401, the method may further include: obtaining query account information, wherein the query account can be the terminal account that submitted the task change request, and then determining whether the query account information has access to the first database. If the query account has access, step S401 is executed, and according to the target inspection strategy corresponding to the inspection task category of the configuration change, a preset interface protocol is called to query the dependency comparison data corresponding to the dependency comparison strategy in the first database according to the unstructured query language.

[0187] In some embodiments, after determining that the query account has the necessary access rights, resource allocation can be further performed to filter out available resources and perform data matching and checking tasks.

[0188] In this embodiment of the disclosure, for the category of inspection tasks involving configuration changes, dependency comparison data, task execution time comparison data, execution resource consumption comparison data, and task baseline comparison data can be queried in the first database using an unstructured query language, thereby improving data query efficiency and the efficiency of determining the target inspection results.

[0189] In this embodiment of the disclosure, such as Figure 5 As shown, Figure 5 A flowchart of a method for determining the target inspection result based on the dependency comparison data, the task execution time comparison data, the execution resource consumption comparison data, and the task baseline comparison data, the method comprising:

[0190] S4051: Based on the dependency comparison strategy, perform a matching check on the dependency comparison data to obtain the dependency check result;

[0191] S4053: Based on the task execution time comparison strategy, perform a matching check on the task execution time comparison data to obtain the execution time check result;

[0192] S4055: Based on the execution resource consumption comparison strategy, perform a matching check on the execution resource consumption comparison data to obtain the resource consumption check result;

[0193] S4057: Based on the task baseline comparison strategy, perform a matching check on the task baseline comparison data to obtain the task baseline check result;

[0194] S4059: Determine the target check result based on the dependency check result, the execution time check result, the resource consumption check result, and the task baseline check result.

[0195] In this embodiment of the disclosure, for dependency comparison data, a dependency comparison strategy can be used for matching checks to obtain dependency check results; for task execution time comparison data, a task execution time comparison strategy can be used for matching checks to obtain execution time check results; for execution resource consumption comparison data, an execution resource consumption comparison strategy can be used for matching checks to obtain resource consumption check results; and for task baseline comparison data, a task baseline comparison strategy can be used for matching checks to obtain task baseline check results. The dependency check results, execution time check results, resource consumption check results, and task baseline check results can all include pass / fail results and fail / fail results. If at least one fail / fail result exists among the dependency check results, execution time check results, resource consumption check results, and task baseline check results, the target check result is determined to be a fail / fail result, and abnormal data after task changes can be quickly located. If all of the dependency check results, execution time check results, resource consumption check results, and task baseline check results are pass / fail results, the target check result is determined to be a pass / fail result.

[0196] In this embodiment of the disclosure, if the target inspection task category is the configuration change inspection task category, the target inspection result corresponding to the task category can be quickly determined based on the dependency inspection result, the execution time inspection result, the resource consumption inspection result, and the task baseline inspection result.

[0197] In this embodiment of the disclosure, if the target inspection task category is the inspection task category of the table information, the step of performing a matching inspection on the test data and the initial data based on the target inspection strategy to obtain the target inspection result includes:

[0198] Based on the target inspection strategy corresponding to the inspection task category of the table information, the software development kit is invoked to query the field comparison data corresponding to the field comparison strategy in the second database using an unstructured query language;

[0199] Query the partition row count comparison data corresponding to the partition row count comparison strategy; and query the primary key comparison data corresponding to the primary key comparison strategy.

[0200] The target inspection result is determined based on the field comparison data, the partition row count comparison data, and the primary key comparison data.

[0201] In this embodiment of the disclosure, if the target inspection task category is the inspection task category of the table information, multiple strategies corresponding to the inspection task category can be obtained, including field comparison strategy, partition row number comparison strategy and primary key comparison strategy. Then, according to the multiple strategies, the corresponding data is queried in the second database to determine the target inspection result.

[0202] In this embodiment of the disclosure, for the inspection task category of table information, a software development kit can be invoked to query field comparison data, partition row number comparison data, and primary key comparison data in the second database according to the unstructured query language, thereby improving data query efficiency and improving the efficiency of determining the target inspection results.

[0203] In some embodiments, before invoking the software development kit (SDK) to query the field comparison data corresponding to the field comparison strategy in the second database according to the target inspection strategy corresponding to the inspection task category of the table information, the process may further include: obtaining query account information, where the query account can be the terminal account that submitted the task change request; then determining whether the query account information has access permissions to the second database; if the query account has access permissions, executing the target inspection strategy corresponding to the inspection task category of the table information, and invoking the SSD to query the field comparison data corresponding to the field comparison strategy in the second database according to the unstructured query language. In some embodiments, after determining that the query account has access permissions, resource allocation may be further performed to filter out idle resources to execute the matching inspection task.

[0204] In this embodiment of the disclosure, determining the target inspection result based on the field comparison data, the partition row count comparison data, and the primary key comparison data includes:

[0205] Based on the field comparison strategy, the field comparison data is matched and checked to obtain the field check results;

[0206] Based on the partition row count comparison strategy, the partition row count comparison data is matched and checked to obtain the partition row count check result;

[0207] Based on the primary key comparison strategy, the primary key comparison data is matched and checked to obtain the primary key check result;

[0208] The target inspection result is determined based on the field inspection result, the partition row count inspection result, and the primary key inspection result.

[0209] In this embodiment of the disclosure, the field comparison data can be matched and checked according to a field comparison strategy to obtain a field check result; the partition row count comparison data can be matched and checked according to a partition row count comparison strategy to obtain a partition row count check result; the primary key comparison data can be matched and checked according to a primary key comparison strategy to obtain a primary key check result; then, the target check result is determined based on the field check result, the partition row count check result, and the primary key check result; if at least one of the field check result, the partition row count check result, and the primary key check result is an abnormal check result, then the target check result corresponding to the task category is determined to be an abnormal check result; if the field check result, the partition row count check result, and the primary key check result are all normal check results, then the target check result corresponding to the task category is determined to be a normal check result.

[0210] In this embodiment of the disclosure, if the target inspection task category is the inspection task category of the table information, the target inspection result corresponding to the task category can be quickly determined based on the field inspection result, the partition row count inspection result, and the primary key inspection result.

[0211] In this embodiment of the disclosure, if the target inspection task category is the inspection task category of the table data, the step of performing a matching inspection on the test data and the initial data based on the target inspection strategy to obtain the target inspection result includes:

[0212] Based on the target inspection strategy corresponding to the inspection task category of the table data, and using structured query language, query the data difference percentage data corresponding to the data difference percentage strategy in the third database;

[0213] Query the data difference details corresponding to the aforementioned data difference details strategy;

[0214] Query the business rule comparison data corresponding to the business rule comparison strategy;

[0215] The target inspection result is determined based on the data difference percentage data, the data difference details data, and the business rule comparison data.

[0216] In this embodiment of the disclosure, if the target inspection task category is the inspection task category of the table data, the data corresponding to each strategy under this category can be obtained from the third database. The data difference percentage data corresponding to the data difference percentage strategy can be queried in the third database; the data difference detail data corresponding to the data difference detail strategy can be queried; and the business rule comparison data corresponding to the business rule comparison strategy can be queried, thereby facilitating the further determination of the target inspection result corresponding to the inspection task category of the table data.

[0217] In this embodiment of the disclosure, for the inspection task category of table data, the data difference percentage data, the data difference details data, and the business rule comparison data can be searched in the third database using structured query language to determine the target inspection result, thereby improving data query efficiency and the efficiency of determining the target inspection result.

[0218] In some embodiments, before querying the data difference percentage data corresponding to the data difference percentage strategy in the third database according to the target inspection strategy corresponding to the inspection task category of the table data and according to Structured Query Language, the process may further include: obtaining query account information, where the query account can be the terminal account that submitted the task change request; then determining whether the query account information has access permissions to the third database; if the query account has access permissions, executing the target inspection strategy corresponding to the inspection task category of the table data and querying the data difference percentage data corresponding to the data difference percentage strategy in the third database according to Structured Query Language. In some embodiments, after determining that the query account has access permissions, resource allocation may be further performed to filter out idle resources to execute the matching inspection task of the data.

[0219] In this embodiment of the disclosure, determining the target inspection result based on the data difference percentage data, the data difference detail data, and the business rule comparison data includes:

[0220] Based on the data difference ratio strategy, a matching check is performed on the data difference ratio data to obtain the data difference ratio check result.

[0221] Based on the aforementioned data difference detail strategy, a matching check is performed on the data difference detail data to obtain the data difference detail check results;

[0222] Based on the business rule comparison strategy, the business rule comparison data is matched and checked to obtain the business rule comparison check result.

[0223] The target inspection result is determined based on the data difference percentage inspection result, the data difference detail inspection result, and the business rule comparison inspection result.

[0224] In this embodiment of the disclosure, the data difference percentage data can be matched and checked according to a data difference percentage strategy to obtain a data difference percentage check result; then, the data difference detail data can be matched and checked according to a data difference detail strategy to obtain a data difference detail check result; and finally, the business rule comparison data can be matched and checked according to a business rule comparison strategy to obtain a business rule comparison check result; thereby, the target check result is determined based on the data difference percentage check result, the data difference detail check result, and the business rule comparison check result. If at least one of the data difference percentage check result, the data difference detail check result, and the business rule comparison check result is an abnormal check result, the target check result is determined to be an abnormal check result. If all three of the data difference percentage check result, the data difference detail check result, and the business rule comparison check result are normal check results, the target check result is determined to be a normal check result.

[0225] In this embodiment of the disclosure, for the inspection task category of table data, the target inspection result can be quickly determined based on the data difference ratio inspection result, the data difference detail inspection result, and the business rule comparison inspection result.

[0226] In some embodiments, the above method further includes:

[0227] If the target inspection result indicates that the test data is normal, continue to execute the offline calculation change task.

[0228] In this embodiment of the disclosure, if the target check result is a pass result, it indicates that the test data is normal, and the offline calculation change task continues to be executed.

[0229] In this embodiment of the disclosure, if the target inspection result indicates that the test data is abnormal, the method further includes:

[0230] The offline computing change task is adjusted to obtain the adjusted offline computing change task;

[0231] The adjusted offline computing change task is re-established as the offline computing change task, and the process jumps to the step of executing the offline computing change task in the test environment.

[0232] In this embodiment of the disclosure, if the target inspection result indicates that the test data is abnormal, abnormal data can be obtained, and the offline computing change task can be adjusted according to the data category corresponding to the abnormal data. The adjusted offline computing change task is then used as the offline computing change task again, and the process jumps to the step of executing the offline computing change task in the test environment, and the adjusted task is checked repeatedly.

[0233] In this embodiment of the disclosure, if the target inspection result indicates that the test data is abnormal, the offline computing change task can be adjusted and the adjusted task can be executed, thereby repeatedly checking the adjusted task to improve the efficiency of task adjustment and task execution.

[0234] In some embodiments, the inspection task categories for configuration changes, table information, and table data can be inspected separately to obtain the target inspection result corresponding to the task. When the inspection result of at least one of the inspection task categories for configuration changes, table information, and table data is an abnormal result, the target inspection result is determined to be an abnormal result. If the inspection results corresponding to the inspection task categories for configuration changes, table information, and table data are all normal results, the target inspection result is determined to be a normal result.

[0235] In one exemplary embodiment, this disclosure provides a task execution method, the method comprising:

[0236] When the initial offline computing task changes, execute the changed offline computing task in the test environment;

[0237] During the execution of the offline computation change task, test data is generated and the target inspection task category corresponding to the test data is determined;

[0238] The system queries the preset strategy database for target inspection strategies that match the target inspection task category; the preset strategy database stores the association between preset inspection task categories and preset inspection strategies.

[0239] According to the target inspection strategy corresponding to the inspection task category of the configuration change, the preset interface protocol is invoked to query the dependency comparison data corresponding to the dependency comparison strategy in the first database using an unstructured query language;

[0240] Query the task execution time comparison data corresponding to the task execution time comparison strategy; query the execution resource consumption comparison data corresponding to the execution resource consumption comparison strategy; and query the task baseline comparison data corresponding to the task baseline comparison strategy.

[0241] Based on the dependency comparison strategy, the dependency comparison data is matched and checked to obtain the dependency check results;

[0242] Based on the task execution time comparison strategy, the task execution time comparison data is matched and checked to obtain the execution time check result;

[0243] Based on the execution resource consumption comparison strategy, the execution resource consumption comparison data is matched and checked to obtain the resource consumption check result;

[0244] Based on the task baseline comparison strategy, the task baseline comparison data is matched and checked to obtain the task baseline check result.

[0245] The target check result is determined based on the dependency check result, the execution time check result, the resource consumption check result, and the task baseline check result.

[0246] Based on the target inspection strategy corresponding to the inspection task category of the table information, the software development kit is invoked to query the field comparison data corresponding to the field comparison strategy in the second database using an unstructured query language;

[0247] Query the partition row count comparison data corresponding to the partition row count comparison strategy; and query the primary key comparison data corresponding to the primary key comparison strategy.

[0248] Based on the field comparison strategy, the field comparison data is matched and checked to obtain the field check results;

[0249] Based on the partition row count comparison strategy, the partition row count comparison data is matched and checked to obtain the partition row count check result;

[0250] Based on the primary key comparison strategy, the primary key comparison data is matched and checked to obtain the primary key check result;

[0251] The target inspection result is determined based on the field inspection result, the partition row count inspection result, and the primary key inspection result.

[0252] Based on the target inspection strategy corresponding to the inspection task category of the table data, and using structured query language, query the data difference percentage data corresponding to the data difference percentage strategy in the third database;

[0253] Query the data difference details corresponding to the aforementioned data difference details strategy;

[0254] Query the business rule comparison data corresponding to the business rule comparison strategy;

[0255] Based on the data difference ratio strategy, a matching check is performed on the data difference ratio data to obtain the data difference ratio check result.

[0256] Based on the aforementioned data difference detail strategy, a matching check is performed on the data difference detail data to obtain the data difference detail check results;

[0257] Based on the business rule comparison strategy, the business rule comparison data is matched and checked to obtain the business rule comparison check result.

[0258] The target inspection result is determined based on the data difference percentage inspection result, the data difference detail inspection result, and the business rule comparison inspection result.

[0259] If at least one of the target check results indicates that the test data is abnormal, stop executing the offline calculation change task.

[0260] like Figure 6 As shown, Figure 6 This is a schematic diagram of the structure of a task execution system, including an automated data comparison library, a scheduling platform (first database), a metadata center (second database), and a data warehouse (third database). The automated data comparison library stores comparison rules for both unstructured and structured query languages, and automatically generates comparison reports by querying comparison data in the scheduling platform or metadata center based on the unstructured query language comparison rules and executing the comparison rules. It also automatically generates comparison reports by querying comparison data in the data warehouse based on the structured query language comparison rules and executing the comparison rules.

[0261] like Figure 7 As shown, Figure 7This is a schematic diagram of the structure of a big data offline comparison testing platform. The platform includes a comparison rule execution module, a comparison rule management module, a comparison management module, and a comparison permission management module. The comparison rule execution module includes both structured query language rule executors and unstructured query language rule executors. The comparison rule management module includes configuration-based comparison rules, table information comparison rules, and table data comparison rules. This module can be used for creating comparison tasks, querying comparison tasks, generating comparison reports, and pushing comparison results. The comparison permission management module is used for table permission checks and resource permission management. The comparison rule management module standardizes and unifies the content to be compared and checked within the big data offline comparison testing platform, defining it into different comparison rules. For example, checks on configuration changes for computing tasks are categorized as configuration-based comparison rules. These rules can check for differences in task dependencies, differences in task execution time caused by the change, and differences in computing resources caused by the change. Other types of rules include table information-based comparison rules, table data-based comparison rules, and custom comparison rules. By managing comparison rules, comparison checks become more standardized. The comparison rule execution module maintains general rules applicable to output tables from various offline big data computing tasks. It automates the execution of comparison rules for specific test tables and online tables, determining execution strategies based on rule configurations and using the permission management module for permission checks and resource allocation. The comparison task management module treats the comparison of results from a single offline big data computing task as a single task. It allows users to create comparison tasks (with multiple rules), query historical comparison tasks, and retrieve comparison reports. The comparison testing platform provides external interfaces for tasks such as task creation, execution status checks, result confirmation, and report retrieval. These interfaces facilitate easy integration with computing or management platforms.

[0262] like Figure 8 As shown, Figure 8 This is an interaction flowchart of a comparison testing platform and a computing platform. The method includes: the computing platform is developed based on offline computing tasks, generates offline computing tasks, and runs offline computing tasks; the computing platform is connected to the comparison testing platform; by executing the comparison strategy of the comparison testing platform, data comparison checks are performed before and after the task change to determine whether there are any abnormal differences; if so, a comparison report is pushed to the user who changed the task; the user who changed the task can further determine whether the comparison results are normal; if the results are normal, subsequent processes such as testing or release continue; if the results are abnormal, the table change task is readjusted, and the adjustment task is restarted and checked again.

[0263] This disclosure provides an automated comparative testing method suitable for offline big data computing. When an offline big data computing task changes, it automatically compares the differences in execution time, table information, and table data before and after the change, and provides a visual comparison report. This provides a reference for assessing the impact of changes; furthermore, when abnormal comparison results are detected, the offline computing task is readjusted, and the next stage of the process is automatically blocked. The standardization of the comparison check content ensures effective coverage; and the automated execution of comparison tests eliminates the need for manually writing check scripts, greatly improving the efficiency of the comparison check. Extensive external interface support facilitates expansion according to actual needs.

[0264] Figure 9 This is a block diagram illustrating a task execution device according to an exemplary embodiment. (Refer to...) Figure 9 The device includes:

[0265] The task execution module 910 is modified to execute an offline computing change task in the test environment when the initial offline computing task is changed.

[0266] The target inspection category determination module 920 is configured to generate test data and determine the target inspection task category corresponding to the test data during the execution of the offline calculation change task.

[0267] The target strategy query module 930 is configured to query a preset strategy database for target inspection strategies that match the target inspection task category; the preset strategy database stores the association between preset inspection task categories and preset inspection strategies.

[0268] The result determination module 940 is configured to perform a matching check on the test data and the initial data based on the target checking strategy to obtain the target checking result; the initial data is the execution result data of the initial offline computing task executed in the application environment;

[0269] The task stop module 950 is configured to stop executing the offline calculation change task if the target check result indicates that the test data is abnormal.

[0270] In one exemplary embodiment, the apparatus further includes:

[0271] The preset category acquisition module is configured to acquire multiple preset inspection task categories;

[0272] The preset strategy building module is configured to execute the preset inspection strategy corresponding to each preset inspection task category;

[0273] The database construction module is configured to construct the preset strategy database based on the association between each preset inspection task category and the preset inspection strategy corresponding to each preset inspection task category.

[0274] In one exemplary embodiment, the plurality of preset inspection task categories include a configuration change inspection task category, a table information inspection task category, and a table data inspection task category, and the preset strategy construction module includes:

[0275] The first comparison strategy construction unit is configured to perform inspection task categories for the configuration changes and construct a dependency comparison strategy, a task execution time comparison strategy, an execution resource consumption comparison strategy, and a task baseline comparison strategy.

[0276] The first inspection strategy generation unit is configured to generate a preset inspection strategy corresponding to the inspection task category of the configuration change based on the dependency comparison strategy, task execution time comparison strategy, execution resource consumption comparison strategy and task baseline comparison strategy.

[0277] The second comparison strategy construction unit is configured to perform inspection task categories for the table information and construct field comparison strategy, partition row number comparison strategy and primary key comparison strategy.

[0278] The second inspection strategy generation unit is configured to execute a preset inspection strategy corresponding to the inspection task category of the table information based on the field comparison strategy, the partition row number comparison strategy and the primary key comparison strategy.

[0279] The third comparison strategy construction unit is configured to perform inspection task categories for the table data and construct data difference ratio strategy, data difference detail strategy and business rule comparison strategy.

[0280] The third inspection strategy generation unit is configured to execute a preset inspection strategy corresponding to the inspection task category of the table data based on the data difference ratio strategy, data difference details strategy, and business rule comparison strategy.

[0281] In one exemplary embodiment, the apparatus further includes:

[0282] The first data parsing module is configured to parse the test data and the initial data to obtain dependency comparison data, task execution time comparison data, execution resource consumption comparison data and task baseline comparison data.

[0283] The first data storage module is configured to execute an unstructured query language to store the dependency comparison data, task execution time comparison data, execution resource consumption comparison data, and task baseline comparison data into the first database.

[0284] The second data parsing module is configured to parse the test data and the initial data to obtain field comparison data, partition row number comparison data and primary key comparison data.

[0285] The second data storage module is configured to execute an unstructured query language to store the field comparison data, partition row number comparison data, and primary key comparison data into the second database;

[0286] The third data parsing module is configured to parse the test data and the initial data to obtain data difference percentage data, data difference detail data, and business rule comparison data.

[0287] The third data storage module is configured to execute a structured query language to store the data difference percentage data, data difference details data, and business rule comparison data into the third database.

[0288] In an exemplary implementation, if the target inspection task category is the configuration change inspection task category, the result determination module includes:

[0289] The dependency comparison data query unit is configured to execute the target inspection strategy corresponding to the inspection task category changed according to the configuration, and call the preset interface protocol to query the dependency comparison data corresponding to the dependency comparison strategy in the first database according to the unstructured query language.

[0290] The baseline comparison data query unit is configured to query the task execution time comparison data corresponding to the task execution time comparison strategy; query the execution resource consumption comparison data corresponding to the execution resource consumption comparison strategy; and query the task baseline comparison data corresponding to the task baseline comparison strategy.

[0291] The first target result determination unit is configured to determine the target check result based on the dependency comparison data, the task execution time comparison data, the execution resource consumption comparison data, and the task baseline comparison data.

[0292] In one exemplary embodiment, the first target result determination unit includes:

[0293] The dependency check result determination subunit is configured to perform a matching check on the dependency comparison data based on the dependency comparison strategy to obtain the dependency check result;

[0294] The execution time check result determination subunit is configured to perform a matching check on the task execution time comparison data based on the task execution time comparison strategy to obtain the execution time check result;

[0295] The resource consumption check result determination subunit is configured to perform a matching check on the execution resource consumption comparison data based on the execution resource consumption comparison strategy to obtain the resource consumption check result.

[0296] The task baseline check result determination subunit is configured to perform a matching check on the task baseline comparison data based on the task baseline comparison strategy to obtain the task baseline check result.

[0297] The first target check result determination subunit is configured to determine the target check result based on the dependency check result, the execution time check result, the resource consumption check result, and the task baseline check result.

[0298] In an exemplary implementation, if the target inspection task category is the inspection task category of the table information, the result determination module includes:

[0299] The field comparison data query unit is configured to execute the target inspection strategy corresponding to the inspection task category based on the table information, and call the software development kit to query the field comparison data corresponding to the field comparison strategy in the second database according to the unstructured query language;

[0300] The primary key comparison data query unit is configured to perform a query for the partition row count comparison data corresponding to the partition row count comparison strategy; and to query the primary key comparison data corresponding to the primary key comparison strategy.

[0301] The second target result determination unit is configured to determine the target inspection result based on the field comparison data, the partition row number comparison data, and the primary key comparison data.

[0302] In one exemplary embodiment, the second target result determination unit includes:

[0303] The field check result determination subunit is configured to perform a matching check on the field comparison data based on the field comparison strategy to obtain the field check result;

[0304] The partition row count check result determination subunit is configured to perform a matching check on the partition row count comparison data based on the partition row count comparison strategy to obtain the partition row count check result.

[0305] The primary key check result determination subunit is configured to perform a matching check on the primary key comparison data based on the primary key comparison strategy to obtain the primary key check result.

[0306] The first target inspection result determination subunit is configured to determine the target inspection result based on the field inspection result, the partition row count inspection result, and the primary key inspection result.

[0307] In an exemplary implementation, if the target inspection task category is the inspection task category of the table data, the result determination module includes:

[0308] The difference data query unit is configured to execute the target inspection strategy corresponding to the inspection task category of the table data, and query the data difference percentage data corresponding to the data difference percentage strategy in the third database according to the structured query language.

[0309] The detailed data query unit is configured to query the detailed data difference data corresponding to the data difference detailed strategy;

[0310] The comparison data query unit is configured to query the business rule comparison data corresponding to the business rule comparison strategy.

[0311] The third target result determination unit is configured to determine the target inspection result based on the data difference ratio data, the data difference details data, and the business rule comparison data.

[0312] In one exemplary embodiment, the third target result determination unit includes:

[0313] The difference result determination subunit is configured to perform a matching check on the data difference ratio data based on the data difference ratio strategy to obtain the data difference ratio check result.

[0314] The detailed inspection result determination subunit is configured to perform a matching check on the data difference detail data based on the data difference detail strategy to obtain the data difference detail inspection result;

[0315] The business inspection result determination subunit is configured to perform a matching check on the business rule comparison data based on the business rule comparison strategy to obtain the business rule comparison inspection result.

[0316] The third target result determination subunit is configured to determine the target inspection result based on the data difference ratio inspection result, the data difference detail inspection result, and the business rule comparison inspection result.

[0317] In one exemplary embodiment, if the target inspection result indicates that the test data is abnormal, the apparatus further includes:

[0318] The adjustment module is configured to perform adjustments to the offline computing change task to obtain the adjusted offline computing change task;

[0319] The jump module is configured to execute the step of re-establishing the adjusted offline computing change task as the offline computing change task and jumping to the step of executing the offline computing change task in the test environment.

[0320] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0321] In one exemplary embodiment, an electronic device is also provided, including a processor; a memory for storing processor-executable instructions; wherein, when the processor is configured to execute the instructions stored in the memory, it implements the task execution method provided in any of the above embodiments.

[0322] The electronic device can be a terminal, a server, or a similar computing device. Taking a server as an example... Figure 10 This is a block diagram illustrating an electronic device according to an exemplary embodiment, such as... Figure 10 As shown, the server 1000 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 1010 (CPUs 1010 may include, but are not limited to, microprocessors (MCUs) or programmable logic devices (FPGAs), a memory 1030 for storing data, and one or more storage media 1020 (e.g., one or more mass storage devices) for storing application programs 1023 or data 1022. The memory 1030 and storage media 1020 may be temporary or persistent storage. The program stored in the storage media 1020 may include one or more modules, each module may include a series of instruction operations on the server. Furthermore, the CPU 1010 may be configured to communicate with the storage media 1020 and execute the series of instruction operations in the storage media 1020 on the server 1000. Server 1000 may also include one or more power supplies 1060, one or more wired or wireless network interfaces 1050, one or more input / output interfaces 1040, and / or one or more operating systems 1021, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0323] The input / output interface 1040 can be used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of server 1000. In one example, the input / output interface 1040 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the input / output interface 1040 may be a radio frequency (RF) module for wireless communication with the Internet.

[0324] Those skilled in the art will understand that Figure 10 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, server 1000 may also include... Figure 10 The more or fewer components shown, or having the same Figure 10 The different configurations shown.

[0325] In one exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 1030 including instructions, which can be executed by a processor 1010 of the device 1000 to perform the above-described method. Optionally, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0326] In one exemplary embodiment, a computer program product is also provided, including a computer program that, when executed by a processor, implements the task execution method provided in any of the above embodiments.

[0327] Figure 11 This is a block diagram illustrating an electronic device for task execution according to an exemplary embodiment. The electronic device may be a terminal, and its internal structure diagram may be as follows: Figure 11As shown, the electronic device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a task execution method. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.

[0328] Those skilled in the art will understand that Figure 11 The structure shown is merely a block diagram of a portion of the structure related to the present disclosure and does not constitute a limitation on the electronic device to which the present disclosure is applied. A specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0329] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0330] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0331] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A task execution method, characterized in that, The method includes: When the initial offline computing task changes, execute the changed offline computing task in the test environment; During the execution of the offline computation change task, test data is generated and the target inspection task category corresponding to the test data is determined; The system queries the preset strategy database for target inspection strategies that match the target inspection task category; the preset strategy database stores the association between preset inspection task categories and preset inspection strategies. Based on the target inspection strategy, the test data and the initial data are matched and inspected to obtain the target inspection result; the initial data is the execution result data of the initial offline computing task executed in the application environment; If the target inspection result indicates that the test data is abnormal, stop executing the offline calculation change task; The method for constructing the preset strategy database includes: obtaining multiple preset inspection task categories, which include configuration change inspection task categories, table information inspection task categories, and table data inspection task categories; Construct a preset inspection strategy for each preset inspection task category; The preset strategy database is constructed based on the association between each preset inspection task category and the preset inspection strategy corresponding to each preset inspection task category.

2. The method according to claim 1, characterized in that, The construction of the preset inspection strategy corresponding to each preset inspection task category includes: For the types of inspection tasks related to the configuration changes, construct dependency comparison strategies, task execution time comparison strategies, execution resource consumption comparison strategies, and task baseline comparison strategies; Based on the dependency comparison strategy, task execution time comparison strategy, execution resource consumption comparison strategy, and task baseline comparison strategy, a preset inspection strategy corresponding to the configuration change inspection task category is generated. For the types of tasks involving the inspection of the table information, construct field comparison strategies, partition row count comparison strategies, and primary key comparison strategies; Based on the field comparison strategy, partition row count comparison strategy, and primary key comparison strategy, a preset inspection strategy corresponding to the inspection task category of the table information is generated. For the inspection task categories of the table data, construct data difference percentage strategy, data difference detail strategy, and business rule comparison strategy; Based on the data difference percentage strategy, data difference detail strategy, and business rule comparison strategy, a preset inspection strategy corresponding to the inspection task category of the table data is generated.

3. The method according to claim 2, characterized in that, The method further includes: By analyzing the test data and the initial data, we obtain dependency comparison data, task execution time comparison data, execution resource consumption comparison data, and task baseline comparison data. Based on an unstructured query language, the dependency comparison data, task execution time comparison data, execution resource consumption comparison data, and task baseline comparison data are stored in the first database; The test data and the initial data are analyzed to obtain field comparison data, partition row count comparison data, and primary key comparison data. Based on an unstructured query language, the field comparison data, partition row count comparison data, and primary key comparison data are stored in a second database; The test data and the initial data are analyzed to obtain data difference percentage data, data difference detail data, and business rule comparison data. Based on a structured query language, the data difference percentage data, data difference details data, and business rule comparison data are stored in a third database.

4. The method according to claim 3, characterized in that, If the target inspection task category is the configuration change inspection task category, the matching inspection of the test data and the initial data based on the target inspection strategy to obtain the target inspection result includes: According to the target inspection strategy corresponding to the inspection task category of the configuration change, the preset interface protocol is invoked to query the dependency comparison data corresponding to the dependency comparison strategy in the first database using an unstructured query language; Query the task execution time comparison data corresponding to the task execution time comparison strategy; query the execution resource consumption comparison data corresponding to the execution resource consumption comparison strategy; and query the task baseline comparison data corresponding to the task baseline comparison strategy. The target inspection result is determined based on the dependency comparison data, the task execution time comparison data, the execution resource consumption comparison data, and the task baseline comparison data.

5. The method according to claim 4, characterized in that, The step of determining the target inspection result based on the dependency comparison data, the task execution time comparison data, the execution resource consumption comparison data, and the task baseline comparison data includes: Based on the dependency comparison strategy, the dependency comparison data is matched and checked to obtain the dependency check results; Based on the task execution time comparison strategy, the task execution time comparison data is matched and checked to obtain the execution time check result; Based on the execution resource consumption comparison strategy, the execution resource consumption comparison data is matched and checked to obtain the resource consumption check result; Based on the task baseline comparison strategy, the task baseline comparison data is matched and checked to obtain the task baseline check result. The target check result is determined based on the dependency check result, the execution time check result, the resource consumption check result, and the task baseline check result.

6. The method according to claim 3, characterized in that, If the target inspection task category is the inspection task category of the table information, the step of performing a matching inspection on the test data and the initial data based on the target inspection strategy to obtain the target inspection result includes: Based on the target inspection strategy corresponding to the inspection task category of the table information, the software development kit is invoked to query the field comparison data corresponding to the field comparison strategy in the second database using an unstructured query language; Query the partition row count comparison data corresponding to the partition row count comparison strategy; and query the primary key comparison data corresponding to the primary key comparison strategy. The target inspection result is determined based on the field comparison data, the partition row count comparison data, and the primary key comparison data.

7. The method according to claim 6, characterized in that, The step of determining the target inspection result based on the field comparison data, the partition row count comparison data, and the primary key comparison data includes: Based on the field comparison strategy, the field comparison data is matched and checked to obtain the field check results; Based on the partition row count comparison strategy, the partition row count comparison data is matched and checked to obtain the partition row count check result; Based on the primary key comparison strategy, the primary key comparison data is matched and checked to obtain the primary key check result; The target inspection result is determined based on the field inspection result, the partition row count inspection result, and the primary key inspection result.

8. The method according to claim 3, characterized in that, If the target inspection task category is the inspection task category of the table data, the matching inspection of the test data and the initial data based on the target inspection strategy to obtain the target inspection result includes: Based on the target inspection strategy corresponding to the inspection task category of the table data, and using structured query language, query the data difference percentage data corresponding to the data difference percentage strategy in the third database; Query the data difference details corresponding to the aforementioned data difference details strategy; Query the business rule comparison data corresponding to the business rule comparison strategy; The target inspection result is determined based on the data difference percentage data, the data difference details data, and the business rule comparison data.

9. The method according to claim 8, characterized in that, The step of determining the target inspection result based on the data difference percentage data, the data difference detail data, and the business rule comparison data includes: Based on the data difference ratio strategy, a matching check is performed on the data difference ratio data to obtain the data difference ratio check result. Based on the aforementioned data difference detail strategy, a matching check is performed on the data difference detail data to obtain the data difference detail check results; Based on the business rule comparison strategy, the business rule comparison data is matched and checked to obtain the business rule comparison check result. The target inspection result is determined based on the data difference percentage inspection result, the data difference detail inspection result, and the business rule comparison inspection result.

10. The method according to claim 1, characterized in that, If the target inspection result indicates that the test data is abnormal, the method further includes: The offline computing change task is adjusted to obtain the adjusted offline computing change task; The adjusted offline computing change task is re-established as the offline computing change task, and the process jumps to the step of executing the offline computing change task in the test environment.

11. A task execution device, characterized in that, include: The task execution module has been modified to execute an offline computing change task in the test environment when the initial offline computing task changes. The target inspection category determination module is configured to generate test data and determine the target inspection task category corresponding to the test data during the execution of the offline computation change task. The target strategy query module is configured to query the preset strategy database for target inspection strategies that match the target inspection task category; The preset strategy database stores the association between preset inspection task categories and preset inspection strategies; The result determination module is configured to perform a matching check on the test data and the initial data based on the target checking strategy to obtain the target checking result; the initial data is the execution result data of the initial offline computing task executed in the application environment; The task stop module is configured to stop executing the offline calculation change task if the target check result indicates that the test data is abnormal. The preset category acquisition module is configured to acquire multiple preset inspection task categories; The multiple preset inspection task categories include configuration change inspection task categories, table information inspection task categories, and table data inspection task categories; The preset strategy building module is configured to execute the preset inspection strategy corresponding to each preset inspection task category; The database construction module is configured to construct the preset strategy database based on the association between each preset inspection task category and the preset inspection strategy corresponding to each preset inspection task category.

12. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the task execution method as described in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by an electronic device processor, the electronic device is able to perform the task execution method as described in any one of claims 1-10.

Images