A batch task testing method and device, a storage medium and a program product

Abstract

The specification provides a batch task test method and device, a storage medium and a program product, which can be applied to the field of big data. The method comprises the following steps: obtaining at least one group of to-be-tested tasks; based on the task properties of the to-be-tested tasks, filtering key tasks from the at least one group of to-be-tested tasks; obtaining a single batch execution result when each key task is executed individually; obtaining a full batch execution result of each key task in a batch execution scenario; and determining a test result based on a comparison result of the single batch execution result and the full batch execution result. The above method reduces the workload of test analysis by filtering key tasks, effectively determines the influence between different tasks during full batch execution by comparing the single batch execution result and the full batch execution result, and then effectively reflects the effect of batch execution through the test result, thereby efficiently and accurately completing the test of batch processing tasks in a batch processing environment, which is conducive to the production research.

Description

TECHNICAL FIELD

[0001] Embodiments of the present specification relate to the technical field of big data, and in particular, relate to a batch processing task testing method and device, a storage medium and a program product. BACKGROUND

[0002] With the development of technology and business level, the use of distributed architecture to support the development of related business has been widely applied. In the distributed architecture, a transaction can be decomposed, so that multiple nodes can process the transaction respectively, thereby improving scalability and availability as a whole. Based on the above characteristics of the distributed architecture, the batch processing operation can maximize the advantages of the distributed architecture and improve the efficiency of business processing.

[0003] Currently, when applying a distributed system to perform batch processing operations, due to the concurrent processing between batch processing tasks, unlike traditional centralized architecture, there is interference between each business based on the influence condition between businesses during business execution, thereby highlighting the difference between the traditional business execution process. At present, when testing batch processing tasks, it is difficult to effectively test batch processing tasks in combination with the characteristics of distributed batch processing, and it is difficult to analyze the test results based on the current application context, so that the appropriate test conclusion cannot be obtained. Therefore, there is an urgent need for a method that can intuitively and effectively obtain the test results of batch processing tasks under the distributed architecture. SUMMARY

[0004] The present application provides a batch processing task testing method, device, storage medium and program product to solve the problem of how to effectively obtain the test results of batch processing tasks under the distributed architecture.

[0005] To solve the above technical problems, the present application provides a batch processing task testing method, comprising: obtaining at least one group of tasks to be tested; based on the task properties of the tasks to be tested, filtering key tasks from the at least one group of tasks to be tested; obtaining a single batch execution result when each key task is executed individually; obtaining a full batch execution result of each key task in a batch execution scenario; determining a test result based on the comparison result of the single batch execution result and the full batch execution result.

[0006] The application further provides a batch processing task testing device, comprising: a to-be-tested task acquisition module, configured to acquire at least one group of to-be-tested tasks; a key task screening module, configured to screen key tasks from the at least one group of to-be-tested tasks based on task properties of the to-be-tested tasks; a single-batch execution result acquisition module, configured to acquire single-batch execution results of respectively executing each key task; a full-batch execution result acquisition module, configured to acquire full-batch execution results of each key task in a batch execution scenario; and a test result determination module, configured to determine a test result based on a comparison result of the single-batch execution results and the full-batch execution results.

[0007] The application further provides a computer readable storage medium, which stores computer programs / instructions, and the computer programs / instructions realize the batch processing task testing method when executed by a processor.

[0008] The application further provides a computer program product, which stores computer programs / instructions, and the computer programs / instructions realize the batch processing task testing method when executed by a processor.

[0009] As can be seen from the technical solutions provided by the application, when testing batch processing tasks, the application first screens key tasks from at least one group of to-be-tested tasks according to task properties of the to-be-tested tasks, then respectively acquires single-batch execution results of respectively executing each key task, and acquires full-batch execution results of each key task in a batch execution scenario, and finally tests the batch processing tasks by comparing the single-batch execution results and the full-batch execution results. The above process reduces the workload of test analysis by screening key tasks, effectively determines the influence between different tasks in full-batch execution by comparing the single-batch execution results and the full-batch execution results, and then effectively reflects the effect of batch execution through the test result, so that the test of batch processing tasks in a batch processing environment is efficiently and accurately completed, and the production research is facilitated. BRIEF DESCRIPTION OF DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only some embodiments described in the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0011] Figure 1 A flow chart of a batch processing task testing method according to an embodiment of the present application;

[0012] Figure 2 A module diagram of a batch processing task testing device according to an embodiment of the present application;

[0013] Figure 3 Figure 1 is a structural diagram of an embodiment of a batch task testing device according to the present application. DETAILED DESCRIPTION

[0014] The technical solutions of the present application will be described clearly and completely below in conjunction with the accompanying drawings of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of protection of the present application.

[0015] To solve the above technical problems, the present application provides a batch task testing method. The execution subject of the batch task testing method is a batch task testing device, which includes but is not limited to a server, an industrial computer, a PC, etc. As shown in Figure 1, the batch task testing method can include the following specific implementation steps. Figure 1

[0016] S110: Obtain at least one group of tasks to be tested.

[0017] The task to be tested can be a task that needs to be tested. Specifically, the test of the task to be tested can be to execute the task to be tested according to the normal business execution process, and the process execution process and the execution result are taken as the corresponding test result. The test of the task to be tested can also be to formulate a specific execution environment, and obtain the task processing result obtained in the execution environment as the test result. In actual application, the test task can be set according to the needs, and no limitation is made to this.

[0018] In the batch task testing method provided by the present application, the task to be tested can be a task executed based on a batch processing execution mode. Batch processing is a batch processing of tasks, which generally executes corresponding operations on a large number of tasks through a pre-set program script. When executing batch processing tasks, the system can call a batch processing program to process the batch processing tasks line by line and automatically, and obtain the corresponding execution result.

[0019] Specifically, in the application scenario to which the present application is directed, the task to be tested can be a task processed based on a batch processing execution mode under a distributed architecture. The distributed architecture is a tool and application for implementing distributed computing. By converging multiple computers through a computer network, the distributed architecture can effectively disperse the computing pressure by allocating corresponding computing tasks based on the load balancing of the nodes corresponding to different computers under the condition of ensuring the high autonomy of each computer node, thereby better implementing the system batch processing function.

[0020] ​In some embodiments, the focus of the test on the to-be-tested task is to consider the influence of the batch execution scenario on the execution effect of the to-be-tested task. Compared with the case where the to-be-tested task is executed alone, when the to-be-tested task is executed in batches, there is an influence between the tasks during the execution process, there may be overlap in the resource occupation and the execution time of the tasks, and thus there is an influence on the execution effect of the to-be-tested task in the batch process. Therefore, an effective test scheme needs to be developed to obtain the test result of the to-be-tested task.

[0021] S120: Based on the task property of the to-be-tested task, a key task is selected from the at least one group of to-be-tested tasks.

[0022] Compared with the traditional centralized architecture, the distributed architecture pays more attention to the concurrency during task execution. The concurrent execution process inevitably has a time period required for execution, interference or overlap between execution resources based on the execution relationship between tasks, and thus there is a difference in the effect when the business is executed alone. In order to analyze the execution effect of the batch processing task under the distributed architecture, when the to-be-tested task is tested, the effect of the to-be-tested task when executed alone can be obtained, and the effect when executed concurrently is compared, so as to analyze the pros and cons of the concurrent processing process according to the comparison result, and then further optimize the business execution process.

[0023] However, if each to-be-tested task is executed alone, the test result of each to-be-tested task in the case of separate execution is obtained, and the batch execution result is compared, it is obvious that a lot of time and resources need to be consumed. In addition, in the case where the execution process of part of the to-be-tested tasks has particularity, the test result of this part of the to-be-tested tasks may be different from the normal test result, which will interfere with the test result, so that the obtained business test result is not representative.

[0024] Therefore, in order to deal with the above situation, in the present application, when obtaining the execution effect of the to-be-tested task in the case of separate execution, a key task can be selected from the at least one group of to-be-tested tasks.

[0025] The key task can be a task with high importance or priority, which can effectively reflect the overall execution effect of the task. The to-be-tested task can be a task with high execution difficulty, such as occupying a lot of computing resources or having a long processing period, so as to more representatively cover the processing flow of the task.

[0026] In the present application, the key task can be selected according to the task property of the to-be-tested task. The task property of the to-be-tested task is used to reflect the characteristics of the task, and the task property is associated with the execution process and the execution effect of the task, so that when determining the key task, the key task needs to be selected from the to-be-tested task in combination with the consideration of the task property.

[0027] In some embodiments, the task property comprises at least one of a task resource consumption amount, a task priority, a task complexity, a task corresponding time window, a task background pressure, and a task operation type.

[0028] The task resource consumption amount is used to represent the amount of resources consumed in executing the task. The resource amount can be a computing resource or a time resource, for example, the resource amount can be used to represent the number of computing nodes required to execute the task, or the time required to consume in executing the task according to a preset process, etc.

[0029] In addition, the task resource consumption amount can also be used to represent the consumption of the to-be-tested task for public resources. In the concurrent processing of tasks, multiple different tasks can involve the occupation of the same resource, which can be a computing resource of a computing node or a shared resource that needs to be locked. When the usage proportion of the to-be-tested task for the public resource is higher, the execution result of this kind of special task is better reflected, and then used to feed back the execution process of the task.

[0030] The task priority can be a pre-set priority level, for example, a manager can pre-set the priority level for different types of tasks according to the different importance of the tasks, and the pre-set task priority can be directly called to determine the key tasks in screening. The task priority can also be associated with the attributes of the to-be-tested task. When the to-be-tested task corresponds to different categories, the priority of the to-be-tested task can be determined according to the corresponding relationship between the category and the priority.

[0031] In some specific embodiments, when determining the task priority, the association between the to-be-tested task and the influence factor can be used for determination. The influence factor can be a factor directly reflecting the execution effect of the task. For example, in the processing process of bank business, the influence factor can include at least one of business operation, year-end settlement, account, and customer experience. In actual application, the corresponding influence factor can be determined according to the application environment of the business and the characteristics of the business itself, and is not limited to the above examples, which will not be described here.

[0032] The task complexity is directly associated with the data amount and scale of the task itself, for example, the task with larger data amount has higher task complexity. The task complexity can also be related to the hierarchical structure of the task. When the task is a task executed in sequence according to a fixed process, the execution of the task can be completed directly according to the corresponding script program, and accordingly has lower complexity; when the task has different sub-tasks, and the sub-tasks have associated relationship or execution sequence, etc., the automation degree is low in executing the to-be-tested task, and accordingly has higher task complexity.

[0033] The task-corresponding time window can correspond to the time window for executing the task. The length of the time window can represent the time consumed in executing the task, and the position of the time window can be used to represent the order or time of task execution. Analyzing the task-corresponding time window can comprehensively consider the influence relationship between time windows.

[0034] Task background pressure can represent the correlation between a task and its execution environment. For example, task background pressure can represent the relationship between a task and an online transaction. When an online transaction process exists, there is a more complex interaction between the task and the execution environment. Task background pressure can also refer to batch processing background. Different tasks under test can have different batch processing backgrounds, which are used to indicate the impact of the task on the batch processing execution environment.

[0035] Task operation types reflect the operations involved in a task and can be categorized by administrators based on different execution methods or characteristics. Specific task operation types may include, for example, large file operations, cross-database operations, and full table processing of a database. Analyzing task execution flows based on operation types allows for effective integration with task analysis.

[0036] When selecting key tasks, one or more of the tasks listed above can be analyzed based on actual application needs, without any restrictions.

[0037] Once the task nature is determined, priority tasks can be filtered based on this nature. Combining the specific descriptions of task nature above, corresponding filtering criteria can be pre-set to intuitively prioritize and identify key tasks. For example, tasks consuming a certain amount of resources can be prioritized, or tasks meeting certain priority conditions can be prioritized. Specific filtering criteria can be set according to the needs of the actual application, and will not be elaborated further here.

[0038] S130: Obtain the single batch execution results when each key task is executed individually.

[0039] After obtaining the key tasks, each key task can be executed individually in a single-batch scenario to obtain the single-batch execution results for each key task. In a single-batch scenario, you can either execute only one task to obtain the corresponding test results, or you can schedule only one key task to be executed in the original batch execution environment, thereby achieving the effect of single-batch execution.

[0040] The single batch execution result can include a task execution result and task execution data. The task execution result is used to directly reflect whether the task is successfully executed. The task execution data can reflect the execution effect of the task, such as the task execution time, the number of computing nodes processing the task, and the occupation of memory / servers / databases and other parameters. The test effect can be more detailed through the task execution data.

[0041] In some embodiments, in the single batch scenario, the non-functional test can be used to obtain the single batch execution result of each key task. Unlike the functional test, which only tests whether the function of the application or program is available, the non-functional test generally tests performance, load, security, reliability, and many other aspects. The non-functional test is generally a test performed in a challenging environment, such as in a low-resource or no-resource environment, to obtain the corresponding parameters during task execution. Accordingly, the test result of the non-functional test cannot be directly obtained by testing, but needs to be indirectly inferred, for example, by testing the failure rate corresponding to different loads to determine the load capacity. The specific project of the non-functional test can be set according to the actual application requirements, and will not be described here.

[0042] Since the execution of each key task is performed separately, there is no resource occupation and public resource contention between the key tasks, and there is a certain difference between the task batch execution process. Therefore, in the subsequent process, the final test result can be determined by comparing the single batch execution result with the result obtained by executing all the tasks in the batch.

[0043] In some embodiments, for the single batch test result, it can also be considered whether the execution of each key task meets the task execution index, so as to complete the test of the key task in advance according to the single batch scenario.

[0044] The task execution index can be a pre-set index for judging the execution effect of the task, for example, the task execution index can be judged by processing capacity, time window, and resource consumption. The processing capacity can represent whether each key task is successfully executed, and the number of exceptions in the execution process. The time window can represent the time spent by the key task and the specific time of execution. The resource consumption can represent the utilization degree of the computing resource, such as the occupation of memory, server, and database. The task execution index can be set based on the above specific content, for example, whether the task execution is successful, or a certain amount of resource consumption and time consumption is set to judge the execution effect of the task. The specific setting of the task execution index can be set based on the actual application requirements, and is not limited.

[0045] S140: In the batch execution scenario, obtain the full batch execution result of each key task.

[0046] After obtaining the execution result of each key task in the single batch execution scenario, in order to facilitate comparison, the full batch execution result of each key task can also be obtained in the batch execution scenario.

[0047] The batch execution scenario is to perform batch processing on all to-be-tested tasks according to the execution flow of batch processing. In the batch execution scenario, each to-be-tested task is batch processed according to a preset program script.

[0048] The full batch execution result is used to reflect the execution result of each to-be-tested task in the batch execution scenario. The full batch execution result can include whether the to-be-tested task is executed successfully, or can include relevant parameters reflected in the process of execution of the to-be-tested task, such as the use of resources, the time consumed in execution, and the like. In some embodiments, the content involved in the full batch execution result can correspond to the content involved in the single batch execution result, so as to facilitate comparison in the subsequent steps to obtain the final test result.

[0049] In some embodiments, the batch execution scenario is an execution scenario under a distributed architecture, that is, each to-be-tested task is processed by a distributed cluster. The distributed cluster includes a plurality of distributed nodes, and each distributed node can complete the processing of its own task in an autonomous manner. When batch processing all to-be-tested tasks, each to-be-tested task can be allocated to each distributed node based on the load balancing between the distributed nodes based on the requirements of the processing flow, and each distributed node can process the to-be-tested task respectively to obtain the corresponding full batch execution result.

[0050] Based on the above embodiments, the data in the database corresponding to the distributed cluster can be stored based on a narrow table structure. The narrow table is a concept compared with the wide table, in which the database table involves more fields, and the narrow table structure strictly follows the three normal forms of database design, thereby greatly reducing the degree of data redundancy, ensuring the simplicity of the database structure, improving the storage capacity of the database, and being more suitable for batch processing task execution environment.

[0051] Correspondingly, the batch execution environment can also be an environment built based on the Spring Batch framework. The Spring Batch framework is a lightweight and comprehensive batch processing framework for developing powerful enterprise-level batch processing applications for daily operations. Based on the Spring Batch framework to build the batch execution environment, each block can be used to process the business respectively, thereby effectively ensuring the execution effect of batch processing.

[0052] Since the subsequent steps mainly involve comparison between the key tasks, although all the to-be-tested tasks are involved in the full batch execution scenario, in order to reduce the time consumed by analysis, only the full batch execution results corresponding to the key tasks can be extracted, so as to effectively obtain the test results while improving the test efficiency.

[0053] The full batch execution result is used to reflect the execution performance of each key task, and specifically can be embodied in task execution integrity, batch processing efficiency, task execution time window state, server resource usage, public resource usage and the like.

[0054] The task execution integrity is used to reflect the integrity of the execution process corresponding to the key task. The integrity can be embodied in the normal execution process of the task, that is, whether the task is completely executed under the condition of normal execution. In some other examples, corresponding abnormal conditions and fault conditions can also be constructed, such as the case that the computing resource is unavailable or the database fails, to determine whether the task can be completely executed, so as to further determine the robustness of the task execution.

[0055] The batch processing efficiency is used to analyze the execution efficiency of the task, which can be associated with the execution time and the complexity of the task itself. For tasks of the same complexity, the less time spent, the higher the execution efficiency, and the more reasonable the allocation and processing of the task.

[0056] The task execution time window state is used to describe the time window corresponding to the task execution, including the time length and the time of task start and end, so as to comprehensively compare the time windows of different tasks to analyze the influence conditions between different tasks.

[0057] The server resource usage can be the usage of computing resources of the server in the task execution process, such as the number of computing nodes occupied, or refined to the number of processes occupied, so as to more specifically determine the related conditions of the resources used in the task execution.

[0058] The public resource usage is used to represent the occupation of the public resources by different tasks in the batch processing environment. In the case of multiple nodes processing tasks in parallel, the public resources can be resources shared between nodes, such as shared storage, shared computing nodes and the like. In order to further analyze the influence relationship between different tasks, obtaining the public resource usage for analysis can effectively determine the specific execution conditions of the tasks.

[0059] The above is only an exemplary introduction to the full batch execution result, and other full batch execution results can be set according to the requirements in actual application, and no limitation is made thereto.

[0060] In some embodiments, the obtaining the full batch execution result includes resource scheduling conditions of each key task. In the obtaining the full batch execution result, the resource scheduling conditions of each key task in the batch execution scenario are obtained. The resource scheduling conditions include at least one of resource load balancing, resource scheduling fault switching, and job automatic scheduling.

[0061] The resource load balancing is a description of load conditions of different distributed nodes in a distributed architecture. Since different distributed nodes have different computing capabilities, and the amount of tasks allocated to different distributed nodes also differs, different distributed nodes have different load conditions, for example, some nodes are overloaded, while some nodes are in an idle state. In order to fully utilize the computing capabilities of each node, the resource load balancing needs to be analyzed to achieve load balancing.

[0062] The resource scheduling fault switching is used to determine whether the resource node can timely switch to other nodes in the case of sudden failure to ensure the continuous and smooth execution of the task. The sudden failure of the resource node can be artificially set to test the processing effect of the to-be-tested task.

[0063] The job automatic scheduling condition can represent the automatic execution condition of the task. Since the automatic execution of the to-be-tested task by using the program script is mainly involved in the batch processing environment, the automatic execution effect of the to-be-tested task can be effectively determined by analyzing the job automatic scheduling condition, which is also used to reflect the full batch execution result of the task.

[0064] After the at least one resource scheduling condition is obtained, the resource scheduling condition can be used as the full batch execution result, or the corresponding analysis result can be used as the full batch execution result after the resource scheduling condition is analyzed. In actual applications, other resource scheduling conditions can also be set according to test requirements and actual application needs, and are not limited to the above examples, which will not be described here.

[0065] In some embodiments, the full batch execution result includes a database calling condition. In the obtaining the full batch execution result, the database calling condition in the batch execution scenario is obtained. Since the to-be-tested task may also involve calling data in the database during the execution process, there may also be data occupation and the like, and therefore the corresponding database calling condition also needs to be analyzed.

[0066] Specifically, the database calling condition can include at least one of an RAC switching availability condition, a database exception influence on task processing condition, and a database shared storage availability condition.

[0067] RAC (Oracle Real Application Cluster) is an Oracle grid computing technology, which is generally supported by two or more homogeneous computers and storage devices to support the grid computing environment. RAC has the advantages of high availability, high performance, and on-demand replacement. When using RAC, the process of executing the to-be-tested task may involve switching between the databases involved in RAC, and verifying the availability of RAC switching can effectively determine the availability of the database and support the application of RAC.

[0068] The influence of the database exception on the task processing can be the execution effect of the to-be-tested task in the case of a database exception. The database exception can be a simulated abnormal situation, which is used to obtain the response capability for the sudden situation of the database exception in the task execution process. In the case of a database exception, whether there is backup data that can be called and whether the task can be completed without relying on the data in the database can also be the test content of the to-be-tested task, so as to further analyze the high availability of the database.

[0069] The database shared storage availability is used to describe the calling situation of the shared content between different databases in the case of shared storage content between databases. In actual application, in order to save storage space, there can be shared storage between databases, that is, there is corresponding content in a shared manner to save storage space and improve the callability of data. When batch tasks are executed, since the concurrent execution of tasks can involve the calling of databases in different databases, the database shared storage availability can be analyzed. In the specific test process, the corresponding data can be placed in the shared storage unit according to the data calling situation in the to-be-tested task execution process, so as to verify the database shared storage situation.

[0070] The above-mentioned RAC switching availability, the influence of the database exception on the task processing, and the database shared storage availability are mainly for analyzing and testing the influence of the database on the batch task execution process. Since the calling situation of the database has a great influence on the test result, the influence of the database on the execution result should also be considered during testing to ensure the effective calling of the data in the database.

[0071] After obtaining the above-mentioned at least one database calling situation, the above-mentioned database calling situation can be used as the full batch execution result, or the corresponding analysis result can be used as the full batch execution result after analyzing the above-mentioned database calling situation. In actual application, other database calling situations can also be set according to the test requirements and actual application needs, and are not limited to the above-mentioned examples, which will not be described here.

[0072] In some embodiments, the construction of the batch execution scenario can be completed in advance before the full batch execution process is performed. When the batch execution scenario is constructed, the execution overlap of each task to be tested can be determined first. The execution overlap is used to reflect the influence of the key tasks on each other during the execution process. The execution overlap includes time period overlap and / or data resource overlap. Since batch processing involves batch processing of a large number of tasks, different tasks can need to be executed in the same time period or different tasks can need to call the same computing resources for execution. The above execution overlap can be used to determine the influence relationship between the key tasks, and then the construction of the batch execution scenario can be completed to ensure the batch execution effect of the key tasks.

[0073] S150: determining a test result based on the comparison result of the single batch execution result and the full batch execution result.

[0074] After the single batch execution result and the full batch execution result are obtained, the single batch execution result and the full batch execution result can be compared. Since the single batch execution result and the full batch execution result are respectively used to reflect the execution result of executing a task alone and the execution result of executing all tasks in batch, according to the comparison result of the single batch execution result and the full batch execution result, the difference between the two can be analyzed, and the interference of the batch processing execution process on the execution process of the task to be tested can be determined intuitively and effectively.

[0075] Specifically, the corresponding analysis index can be set in advance. If the difference determined according to the comparison result meets the definition of the analysis index, it can be judged that the batch processing execution condition interferes with the effective execution of the task.

[0076] After the test result is determined, the task execution process can be adjusted based on the test result, for example, the allocation condition of the resource is changed, or the order between specific tasks is staggered to ensure the effective performance of the task in the batch processing process. The specific adjustment process can be set based on the demand of actual application, which will not be described here.

[0077] In some embodiments, the abnormal processing condition corresponding to each key task can also be obtained under an abnormal test scenario. The abnormal test scenario can be a special scenario set by a person, such as resource unavailability, execution process error, etc. Correspondingly, the abnormal processing condition includes at least one of emergency handling condition, associated job influence condition and abnormal recovery condition. The emergency handling condition can be that the task execution process can automatically and effectively cope with the abnormal situation. The associated job influence condition can be the execution condition of other associated tasks of the task involved in the abnormal situation. The abnormal recovery condition can be whether the task can effectively recover from the abnormality, continue or re-execute to normally complete the execution process.

[0078] According to the abnormal processing condition, a corresponding test result can be determined, and the coping ability for the abnormality in the task execution process can be effectively analyzed.

[0079] According to the test method for the batch processing task in the above embodiments, it can be seen that when the batch processing task is tested, the key tasks are first selected from at least one group of to-be-tested tasks according to the task properties of the to-be-tested tasks, the single batch execution results of executing each key task are respectively obtained, the full batch execution results of each key task in the batch execution scene are obtained, and the test of the batch processing task is realized by comparing the single batch execution results and the full batch execution results. The above process reduces the workload of test analysis by screening key tasks, compares the single batch execution results and the full batch execution results to effectively determine the influence between different tasks in the full batch execution, and then effectively reflects the effect of batch execution through the test result, so that the test of the batch processing task in the batch processing environment is efficiently and accurately completed, and the production research is facilitated.

[0080] Based on Figure 1 According to the batch processing task test method, a batch processing task test device is introduced. The batch processing task test device is arranged in the batch processing task test equipment. As shown in Figure 2 The batch processing task test device includes the following modules.

[0081] The to-be-tested task acquisition module 210 is configured to acquire at least one group of to-be-tested tasks.

[0082] The key task screening module 220 is configured to screen key tasks from the at least one group of to-be-tested tasks based on the task properties of the to-be-tested tasks.

[0083] The single batch execution result acquisition module 230 is configured to acquire single batch execution results of executing each key task respectively.

[0084] The full batch execution result acquisition module 240 is configured to acquire full batch execution results of each key task in a batch execution scene.

[0085] The test result determination module 250 is configured to determine a test result based on a comparison result of the single batch execution results and the full batch execution results.

[0086] In some embodiments, the task properties include at least one of a task resource consumption amount, a task priority, a task complexity, a task corresponding time window, a task background pressure, and a task operation type.

[0087] In some embodiments, the task priority is determined based on the relevance between the to-be-tested task and an impact factor; the impact factor comprises at least one of business operation, year-end settlement, account and customer experience.

[0088] In some embodiments, the batch task testing apparatus further comprises: an execution overlap determining module configured to determine an execution overlap of each to-be-tested task; the execution overlap comprises execution time period overlap and / or data resource utilization overlap; and a batch execution scenario constructing module configured to construct the batch execution scenario based on the execution overlap.

[0089] In some embodiments, the full batch execution result obtaining module 240 comprises: a resource scheduling condition obtaining unit configured to obtain a resource scheduling condition of each key task under the batch execution scenario; the resource scheduling condition comprises at least one of resource load balancing condition, resource scheduling fault switching and job automatic scheduling condition; and a full batch execution result determining unit configured to determine the full batch execution result according to the resource scheduling condition.

[0090] In some embodiments, the full batch execution result obtaining module 240 comprises: a database calling condition obtaining unit configured to obtain a database calling condition under the batch execution scenario; the database calling condition comprises at least one of RAC switching availability condition, database exception influence on task processing condition and database common storage availability condition.

[0091] In some embodiments, the to-be-tested task corresponds to at least one execution system; the execution systems have an association relationship; and the single batch execution result and / or the full batch execution result comprises data supply influence effect and / or data dependency effect between the execution systems with the association relationship.

[0092] In some embodiments, each to-be-tested task under the batch execution scenario is processed by a distributed cluster.

[0093] In some embodiments, data in a database corresponding to the distributed cluster is stored based on a narrow table structure.

[0094] In some embodiments, the batch execution scenario is constructed based on a Spring Batch framework.

[0095] In some embodiments, the full batch execution result comprises task execution integrity, batch processing efficiency, task execution time window state, server resource usage and public resource usage.

[0096] In some embodiments, the batch task testing device further comprises an exception handling condition acquisition module configured to acquire an exception handling condition of each key task in an exception testing scenario; the exception handling condition comprises at least one of an emergency handling condition, an associated operation influence condition and an exception recovery condition; and a test result acquisition module configured to determine a test result according to the exception handling condition.

[0097] Based on Figure 1 Corresponding to the batch task testing method, the present application provides a computer readable storage medium having computer programs / instructions stored thereon. As shown in Figure 3 The computer readable storage medium can be read by a processor based on an internal bus of the device, and then the program instructions in the computer readable storage medium are implemented by the processor.

[0098] In the present embodiment, the computer readable storage medium can be implemented in any appropriate manner. The computer readable storage medium includes, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a cache, a hard disk drive (HDD), a memory card, and the like. The computer storage medium stores computer program instructions. When the computer program instructions are executed, the program instructions implement the steps of the present specification Figure 1 The program instructions or modules of the corresponding embodiments.

[0099] In the present embodiment, the processor can be implemented in any appropriate manner. For example, the processor can take the form of, for example, a microprocessor or processor and a computer readable medium storing computer readable program code (e.g. software or firmware) executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller and an embedded microcontroller, and the like. Specifically, the processor can execute the computer program instructions to implement the following steps: acquiring at least one group of tasks to be tested; based on the task properties of the tasks to be tested, screening key tasks from the at least one group of tasks to be tested; respectively acquiring single batch execution results when each key task is executed individually; acquiring full batch execution results of each key task in a batch execution scenario; and determining a test result based on a comparison result of the single batch execution results and the full batch execution results.

[0100] Based on Figure 1Corresponding to the batch task test method, the present application provides a computer program product, which stores computer programs / instructions. The computer program product can be a program written by a corresponding computer program language, stored in a corresponding storage device in a program manner, and can be transmitted through a computer network. The computer program product can be executed by a processor. In the present application, the computer program product realizes the method as Figure 1 The program instructions or modules of the corresponding embodiments.

[0101] It should be noted that the above batch task test method, device, storage medium and program product can be applied to the field of big data technology, and can also be applied to other technical fields except the field of big data technology, and no limitation is made to this.

[0102] Although the process flow described above includes a plurality of operations appearing in a specific order, it should be clearly understood that the processes can include more or fewer operations, which can be executed sequentially or in parallel (for example, using parallel processors or multi-threaded environments).

[0103] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system) and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one flow or multiple flows and / or blocks Figure 1 The functions specified in one flow or multiple flows and / or blocks

[0104] These computer program instructions can also be stored in a computer readable storage medium, which can guide the computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable storage medium produce a product including instruction devices, which realize the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one flow or multiple flows and / or blocks Figure 1 The functions specified in one flow or multiple flows and / or blocks

[0105] These computer program instructions can also be loaded into a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide a process for realizing the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one flow or multiple flows and / or blocks ​steps of a function specified in one or more blocks.

[0106] In one typical arrangement, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0107] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) having a cache area for the temporary storage of data. The memory can also include non-volatile memory, such as read only memory (ROM), electrically erasable read-only memory (EEPROM), flash memory, or other non-volatile memory technology. The memory can be an example of computer readable media.

[0108] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for the storage of information. The information can be computer readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile discs (DVDs) or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer readable media does not include transitory media, such as modulated data signals and carrier waves.

[0109] Those skilled in the art will appreciate that embodiments of the present specification can be provided as methods, systems or computer program products. Accordingly, embodiments of the present specification can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present specification can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage and the like) containing computer usable program code.

[0110] Embodiments of the present specification can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. Embodiments of the present specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including memory storage devices.

[0111] Each of the embodiments in the present specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, for the system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the embodiments of the present specification. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, a person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0112] The above only describes the embodiments of the present application and is not intended to limit the present application. The present application can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. A method of testing batch tasks, characterized by, The method comprises the following steps: obtaining at least one group of to-be-tested tasks, wherein the to-be-tested tasks are tasks processed based on a batch processing execution mode under a distributed architecture; screening key tasks from the at least one group of to-be-tested tasks based on task properties of the to-be-tested tasks, wherein the task properties of the to-be-tested tasks reflect characteristics of the tasks, and the task properties are associated with execution processes and execution effects of the tasks, so that the key tasks are screened from the to-be-tested tasks by taking the task properties into consideration when the key tasks are determined; obtaining single-batch execution results of separately executing each key task, wherein the single-batch execution results include task execution results and task execution data, wherein the task execution results directly reflect whether the tasks are successfully executed, and the task execution data reflects execution effects of the tasks, wherein the task execution data includes task execution time, the number of computing nodes processing the tasks, and the occupation of memory / servers / databases; obtaining full-batch execution results of each key task under a batch execution scenario; determining a test result based on a comparison result of the single-batch execution results and the full-batch execution results.

2. The method of claim 1, wherein, The task properties include at least one of a task resource consumption amount, a task priority, a task complexity, a task corresponding time window, a task background pressure, and a task operation type.

3. The method of claim 2, wherein, The task priority is determined based on an association between the to-be-tested tasks and influence factors, wherein the influence factors include at least one of business operation, year-end settlement, account, and customer experience.

4. The method of claim 1, wherein, Before obtaining the full-batch execution results of each key task under the batch execution scenario, the method further comprises the following steps: determining execution overlap conditions of each to-be-tested task, wherein the execution overlap conditions include execution time period overlap and / or data resource utilization overlap; constructing the batch execution scenario based on the execution overlap conditions.

5. The method of claim 1, wherein, The method of obtaining the full-batch execution results of each key task under the batch execution scenario comprises the following steps: obtaining resource scheduling conditions of each key task under the batch execution scenario, wherein the resource scheduling conditions include at least one of resource load balancing conditions, resource scheduling fault switching, and job automatic scheduling conditions; determining the full-batch execution results according to the resource scheduling conditions.

6. The method of claim 1, wherein, The full-batch execution results include database invocation conditions. The method of obtaining the full-batch execution results of each key task under the batch execution scenario comprises the following steps: obtaining database invocation conditions under the batch execution scenario; The database invocation conditions include at least one of RAC switching availability, the influence of database exceptions on task processing, and database shared storage availability.

7. The method of claim 1, wherein, The to-be-tested tasks correspond to at least one execution system, and the execution systems have an association relationship; the single-batch execution results and / or the full-batch execution results include data supply influence effects and / or data dependency effects between the execution systems having the association relationship.

8. The method of claim 1, wherein, Each to-be-tested task is processed by a distributed cluster under the batch execution scenario.

9. The method of claim 8, wherein, Data in a database corresponding to the distributed cluster is stored based on a narrow table structure.

10. The method of claim 8, wherein, The batch execution scenario is constructed based on a Spring Batch framework.

11. The method of claim 1, wherein, The full batch execution result includes task execution integrity, batch processing efficiency, task execution time window state, server resource usage, and public resource usage.

12. The method of claim 1, wherein, After obtaining the full batch execution result of each key task in the batch execution scenario, the method further includes: In the abnormal test scenario, obtaining an abnormal processing condition corresponding to each key task, wherein the abnormal processing condition includes at least one of emergency processing condition, associated job impact condition, and abnormal recovery condition; Determining a test result according to the abnormal processing condition.

13. A batch task testing apparatus characterized by comprising: The method includes: a to-be-tested task acquisition module configured to acquire at least one group of to-be-tested tasks, wherein the to-be-tested tasks are tasks processed based on a batch processing execution mode under a distributed architecture; a key task screening module configured to screen key tasks from the at least one group of to-be-tested tasks based on task properties of the to-be-tested tasks, wherein the task properties of the to-be-tested tasks reflect characteristics of the tasks, and the task properties are associated with execution processes and execution effects of the tasks, so that the key tasks are screened from the to-be-tested tasks by taking the task properties into consideration when determining the key tasks; a single batch execution result acquisition module configured to acquire single batch execution results of executing each key task individually, wherein the single batch execution results include task execution results and task execution data, wherein the task execution results directly reflect whether the tasks are successfully executed, and the task execution data reflects execution effects of the tasks, and wherein the task execution data includes task execution time, a number of computing nodes processing the tasks, and memory / server / database occupation; a full batch execution result acquisition module configured to acquire full batch execution results of each key task in a batch execution scenario; a test result determination module configured to determine a test result based on a comparison result of the single batch execution results and the full batch execution results.

14. A computer readable storage medium having stored thereon computer programs / instructions, characterized in that, The computer program / instructions, when executed by the processor, implement the steps of the method of any one of claims 1-12.

15. A computer program product having stored thereon computer program / instructions, characterized in that, The computer program / instructions, when executed by the processor, implement the steps of the method of any one of claims 1-12.

Images