Data processing job scheduling delay test method and program product

Abstract

The invention discloses a data processing job scheduling delay test method and a program product. The method includes performing a plurality of data processing jobs; wherein a part of the data processing jobs are provided with job concurrency limitation; an upstream and downstream dependency relationship exists among part of the data processing operations; recording the script actual execution time of the data processing script of the data processing job without job concurrency limitation, and determining the single scheduling delay of the data processing job according to the upstream and downstream information of the data processing job, the job plan execution time and the script actual execution time; determining a target scheduling delay according to the single scheduling delays and the total job execution duration of the plurality of data processing jobs, and determining performance optimization prompt information of the execution system according to the target scheduling delay; therefore, accurate evaluation of the scheduling delay of the data processing operation is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a data processing operation scheduling delay testing method and program product. BACKGROUND

[0002] In a large-scale data processing scenario, the number of operations is often large, and the dependency relationship between upstream and downstream operations is complex. In addition, the planned execution time of the operation is not fixed, and the actual running time is uncertain, which makes the execution system of the data processing operation face great scheduling pressure and high uncertainty. At the same time, there is currently a lack of effective scheduling delay testing strategies, making it difficult to effectively evaluate and quantitatively analyze the scheduling performance of the execution system, and unable to accurately identify performance bottlenecks in operation scheduling.

[0003] The existing general delay testing method usually constructs a small number of random running time and no dependency relationship data processing operations, and judges the scheduling delay by comparing the operation plan execution time of each data processing operation and the operation state conversion start time in running, but this method cannot truly reflect the scheduling performance under complex operation link, lacks representativeness and universality, and is difficult to effectively evaluate the data scheduling performance of the execution system. Therefore, there is an urgent need for a data processing operation scheduling delay testing method to accurately quantify and evaluate the scheduling delay performance of the execution system. SUMMARY

[0004] The present application provides a data processing operation scheduling delay testing method and program product to solve the problem of lack of testing method for data processing operation scheduling delay under complex dependency relationship in the prior art, and difficulty in accurately quantitatively evaluating the scheduling performance of the execution system and identifying performance bottlenecks.

[0005] According to an aspect of the present application, a data processing operation scheduling delay testing method is provided, which comprises:

[0006] executing a plurality of data processing operations; wherein part of the data processing operations is provided with operation concurrency limitation; there is an upstream and downstream dependency relationship between part of the data processing operations; the operation concurrency limitation is less than the maximum concurrency limitation of the execution system of the data processing operation;

[0007] recording the script actual execution time of the data processing script of the data processing operation which is not provided with operation concurrency limitation, and determining the single scheduling delay of the data processing operation according to the upstream and downstream information, operation plan execution time and script actual execution time of the data processing operation;

[0008] The target scheduling delay is determined according to the single scheduling delay and the total job execution duration of the plurality of data processing jobs, and the performance optimization prompt information of the execution system is determined according to the target scheduling delay; wherein the performance optimization prompt information comprises a performance item to be optimized and / or a performance optimization mode.

[0009] According to another aspect of the present application, a data processing job scheduling delay testing device is provided, which comprises:

[0010] A data processing job execution module is configured to execute a plurality of data processing jobs; wherein part of the data processing jobs is provided with a job concurrency degree limit; there is an upstream and downstream dependency relationship between part of the data processing jobs; and the job concurrency degree limit is less than the maximum concurrency degree limit of the execution system of the data processing jobs.

[0011] A single scheduling delay determination module is configured to record a script actual execution time of a data processing script of the data processing job which is not provided with a job concurrency degree limit, and determine a single scheduling delay of the data processing job according to upstream and downstream information of the data processing job, a job plan execution time and the script actual execution time.

[0012] A performance optimization prompt module is configured to determine a target scheduling delay according to the single scheduling delay and the total job execution duration of the plurality of data processing jobs, and determine performance optimization prompt information of the execution system according to the target scheduling delay; wherein the performance optimization prompt information comprises a performance item to be optimized and / or a performance optimization mode.

[0013] According to another aspect of the present application, an electronic device is provided, which comprises:

[0014] at least one processor; and

[0015] a memory connected in communication with the at least one processor; wherein

[0016] The memory stores a computer program which can be executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the data processing job scheduling delay testing method according to any one of the embodiments of the present application.

[0017] According to another aspect of the present application, a computer readable storage medium is provided, which stores computer instructions for enabling a processor to execute the data processing job scheduling delay testing method according to any one of the embodiments of the present application.

[0018] According to another aspect of the present application, the present disclosure further provides a computer program product comprising a computer program which, when executed by a processor, implements the data processing job scheduling delay testing method according to any one of the present disclosure.

[0019] The technical scheme of the present application comprises the following steps: firstly, a plurality of data processing jobs are executed, wherein some of the data processing jobs are set with job concurrency degree limits, and there is an upstream and downstream dependency relationship between some of the data processing jobs; the job concurrency degree limit is less than the maximum concurrency degree limit of the execution system of the data processing jobs; setting the concurrency degree limit and the upstream and downstream dependency relationship can simulate the complex scheduling environment in a real business scenario, improve the representativeness of the test environment, and control the concurrency degree below the maximum limit of the execution system, so as to evaluate the response capability and stability of the scheduling performance under resource limited conditions within a controllable range; then, the script actual execution time of the data processing script of the data processing job without setting the job concurrency degree limit is recorded, the single scheduling delay of the data processing job is determined according to the upstream and downstream information of the data processing job, the job plan execution time and the script actual execution time; by collecting the deviation between the actual execution time and the planned execution time, the scheduling delay is accurately identified, and the quantifiability and analyzability of the test results are enhanced; combined with the upstream and downstream dependency information, the non-planned delay caused by dependency blocking or resource waiting can be effectively identified, and the accuracy of the scheduling behavior is improved; finally, the target scheduling delay is determined according to the single scheduling delay of the data processing job and the total job execution time, and the performance optimization prompt information of the execution system is determined according to the target scheduling delay; wherein the performance optimization prompt information comprises the performance item to be optimized and / or the performance optimization mode; by summarizing and analyzing the scheduling behaviors of multiple jobs, the target scheduling delay is determined, the scheduling bottleneck at the system level is identified, the statistical significance and guiding value of the test results are improved, the performance optimization prompt information with guiding value is generated, a clear direction and basis for system optimization are provided, and the scientificity, accuracy and practicality of the scheduling delay testing are improved, thereby providing strong support for performance evaluation and optimization of the execution system.

[0020] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to make the technical solution in the embodiments of the present application clearer, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0022] Figure 1 is a flow chart of a data encryption operation scheduling delay test method according to the first embodiment of the present application;

[0023] Figure 2 is a flow chart of a data encryption operation scheduling delay test method according to the second embodiment of the present application;

[0024] Figure 3 is a flow chart of a data encryption operation scheduling delay test method according to the third embodiment of the present application.

[0025] Figure 4 is a structural schematic diagram of a data encryption operation scheduling delay test device according to the fourth embodiment of the present application;

[0026] Figure 5 is a structural schematic diagram of an electronic device implementing the data encryption operation scheduling delay test method according to the embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the technical solution in the embodiments of the present application clearer, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0028] It should be noted that the terms "first", "second", "target" and the like in the specification and claims of the present application and the above drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to include only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product or device.

[0029] It should be noted that the modification of "one", "multiple" mentioned in the present disclosure is illustrative but not restrictive, and those skilled in the art should understand that "one or more" should be understood unless otherwise explicitly indicated in the context.

[0030] The names of the messages or information exchanged between the plurality of devices in the embodiments of the present disclosure are only for illustrative purposes, and are not used to limit the scope of the messages or information.

[0031] It can be understood that before using the technical solutions disclosed in the embodiments of the present disclosure, the type, use range, use scenario, etc. of the personal information involved in the present disclosure should be informed to the user and the authorization of the user should be obtained through appropriate means according to relevant laws and regulations.

[0032] For example, in response to receiving the active request of the user, the prompt information is sent to the user to explicitly prompt the user that the operation requested to be performed will require obtaining and using the personal information of the user. Thus, the user can voluntarily choose whether to provide the personal information to the software or hardware such as electronic device, application program, server or storage medium, etc. that performs the operation of the technical solutions of the present disclosure according to the prompt information.

[0033] As an optional but non-limiting implementation manner, in response to receiving the active request of the user, the prompt information can be sent to the user in the form of a pop-up window, and the prompt information can be presented in the form of text in the pop-up window. In addition, the pop-up window can also carry selection controls for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0034] It can be understood that the above notification and user authorization process is only illustrative, and does not limit the implementation manner of the present disclosure, and other manners meeting the relevant laws and regulations can also be applied to the implementation manner of the present disclosure.

[0035] It can be understood that the data involved in the present technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of the relevant laws and regulations and relevant provisions.

[0036] Embodiment one

[0037] Figure 1 A flowchart of a data encryption operation scheduling delay test method is provided for the first embodiment of the present application. The present embodiment can be applied to test the scheduling delay of data encryption operation. The method can be executed by a data encryption operation scheduling delay test device, which can be realized in the form of hardware and / or software. Optionally, the data encryption operation scheduling delay test device can be realized by an electronic device, which can be a mobile terminal, a PC terminal or a server, etc. Figure 1As shown, the method can specifically include:

[0038] S110, performing a plurality of data processing jobs; wherein part of the data processing jobs is provided with a job concurrency limit; there is an upstream and downstream dependency relationship between part of the data processing jobs; and the job concurrency limit is less than the maximum concurrency limit of the execution system of the data processing jobs.

[0039] In the embodiments of the present application, the data processing job can be understood as an independent task unit that processes, converts and calculates original data according to certain business logic in the execution system of the data processing job. The job concurrency limit refers to the maximum number of concurrent executions set for a certain data processing job. The job concurrency limit can be used to control the number of concurrent executions, prevent system resources from being excessively occupied, and affect the overall scheduling efficiency and stability. The maximum concurrency limit of the execution system can be understood as the upper limit of the maximum number of concurrent tasks that the entire data processing execution system can support when scheduling and executing tasks. The maximum concurrency limit of the execution system can be determined by the system hardware and scheduling mechanism.

[0040] Optionally, the job concurrency limit can be set for part of the data processing jobs in the plurality of data processing jobs. For example, 60% of the data processing jobs are configured with concurrency limit, and 40% of the data processing jobs are not subjected to data concurrency limit, to simulate the difference in system resource demand of different tasks in a real scheduling scenario, avoid excessive resource occupation by a single type of job, prevent local congestion or resource depletion of the execution system due to unbalanced resource allocation, and thus more accurately reproduce the scheduling state of resource competition and dynamic allocation in a real scheduling environment, to provide a more reliable scenario basis for subsequent scheduling delay testing.

[0041] Further, the job concurrency limit can be set to be less than the maximum concurrency limit of the execution system of the data processing jobs, to ensure that the execution system operates within the capability range and avoid scheduling performance degradation due to resource contention.

[0042] In an actual execution system, the upstream and downstream dependency relationship between data processing jobs can be complex. Testing only the jobs without dependency relationship can not comprehensively evaluate the scheduling performance of the execution system under complex dependency relationship. Optionally, the upstream and downstream dependency relationship can be set for part of the data processing jobs, to achieve more comprehensive and representative testing.

[0043] Before the execution of the data processing jobs, one or more data processing scripts can be constructed for the data processing jobs, to define the execution logic and rules of the data processing jobs, so as to perform the job execution task according to the script execution logic of the data processing script. The data processing script can be understood as a program code or script file that implements the specific execution logic of the data processing job.

[0044] As an optional technical solution of the embodiment of the application, before the execution of the plurality of data processing operations, the method further comprises: constructing a plurality of data processing scripts of data processing operations with upstream and downstream dependencies, configuring a script execution logic of the data processing script of each data processing operation, and setting operation concurrency degree limit information for part of the data processing operations; wherein the script execution logic at least comprises a script execution timing.

[0045] The script execution logic refers to a business execution flow and execution rule defined in the data processing script. The script execution timing can be understood as a specific time point or execution condition at which the data processing script is triggered to execute, for example, timing execution, waiting for a preset interval to execute, and the like.

[0046] Optionally, a plurality of data processing scripts of data processing operations with upstream and downstream dependencies can be constructed. The data processing operation and the data processing script can have a one-to-one or many-to-one correspondence, that is, one data processing script can correspond to one or more data processing operations.

[0047] Specifically, the script execution logic of the data processing script of the data processing operation can be configured, for example, the name of the data processing operation corresponding to the data processing script, the script execution timing, and the like.

[0048] Further, the execution timing of the data processing script can also be configured as a preset time length or an average execution time length of the data processing operation obtained according to actual running conditions, for example, the preset time length is set to 1 second to simulate the execution of a second-level task, or the actual execution time length is counted according to historical running data, so as to construct a test environment closer to the real scene. By configuring data processing scripts with different execution time lengths, the scheduling system can be comprehensively tested and evaluated from multiple dimensions.

[0049] By constructing and configuring the data processing script, a clear execution logic path can be provided for the data processing operation, so as to ensure that it is efficiently and stably executed according to the preset test direction.

[0050] Specifically, after multiple data processing jobs start to be executed, they first enter a waiting running queue, then, by executing a dependency detection program of the system, the execution time of each data processing job is compared with the current time, a list of executable jobs and a list of upstream jobs that meet the execution time condition are obtained from the database, the execution status of all upstream jobs of the data processing job is checked, if there is no upstream job or all upstream jobs are executed, the time identifier and the dependency identifier of the data processing job are updated and written into the executable queue. Then, the executable queue is scanned, according to the recorded concurrency limit, the data processing job is obtained from the executable queue, written into the execution queue, the corresponding concurrency is updated, and the job scheduling request is sent; the execution system executes the data processing script of the data processing job according to the job scheduling request and obtains the execution result.

[0051] S120, record the script actual execution time of the data processing script of the data processing job without setting the job concurrency limit, and determine the single scheduling delay of the data processing job according to the upstream and downstream information of the data processing job, the job planned execution time and the script actual execution time.

[0052] Wherein, the upstream and downstream information refers to the dependency relationship information between data processing jobs. For example, the execution of a data processing job depends on the completion state of its upstream task. The script actual execution time refers to the time point when the data processing script actually starts to execute. The job planned execution time refers to the time point when the data processing job is expected to start to execute according to the data processing job scheduling plan. The single scheduling delay can be understood as the deviation value between the actual execution time of the data processing job and its corresponding planned execution time. The single scheduling delay can be used to measure the delay degree of single data scheduling.

[0053] Considering the influence of concurrency setting on scheduling performance test, in the embodiment of the application, only the script actual execution time of the data processing job without setting the job concurrency limit is recorded, and scheduling delay analysis is carried out based on this, so as to exclude the interference of concurrency limit on the test result.

[0054] For the data processing job without upstream data processing job, the single scheduling delay of the data processing job can be directly determined. Specifically, the single scheduling delay of the data processing job is determined according to the upstream and downstream information of the data processing job, the job planned execution time and the script actual execution time, including: for a first data processing job without upstream data processing job in the multiple data processing jobs, the single scheduling delay of the data processing job is determined according to the job planned execution time of the first data processing job and the script actual execution time of its data processing script.

[0055] The first data processing job refers to a data processing job without a preceding dependent task in a plurality of data processing jobs with upstream and downstream dependencies.

[0056] Since the first data processing job does not need to wait for the completion of the upstream task, the single scheduling delay of the data processing job can be determined directly according to the job plan execution time of the first data processing job and the script actual execution time of the data processing script.

[0057] For a data processing job with an upstream processing job, the single scheduling delay of the data processing job is determined according to the execution end time of the nearest upstream processing job, the job plan execution time, and the script actual execution time. Alternatively, the single scheduling delay of the data processing job is determined according to the upstream and downstream information of the data processing job, the job plan execution time, and the script actual execution time, which further includes determining the execution end time of the nearest upstream data processing job of a second data processing job with an upstream data processing job in the plurality of data processing jobs; in the case that the execution end time is earlier than the job plan execution time of the second data processing job, determining the single scheduling delay of the data processing job according to the job plan execution time of the second data processing job and the script actual execution time of the data processing script of the second data processing job; in the case that the execution end time is later than the job plan execution time of the second data processing job, determining the single scheduling delay of the data processing job according to the execution end time and the script actual execution time of the data processing script of the second data processing job.

[0058] The second data processing job refers to a data processing job with a preceding dependent task in a plurality of data processing jobs with upstream and downstream dependencies. The execution end time of the nearest upstream data processing job can be understood as the time point at which the nearest upstream task of the second data processing job with dependencies completes execution.

[0059] For the upstream data processing job of the second data processing job, the difference between the job plan execution time and the script start execution time of the upstream data processing job can be calculated to more accurately identify whether the scheduling delay is caused by the delay of the upstream task or the scheduling problem of the current task itself.

[0060] Further, for the second data processing job with an upstream data processing job in the plurality of data processing jobs, the execution end time of the nearest upstream data processing job of the second data processing job can be calculated according to the execution time of the data processing job configured in the data processing script, and then it is determined whether it is less than the job plan execution time of the second data processing job.

[0061] Optionally, in the case that the execution end time is earlier than the job plan execution time of the second data processing job, the single scheduling delay of the data processing job can be determined by calculating the difference between the job plan execution time of the second data processing job and the actual script execution time of the data processing script thereof.

[0062] Optionally, in the case that the execution end time is later than the job plan execution time of the second data processing job, the single scheduling delay of the data processing job can be determined by calculating the difference between the execution end time and the actual script execution time of the data processing script of the second data processing job.

[0063] By judging and comparing the execution end time of the nearest upstream task of the second data processing job, and then calculating the single scheduling delay thereof in different ways, more accurate data support is provided for scheduling delay analysis, so as to more comprehensively reflect the scheduling performance of the execution system under complex dependency relationships.

[0064] S130, determining a target scheduling delay according to the single scheduling delays and total job execution time of the plurality of data processing jobs, and determining performance optimization prompt information of the execution system according to the target scheduling delay.

[0065] The total job execution time can be understood as the total time consumed from the beginning of the execution of the plurality of data processing jobs to the completion thereof. The target scheduling delay refers to the overall delay index obtained by comprehensively evaluating the scheduling delay of the plurality of data processing jobs.

[0066] Optionally, the target scheduling delay can be calculated according to the single scheduling delay and the total job execution time of the plurality of data processing jobs in a preset time period.

[0067] Specifically, the target scheduling delay can be analyzed to further identify the performance bottleneck in the system scheduling process, and the performance optimization prompt information of the execution system can be generated accordingly. The performance optimization prompt information refers to the optimization suggestions provided for improving the execution scheduling efficiency based on the analysis of the target scheduling delay. The performance optimization prompt information includes but is not limited to the performance items to be optimized and / or the performance optimization mode. The performance items to be optimized refer to the key performance indicators or configuration items in the execution system that affect the scheduling efficiency. The performance optimization mode refers to the specific improvement measures or ways that can be taken to improve the scheduling performance of the execution system.

[0068] For example, when the execution time of the data processing script configured data processing job is in seconds, the data processing job runs for a short time, and the internal flow of the execution system is fast, and it is not easy to cause job accumulation phenomenon. At this time, if the test scheduling delay is high, it can be identified that the performance optimization point is one or more factors such as insufficient efficiency of dependent condition query update or concurrency query update. When the execution time of the configured data processing job is the actual execution time, the data processing job runs for a long time, and is prone to job accumulation, and the waiting running queue is large. At this time, if the test scheduling delay is high, it can be identified that the performance optimization point is one or more factors such as long time of scanning the waiting running queue to obtain executable jobs.

[0069] Further, corresponding optimization suggestions can be provided for different performance items to be optimized.

[0070] When the performance item to be optimized includes dependent condition query update, the performance optimization mode corresponding to the performance item to be optimized can include at least one of the following modes: optimizing the database query statement with too long execution time, read-write separation, database and table separation, using a graph database to store the dependency relationship, and multi-thread query update.

[0071] When the performance item to be optimized includes concurrency query update efficiency, the performance optimization mode corresponding to the performance item to be optimized can include one or more of the following modes: introducing a cache mechanism and / or splitting transactions.

[0072] When the performance item to be optimized includes the way of obtaining executable jobs, the performance optimization mode corresponding to the performance item to be optimized can include one or more of the following modes: adjusting the sequential scanning queue mode to a mode of scanning triggered by a preset event. The preset event can include one or more triggering modes such as time reaching and / or the data processing job of the upstream being completed, reducing full traversal of the queue and redundant scanning of data processing jobs that do not meet the dependent conditions.

[0073] By analyzing the target scheduling delay and determining the performance optimization prompt information, the performance of the data processing job execution system can be effectively improved, and the stability and response capability of the system in a complex task environment can be enhanced.

[0074] The technical scheme of the embodiment of the present application firstly, by executing a plurality of data processing operations; wherein, part of the data processing operation is provided with operation concurrency limit; there is an upstream and downstream dependency relationship between part of the data processing operation; the operation concurrency limit is less than the maximum concurrency limit of the execution system of the data processing operation; setting the concurrency limit and the upstream and downstream dependency relationship can simulate the complex scheduling environment in the real business scenario, improve the representativeness of the test environment, and control the concurrency within the maximum limit of the execution system, so as to evaluate the response ability and stability of the scheduling performance under the condition of limited resources in the controllable range; then, by recording the script actual execution time of the data processing script of the data processing operation which is not provided with operation concurrency limit, determining the single scheduling delay of the data processing operation according to the upstream and downstream information, operation plan execution time and script actual execution time of the data processing operation; by collecting the deviation of the actual execution time and the planned execution time of the operation, the precise identification of the scheduling delay is realized, and the quantifiability and analyzability of the test result are enhanced; combined with the upstream and downstream dependency information, the non-planned delay caused by dependency blockage or resource waiting can be effectively identified, and the description accuracy of the scheduling behavior is improved; finally, by determining the target scheduling delay according to the single scheduling delay and total operation execution time of a plurality of data processing operations, determining the performance optimization prompt information of the execution system according to the target scheduling delay; wherein, the performance optimization prompt information includes the performance item to be optimized and / or the performance optimization mode; by summarizing and analyzing a plurality of operation scheduling behaviors, the target scheduling delay is determined, the scheduling bottleneck at the system level can be identified, the statistical significance and guiding value of the test result are improved, the performance optimization prompt information with guiding value is generated, which provides a clear direction and basis for system optimization, improves the scientificity, accuracy and practicality of the scheduling delay test, and provides strong support for performance evaluation and optimization of the execution system.

[0075] Embodiment two

[0076] Figure 2A flowchart of a data processing operation scheduling delay test method provided for the second embodiment of the present application further describes a specific implementation of determining a target scheduling delay according to the single scheduling delay and total job execution time length of a plurality of data processing jobs. Optionally, the script execution timing of the data processing script of the data processing job comprises waiting for a preset interval time length to execute. The determining of the target scheduling delay according to the single scheduling delay and total job execution time length of a plurality of data processing jobs comprises: after the total job execution time length of a plurality of data processing jobs reaches a first preset time length, determining a first average scheduling delay of the single scheduling delay of a plurality of data processing jobs within the first preset time length, and determining the first average scheduling delay as the target scheduling delay. The specific implementation can be referred to the description of the present embodiment. Wherein, the same or similar technical features as the foregoing embodiments are not described herein. As shown in Figure 2 The method can specifically include:

[0077] S210, a plurality of data processing jobs are executed; wherein, part of the data processing jobs are provided with a job concurrency degree limit; there is an upstream and downstream dependency relationship between part of the data processing jobs; the job concurrency degree limit is less than the maximum concurrency degree limit of the execution system of the data processing job.

[0078] S220, the script actual execution time of the data processing script of the data processing job without setting a job concurrency degree limit is recorded, and the single scheduling delay of the data processing job is determined according to the upstream and downstream information, job plan execution time and script actual execution time of the data processing job.

[0079] S230, after the total job execution time length of a plurality of data processing jobs reaches a first preset time length, a first average scheduling delay of the single scheduling delay of a plurality of data processing jobs within the first preset time length is determined, and the first average scheduling delay is determined as the target scheduling delay.

[0080] In order to more accurately evaluate the scheduling performance of the data processing job execution system, optionally, after the total job execution time length of a plurality of data processing jobs reaches a first preset time length, the scheduling delay of a plurality of data processing jobs within the first preset time length can be statistically analyzed to avoid the deviation of the overall evaluation result caused by the scheduling delay of a single data processing job. Wherein, the first preset time length can be understood as a time window for statistical scheduling delay, which can be flexibly set according to the actual test scene, system load condition and performance evaluation demand.

[0081] Further, the single scheduling delay of each data processing job is recorded and counted within the first preset time length, and an average value thereof is calculated to obtain a first average scheduling delay, which is determined as the target scheduling delay. The first average scheduling delay refers to the average value of the single scheduling delays of the multiple data processing jobs within the first preset time length. The first average scheduling delay can be used to measure the average scheduling delay level of the execution system within the time window.

[0082] Specifically, the single scheduling delay corresponding to each scheduling of each data processing job within the first preset time length can be counted respectively, and the average single scheduling delay corresponding to each data processing job can be obtained by summing and averaging all the single scheduling delays of the data processing jobs. Further, the average single scheduling delays corresponding to all the data processing jobs are summed and averaged again to obtain the first average scheduling delay.

[0083] Optionally, after the execution of the data processing script according to the execution logic of the data processing script and the target scheduling delay of the execution system is counted, the data processing script can be adjusted and modified to test the scheduling delay of the data processing job from multiple dimensions.

[0084] In order to comprehensively test and evaluate the scheduling delay of the execution system under different scheduling strategies and running scenarios, as an optional technical solution of the embodiment, the method further comprises: after the execution time of the data processing job reaches the first preset time length, adjusting the script execution timing of the data processing script of the data processing job according to the data processing script of the data processing job, and returning to execute the operation of executing multiple data processing jobs.

[0085] Optionally, after the execution time of the data processing job reaches the first preset time length, the single scheduling delay data of the data processing job within the first preset time length can be collected and statistically analyzed, and the script execution timing of the data processing script corresponding to the current data processing job can be dynamically adjusted according to the historical execution of the data processing script, wherein the script execution timing of the data processing script of the data processing job can include one or more execution modes such as waiting for a preset interval time length to execute. For example, in the initial state, the execution interval of the data processing script can be set to 1 second, that is, the job runs every 1 second, which can be suitable for performance stress testing under a high-frequency scheduling scenario, and can simulate the scheduling delay problem caused by the rapid flow of a large number of tasks. If it is necessary to test the scheduling delay problem caused by the accumulation of a large number of jobs due to resource competition or insufficient processing capacity during the execution process, the execution timing of the next script can be dynamically adjusted to the actual execution time of the data processing job based on the actual execution time of the data processing job, so that the next execution of the script is started after the actual execution of the previous job is completed.

[0086] Further, after adjusting the script execution timing of the data processing script, the operation of executing multiple data processing jobs can be returned, and then the target scheduling delay is determined according to the single scheduling delay of multiple data processing jobs and analyzed. By adjusting the script execution timing, the system performance under different scheduling strategies can be simulated, so as to realize comprehensive testing and evaluation of scheduling delay characteristics under different scenarios.

[0087] S240, determining performance optimization prompt information of the execution system according to the target scheduling delay.

[0088] The technical scheme of the embodiment of the application avoids misjudgment of scheduling performance evaluation caused by job quantity fluctuation or short-time abnormality and the like by setting the first preset time length as an evaluation window, improves the accuracy and stability of scheduling performance evaluation, determines the performance optimization prompt information of the execution system according to the target scheduling delay, realizes quantitative analysis and dynamic optimization guidance of scheduling efficiency, and enables dynamic adjustment of resource scheduling strategy according to the scheduling running state, thereby improving the overall execution efficiency and response capability.

[0089] Embodiment three

[0090] As an optional example of the embodiment of the application, the data processing job scheduling delay test method of the embodiment of the application can include:

[0091] S310, constructing a data processing script with a second-level running time, recording the data processing job name, the actual script execution time, and configuring the specific script execution logic sleep 1 second. The data processing script corresponds to one or more data processing jobs.

[0092] S320, constructing a large number of upstream and downstream dependent data processing jobs, and configuring the corresponding planned execution time to be distributed at multiple time points.

[0093] S330, 60% of the jobs are configured with concurrency degree limitation, and 40% of the jobs are not configured with concurrency degree limitation (configured with MAX concurrency degree).

[0094] S340, executing the data processing jobs, and counting the time information of the jobs without concurrency degree limitation.

[0095] Firstly, the data processing job enters the waiting running queue, and the detection program compares the scheduled time with the current time to obtain the executable job list and the job upstream list that meet the time condition from the database. The execution status of all upstream jobs is detected. If all upstream jobs have been completed, the time identifier and the dependency identifier of the job are updated, and the job is written into the executable queue. Then, the scheduling execution program scans the executable queue, obtains the job from the executable queue according to the concurrency limit recorded in the database, writes the job into the execution queue, updates the concurrency, and sends a job scheduling request. Finally, the execution system accepts the job scheduling request, executes the data processing script, and obtains the execution result.

[0096] For jobs without upstream jobs, the scheduled execution time of the job and the actual execution time of the script are recorded. For jobs with upstream jobs, the end time of the last upstream job, the scheduled execution time of the job, and the actual execution time of the script are recorded.

[0097] S350, for upstream jobs, the single scheduling delay is calculated by the difference between the scheduled execution time of the job and the start execution time of the script. For downstream jobs, it is judged whether the end time of the last upstream job is less than the scheduled execution time of the job. If it is less, the difference between the scheduled execution time of the job and the start execution time of the script is calculated. If it is greater, the difference between the end time of the last upstream job and the actual execution time of the script is calculated to obtain the single scheduling delay.

[0098] S360, the scheduling delay of the first preset number of days is recorded, and the average value is calculated to obtain the target scheduling delay and analyze the test results.

[0099] For each data processing job, the average single scheduling delay corresponding to each data processing job is obtained by summing and averaging the single scheduling delay of each data processing job within the preset number of days. Further, the average single scheduling delay corresponding to each data processing job is summed and averaged to obtain the first average scheduling delay, which is determined as the target scheduling delay.

[0100] In this scenario, the job running time is short, and the internal flow in the scheduling system is fast, so it is not easy to occur job accumulation phenomenon. At this time, if the test scheduling delay is high, the performance optimization point is identified as the insufficient efficiency of the dependency condition query update and the concurrency query update. For the dependency condition query update, slow SQL optimization, read-write separation, database and table separation, using a graph database to store the dependency relationship, multi-threaded query update, and other optimization methods can be used. For the insufficient concurrency query update rate, the cache mechanism can be introduced, and the transaction can be split to reduce the concurrency update lock conflict.

[0101] S370, modify the data processing script to the actual running time of the scene, record the data processing job name, script actual execution time, and modify the script execution specific logic sleep script actual execution time, to test the scheduling scene of the actual running time.

[0102] S380, re-execute S340 to S360 steps, obtain the target scheduling delay of the average value calculated from the scheduling delay of the second preset number of days, and analyze the test results; the second preset number of days includes the first preset number of days.

[0103] In this scenario, the data processing job running time is long, and the job accumulation phenomenon is easy to occur, the waiting running queue is large, at this time, the test scheduling delay is high, and the performance optimization point is identified as the long time of scanning the waiting running queue to obtain executable jobs, etc., the executable job obtaining mode can be optimized, the general sequential scanning queue mode is changed to the time-to-position, upstream job completion and other event triggering modes, and the full traversal of the queue and the redundant scanning of the jobs not meeting the dependency condition are reduced.

[0104] The technical scheme of the embodiment of the application first constructs different types of data processing scripts to test different scenes, and configures the dependency relationship and concurrency degree limit of the data processing job, which can more comprehensively cover various business requirements and abnormal situations, and improve the stability and adaptability of the scheduling test; then, the data processing job is executed, the scheduling delay of the independent job is calculated according to the job plan execution time and the script start execution time, the scheduling punctuality of the independent job can be accurately evaluated, which helps to find the scheduling delay problem, the scheduling delay of the dependent job is calculated according to the end time of the last upstream job, the job plan execution time and the script actual execution time, the upstream completion time and the plan execution time are combined, which can more accurately identify the delay source of the dependent job, and helps to locate the problem link of the scheduling delay; finally, the performance optimization point is determined by statistical analysis of the delay of multiple days, and optimization suggestions are provided, which provides an optimization direction for the system scheduling performance.

[0105] Embodiment four

[0106] Figure 4 A structural schematic diagram of a data processing job scheduling delay test device provided by the fourth embodiment of the application. The device is used to execute the data processing job scheduling delay test method provided by any of the above embodiments. The device and the data processing job scheduling delay test method of each of the above embodiments belong to the same inventive concept, and the details not described in detail in the embodiment of the data processing job scheduling delay test device can be referred to the embodiment of the data processing job scheduling delay test method. As shown in the figure, the device comprises a data processing job execution module 410, a single scheduling delay determination module 420 and a performance optimization prompting module 430. Figure 4 As shown in the figure, the device comprises a data processing job execution module 410, a single scheduling delay determination module 420 and a performance optimization prompting module 430.

[0107] The data processing operation execution module 410 is configured to execute a plurality of data processing operations, wherein some of the data processing operations are provided with operation concurrency degree limits, and some of the data processing operations have upstream and downstream dependency relationships; the operation concurrency degree limit is less than the maximum concurrency degree limit of an execution system of the data processing operations; the single scheduling delay determination module 420 is configured to record a script actual execution time of a data processing script of the data processing operation which is not provided with the operation concurrency degree limit, and determine a single scheduling delay of the data processing operation according to upstream and downstream information of the data processing operation, a scheduled execution time of the operation and the script actual execution time; the performance optimization prompt module 430 is configured to determine a target scheduling delay according to the single scheduling delay and a total operation execution time length of a plurality of the data processing operations, and determine performance optimization prompt information of the execution system according to the target scheduling delay; wherein the performance optimization prompt information comprises a performance item to be optimized and / or a performance optimization mode.

[0108] The technical scheme of the embodiment of the application first executes multiple data processing jobs through a data processing job execution module 410; wherein, part of the data processing jobs are provided with a job concurrency degree limit; there is an upstream and downstream dependency relationship between part of the data processing jobs; the job concurrency degree limit is less than the maximum concurrency degree limit of the execution system of the data processing jobs; the concurrency degree limit and the upstream and downstream dependency relationship can simulate a complex scheduling environment in a real business scenario, improve the representativeness of the test environment, and control the concurrency degree below the maximum limit of the execution system, so as to evaluate the response capability and stability of the scheduling performance under the condition of limited resources within a controllable range; then, a single scheduling delay determination module 420 records the script actual execution time of the data processing script of the data processing job which is not provided with a job concurrency degree limit, determines the single scheduling delay of the data processing job according to the upstream and downstream information of the data processing job, the job plan execution time and the script actual execution time; by collecting the deviation between the actual execution time and the planned execution time, the scheduling delay is accurately identified, and the quantifiability and analyzability of the test result are enhanced; in combination with the upstream and downstream dependency information, the non-planned delay caused by dependency blockage or resource waiting can be effectively identified, and the description precision of the scheduling behavior is improved; finally, a performance optimization prompt module 430 determines a target scheduling delay according to the single scheduling delay and the total job execution time of multiple data processing jobs, and determines performance optimization prompt information of the execution system according to the target scheduling delay; wherein, the performance optimization prompt information includes a performance item to be optimized and / or a performance optimization mode; by summarizing and analyzing multiple job scheduling behaviors, the target scheduling delay is determined, the scheduling bottleneck at the system level is identified, the statistical significance and guiding value of the test result are improved, the performance optimization prompt information with guiding value is generated, an explicit direction and basis are provided for system optimization, the scientificity, accuracy and practicality of the scheduling delay test are improved, and strong support is provided for performance evaluation and optimization of the execution system.

[0109] On the basis of the above scheme, optionally, the single scheduling delay determination module 420 comprises a no-upstream single scheduling delay determination submodule. Wherein, the no-upstream single scheduling delay determination submodule is configured to determine the single scheduling delay of the data processing job according to the job plan execution time of the first data processing job and the script actual execution time of the data processing script thereof, for the first data processing job without upstream data processing job in the multiple data processing jobs.

[0110] On the basis of the above scheme, optionally, the single scheduling delay determination module 420 further comprises an execution end time determination submodule, a first upstream single scheduling delay determination submodule, and a second upstream single scheduling delay determination submodule. The execution end time determination submodule is configured to determine, for a second data processing job having an upstream data processing job among the multiple data processing jobs, an execution end time of a nearest upstream data processing job of the second data processing job. The first upstream single scheduling delay determination submodule is configured to determine, in a case where the execution end time is earlier than a job plan execution time of the second data processing job, a single scheduling delay of the data processing job according to the job plan execution time of the second data processing job and a script actual execution time of a data processing script of the second data processing job. The second upstream single scheduling delay determination submodule is configured to determine, in a case where the execution end time is later than the job plan execution time of the second data processing job, the single scheduling delay of the data processing job according to the execution end time and the script actual execution time of the data processing script of the second data processing job.

[0111] On the basis of the above scheme, optionally, the script execution timing of the data processing script of the data processing job comprises waiting for a preset interval length to execute.

[0112] On the basis of the above scheme, optionally, the performance optimization prompt module 430 comprises a target scheduling delay determination submodule. The target scheduling delay determination submodule is configured to, after a total job execution length of the multiple data processing jobs reaches a first preset length, determine a first average scheduling delay of the single scheduling delay of the multiple data processing jobs within the first preset length, and determine the first average scheduling delay as a target scheduling delay.

[0113] On the basis of the above scheme, optionally, the performance optimization prompt module 430 further comprises an adjustment submodule. The adjustment submodule is configured to, after an execution time of the data processing job reaches a first preset length, adjust a script execution timing of a data processing script of the data processing job according to the data processing script, and return to perform the operation of executing the multiple data processing jobs.

[0114] On the basis of the above scheme, optionally, the apparatus further comprises a data processing script configuration module. The data processing script configuration module is configured to, before the multiple data processing jobs are executed, construct data processing scripts of multiple data processing jobs having an upstream and downstream dependency relationship, configure a script execution logic of the data processing script of each data processing job, and set job concurrency degree limit information for part of the data processing jobs. The script execution logic at least comprises a script execution timing.

[0115] On the basis of the above scheme, optionally, the performance item to be optimized includes a dependent condition query update; and the performance optimization mode corresponding to the performance item to be optimized includes at least one of the following: optimizing a database query statement with a long execution time, read-write separation, database and table separation, using a graph database to store a dependency relationship, and multi-threaded query update.

[0116] On the basis of the above scheme, optionally, the performance item to be optimized includes a concurrent degree query update efficiency; and the performance optimization mode corresponding to the performance item to be optimized includes introducing a cache mechanism and / or transaction splitting.

[0117] On the basis of the above scheme, optionally, the performance item to be optimized includes a manner of obtaining an executable job; and the performance optimization mode corresponding to the performance item to be optimized includes adjusting a sequential scanning queue manner to a manner of triggering scanning through a preset event; and the preset event includes time reaching a position and / or the data processing job of an upstream being completed.

[0118] The data processing job scheduling delay test device provided in the embodiments can execute the data processing job scheduling delay test method provided in any of the embodiments, and has the corresponding function modules and beneficial effects of the execution method.

[0119] Embodiment Five

[0120] Figure 5 A structural diagram of an electronic device 10 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0121] As Figure 5As shown, the electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., communicatively connected to the at least one processor 11, where the memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer programs stored in the read-only memory (ROM) 12 or loaded into the random access memory (RAM) 13 from the storage unit 18. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0122] Various components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc., an output unit 17, such as various types of displays, a speaker, etc., a storage unit 18, such as a magnetic disk, an optical disk, etc., and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0123] The processor 11 can be various general and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the data job scheduling latency testing method.

[0124] In some embodiments, the data job scheduling latency testing method can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the data job scheduling latency testing method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the data job scheduling latency testing method by any other appropriate means, such as by means of firmware.

[0125] The various embodiments of the systems and techniques described above can be implemented in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a load programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0126] Computer programs used to implement the processes of the application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer program, when executed, can cause instructions defined in the flow charts and / or block diagrams to be implemented. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package and partially on a remote machine or entirely on a remote machine or server.

[0127] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store computer programs for use by or in connection with an instruction execution system, apparatus, or device. Computer-readable storage media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0128] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0129] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0130] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0131] In particular, the processes described above with reference to the flowcharts can be implemented as computer software programs in accordance with embodiments of the application. For example, embodiments of the application include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program comprising program code for executing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication unit 19, or installed from the storage unit 18, or installed from the ROM 12. When the computer program is executed by the processor 11, the above-described functions defined in the methods of embodiments of the application are performed.

[0132] It should be understood that the various forms of flow shown above can be used to reorder, add, or remove steps. For example, the steps recited in the present application can be performed in parallel, in series, or in a different order, as long as the desired results of the technical solutions of the present application can be achieved, which are not limited herein.

[0133] The above detailed description does not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for testing the delay of data processing job scheduling, characterized in that, The method comprises the following steps: performing a plurality of data processing operations; wherein part of the data processing operations is provided with a job concurrency limit; there is an upstream and downstream dependency relationship between part of the data processing operations; the job concurrency limit is less than the maximum concurrency limit of the execution system of the data processing operation; recording the script actual execution time of the data processing script of the data processing operation which is not provided with a job concurrency limit, determining the single scheduling delay of the data processing operation according to the upstream and downstream information of the data processing operation, the job plan execution time and the script actual execution time of the data processing script; determining the target scheduling delay according to the single scheduling delay and the total job execution time of a plurality of data processing operations, and determining the performance optimization prompt information of the execution system according to the target scheduling delay; wherein the performance optimization prompt information comprises the performance item to be optimized and / or the performance optimization mode.

2. The data cryptographic job schedule latency testing method of claim 1, wherein, The method comprises the following steps: for a first data processing operation without upstream data processing operation in a plurality of data processing operations, determining the single scheduling delay of the data processing operation according to the job plan execution time of the first data processing operation and the script actual execution time of the data processing script thereof.

3. The data cryptographic job schedule latency testing method of claim 1, wherein, The method comprises the following steps: for a second data processing operation with upstream data processing operation in a plurality of data processing operations, determining the execution end time of the nearest upstream data processing operation of the second data processing operation; in the case that the execution end time is earlier than the job plan execution time of the second data processing operation, determining the single scheduling delay of the data processing operation according to the job plan execution time of the second data processing operation and the script actual execution time of the data processing script thereof; in the case that the execution end time is later than the job plan execution time of the second data processing operation, determining the single scheduling delay of the data processing operation according to the execution end time and the script actual execution time of the data processing script thereof.

4. The data cryptographic job schedule delay test method of claim 1, wherein, The script execution time of the data processing script of the data processing operation comprises waiting for a preset interval time; the method comprises the following steps: after the total job execution time of a plurality of data processing operations reaches a first preset time, determining the first average scheduling delay of the single scheduling delay of a plurality of data processing operations within the first preset time, and determining the first average scheduling delay as the target scheduling delay.

5. The data cryptographic job schedule latency testing method of claim 4, wherein, The method further comprises the following steps: after the execution time of the data processing operation reaches a first preset time, adjusting the script execution time of the data processing script according to the data processing script of the data processing operation, and returning to perform the operation of performing a plurality of data processing operations.

6. The data cryptographic job schedule delay test method of claim 1, wherein, Before the performing of the plurality of data processing operations, further comprising: constructing a data processing script of a plurality of data processing operations with an upstream and downstream dependency relationship, configuring a script execution logic of the data processing script of each of the data processing operations, and setting a job concurrency degree limit information for part of the data processing operations; wherein the script execution logic at least includes a script execution timing.

7. The data cryptographic job schedule latency testing method of claim 1, wherein, The performance item to be optimized includes a dependent conditional query update; and the performance optimization mode corresponding to the performance item to be optimized includes at least one of the following: optimizing a database query statement with a long execution time, read-write separation, database and table separation, using a graph database to store a dependency relationship, and multi-thread query update.

8. The data cryptographic job schedule latency testing method of claim 1, wherein, The performance item to be optimized includes a concurrency degree query update efficiency; and the performance optimization mode corresponding to the performance item to be optimized includes introducing a cache mechanism and / or transaction splitting.

9. The data cryptographic operation scheduling delay test method of claim 1, wherein, The performance item to be optimized includes a way of obtaining executable jobs; and the performance optimization mode corresponding to the performance item to be optimized includes adjusting a sequential scanning queue to a scanning mode triggered by a preset event; and the preset event includes time being in place and / or the data processing operation of the upstream being completed.

10. A computer program product comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the data processing operation scheduling delay test method of any one of claims 1-9.

Images