Time-consuming statistics method and device

Abstract

The disclosure provides a time-consuming statistical method, device, equipment, storage medium and computer program product, relates to the technical field of artificial intelligence, in particular to the technical field of deep learning, and can be applied to scenes such as time-consuming statistics. The specific implementation scheme is: based on the task dependency relationship, a plurality of data tasks on the same task link are determined; external preparation time consumption generated by executing the plurality of data tasks is obtained; based on a plurality of resource listeners pre-registered in the plurality of data tasks, a plurality of groups of resource time consumption and running time consumption generated by executing the plurality of data tasks are obtained; the external preparation time consumption, the plurality of groups of resource time consumption and the running time consumption are concatenated according to the task dependency relationship to obtain a time consumption result. The resource time consumption and the running time consumption are obtained based on the resource listener, and the time consumption result is obtained based on the task dependency relationship, thereby improving the accuracy of time consumption statistics.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of artificial intelligence, in particular to the field of deep learning, which can be applied to time-consuming statistics and the like, and more particularly to a time-consuming statistics method, device, equipment, storage medium and computer program product. BACKGROUND

[0002] At present, when the time consumption of data jobs of big data is counted, the total time consumption of data jobs is usually monitored, or single data time consumption monitoring is adopted in a pull mode, and each upstream time consumption information on the job link cannot be monitored, and the whole-link time consumption data cannot be analyzed. SUMMARY

[0003] The present disclosure provides a time-consuming statistics method, device, equipment, storage medium and computer program product, which improves the accuracy of time-consuming statistics.

[0004] According to an aspect of the present disclosure, a time-consuming statistics method is provided, comprising: determining a plurality of data tasks on a same task link based on a task dependency relationship; obtaining external preparation time consumption generated by executing the plurality of data tasks; obtaining a plurality of groups of resource time consumption and running time consumption generated by executing the plurality of data tasks based on a plurality of resource listeners pre-registered in the plurality of data tasks; and concatenating the external preparation time consumption, the plurality of groups of resource time consumption and the running time consumption according to the task dependency relationship to obtain a time consumption result.

[0005] According to another aspect of the present disclosure, a time-consuming statistics device is provided, comprising: a determination module configured to determine a plurality of data tasks on a same task link based on a task dependency relationship; a first obtaining module configured to obtain external preparation time consumption generated by executing the plurality of data tasks; a second obtaining module configured to obtain a plurality of groups of resource time consumption and running time consumption generated by executing the plurality of data tasks based on a plurality of resource listeners pre-registered in the plurality of data tasks; and a concatenation module configured to concatenate the external preparation time consumption, the plurality of groups of resource time consumption and the running time consumption according to the task dependency relationship to obtain a time consumption result.

[0006] According to still another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the above-mentioned time-consuming statistics method.

[0007] According to still another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the above-mentioned time-consuming statistics method.

[0008] According to still another aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the time consumption statistical method described above.

[0009] It should be understood that the contents described in this part are not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0010] The accompanying drawings are used to better understand the present scheme, and do not constitute a limitation on the present disclosure. Among them:

[0011] Figure 1 is an exemplary system architecture diagram to which the time consumption statistical method or the time consumption statistical device according to the present disclosure can be applied;

[0012] Figure 2 is a flowchart of one embodiment of the time consumption statistical method according to the present disclosure;

[0013] Figure 3 is a flowchart of another embodiment of the time consumption statistical method according to the present disclosure;

[0014] Fig. 4(a) and Fig. 4(b) are one schematic diagram of the time consumption statistical method according to the present disclosure;

[0015] Fig. 5(a) and Fig. 5(b) are another schematic diagram of the time consumption statistical method according to the present disclosure;

[0016] Figure 6 is a structural schematic diagram of one embodiment of the time consumption statistical device according to the present disclosure;

[0017] Figure 7 is a block diagram of an electronic device for implementing the time consumption statistical method according to the embodiments of the present disclosure. DETAILED DESCRIPTION

[0018] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to assist in understanding, and should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, in order to be clear and concise, descriptions of well-known functions and structures are omitted in the following description.

[0019] Figure 1 An exemplary system architecture 100 to which the embodiments of the time consumption statistical method or the time consumption statistical device according to the present disclosure can be applied is shown.

[0020] As Figure 1As shown, the system architecture 100 can include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is a medium to provide communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0021] A user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to obtain time-consuming results, etc. Various client applications, such as resource monitoring applications, etc. can be installed on the terminal devices 101, 102, 103.

[0022] The terminal devices 101, 102, 103 can be hardware or software. When the terminal devices 101, 102, 103 are hardware, they can be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc. When the terminal devices 101, 102, 103 are software, they can be installed in the above-mentioned electronic devices. They can be implemented as multiple software or software modules, or as a single software or software module. No specific limitation is made herein.

[0023] The server 105 can provide various time-consuming statistical services. For example, the server 105 can analyze and process multiple data tasks obtained from the terminal devices 101, 102, 103, and generate processing results (such as time-consuming results, etc.).

[0024] It should be noted that the server 105 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When the server 105 is software, it can be implemented as multiple software or software modules (such as to provide distributed services), or as a single software or software module. No specific limitation is made herein.

[0025] It should be noted that the time-consuming statistical method provided by the embodiments of the present disclosure is generally executed by the server 105, and accordingly, the time-consuming statistical apparatus is generally provided in the server 105.

[0026] It should be understood that Figure 1 The number of terminal devices, networks, and servers in the system architecture 100 is merely illustrative. According to the implementation needs, there can be any number of terminal devices, networks, and servers.

[0027] Continuing to refer to Figure 2 which shows a flow 200 of one embodiment of the time-consuming statistical method according to the present disclosure. The time-consuming statistical method includes the following steps:

[0028] Step 201, based on the task dependency relationship, determine multiple data tasks on the same task link.

[0029] In this embodiment, the execution subject of the time-consuming statistical method (for example Figure 1 The server 105 shown in the figure) can determine multiple data tasks on the same task link based on the task dependency relationship. Wherein, the task dependency relationship refers to the link relationship of data generation, and the multiple data tasks on the same task link refer to multiple data tasks with link relationship, and the multiple data tasks depend on each other, for example, the first data task A is processed to obtain the processing result A1, the data task B is processed based on A1 to obtain the processing result B1, the data task C is processed based on B1 to obtain the processing result C1, the data task A, the data task B and the data task C exist between them The task dependency relationship belongs to three data tasks on the same task link. Specifically, the data processing target can be determined first, and the first data task required to achieve the data processing target is determined in the data task pool, the data task with link relationship as the second data task on the same task link is obtained from the data task pool. Get all data tasks with link relationship from the data task pool as multiple data tasks on the same task link.

[0030] Step 202, obtain the external preparation time-consuming generated by executing multiple data tasks.

[0031] In this embodiment, after the above-mentioned execution subject obtains multiple data tasks, it can obtain the external preparation time-consuming generated by executing multiple data tasks. Wherein, the external preparation time-consuming refers to the time length from the beginning of the task to the readiness of the external data, and the external data can be the input data required to execute multiple data tasks. Specifically, after detecting the start of data task execution, the timing operator and the external data preparation operator can be executed at the same time. When the external data is ready, the timing operator stops timing, and the timing duration is output from the timing operator as the external preparation time-consuming.

[0032] Step 203, based on multiple resource listeners pre-registered in multiple data tasks, obtain multiple groups of resource time-consuming and running time-consuming generated by executing multiple data tasks.

[0033] In the embodiment, the execution subject can further acquire a plurality of sets of resource consumption time and running consumption time after acquiring the external preparation consumption time. The resource listener is a listener for monitoring the number of executioners applied for during the execution of a data task, and a resource listener can be registered in each data task in advance. Specifically, when a plurality of data tasks are executed in sequence according to the task dependency relationship, when it is detected that a data task starts to execute, the resource listener in the data task is activated, the resource listener starts to monitor the number of executioners applied for by the data task, and records the time when the monitoring starts. When it is detected that the number of executioners applied for has reached the number of executioners required by the data task, the application is stopped, and the time when the application is stopped is recorded, and the data calculation of the data task is started. When it is detected that the data task stops calculating, the time when the calculation stops is recorded, and the resource listener is closed. The time difference between the stop application time of the data task and the start monitoring time is the resource consumption time of the data task, and the time difference between the stop calculation time of the data task and the stop application time is the running consumption time of the data task. The resource consumption time and the running consumption time of the same data task can be regarded as a set of resource consumption time and running consumption time, and a plurality of sets of resource consumption time and running consumption time generated by executing a plurality of data tasks can be acquired in this way.

[0034] In step 204, the external preparation consumption time, a plurality of sets of resource consumption time and running consumption time are concatenated according to the task dependency relationship to obtain a consumption time result.

[0035] In the embodiment, the execution subject can further obtain a consumption time result after acquiring the external preparation consumption time, a plurality of sets of resource consumption time and running consumption time. Specifically, the external preparation consumption time can be regarded as a concatenation unit, and a set of resource consumption time and running consumption time can be regarded as a concatenation unit. Based on the task dependency relationship between the data tasks generating the external preparation consumption time or a set of resource consumption time and running consumption time, the respective concatenation units are concatenated to obtain a consumption time result. For example, based on the task dependency relationship between the data tasks, the external preparation consumption time can be regarded as a first concatenation unit, a set of resource consumption time and running consumption time generated by data task A can be regarded as a second concatenation unit, a set of resource consumption time and running consumption time generated by data task C can be regarded as a third concatenation unit, a set of resource consumption time and running consumption time generated by data task B can be regarded as a fourth concatenation unit, and the concatenation result can be determined as the consumption time result.

[0036] The time consumption statistical method provided in the embodiments of the present disclosure first determines a plurality of data tasks on a same task link based on a task dependency relationship, then acquires external preparation time consumption generated by executing the plurality of data tasks, then acquires a plurality of groups of resource time consumption and running time consumption generated by executing the plurality of data tasks based on a plurality of resource listeners pre-registered in the plurality of data tasks, and finally concatenates the external preparation time consumption, the plurality of groups of resource time consumption and the running time consumption according to the task dependency relationship to obtain a time consumption result. Based on the above method, the obtained external preparation time consumption, resource time consumption and running time consumption are more accurate, so that the obtained time consumption result is more accurate, and the obtained external preparation time consumption, plurality of groups of resource time consumption and running time consumption are concatenated, which facilitates phased time consumption analysis and provides data support for time efficiency optimization of data tasks.

[0037] Further continuing to refer to Figure 3 which shows a flow 300 of another embodiment of the time consumption statistical method according to the present disclosure. The time consumption statistical method comprises the following steps:

[0038] Step 301, determining a plurality of data tasks on a same task link based on a task dependency relationship.

[0039] In the present embodiment, the specific operation of step 301 has been described in detail in the embodiment shown in Figure 2 The step 201 in the embodiment shown in has been described in detail and will not be repeated here.

[0040] Step 302, determining a first data task on the same task link based on the task dependency relationship.

[0041] In the present embodiment, the above execution subject can determine the first data task on the same task link based on the task dependency relationship. Wherein, the first data task is the first executed data task on the same task link. Specifically, the plurality of data tasks on the same task link can be analyzed, and the data task that generates data results used by other data tasks but does not use data results generated by other data tasks is found out from the plurality of data tasks as the first data task.

[0042] Step 303, in response to executing a dependency check operator in the first data task, executing a first callback operator pre-added in the first data task to obtain a start time.

[0043] In the embodiment, the execution subject can obtain the start time based on the first callback operator in the first task data. Specifically, the first task data can include the dependency check operator and the calculation operator. In order to record the required time points, the first callback operator can be added before the dependency check operator, the second callback operator can be added after the dependency check operator, and the third callback operator can be added after the calculation operator in the form of an aspect. The aspect is to find three division points in the first data task in the form of aspect-oriented programming, and add a callback operator at each division point. The first callback operator, the second callback operator, and the third callback operator are all operators for requesting to record time. When the program runs to the position of any callback operator, the callback operator can call the time-consuming monitoring service in the form of an application programming interface. The time-consuming monitoring service can record the current time point. The time-consuming monitoring service is a time recording service independent of any data task. Through the aspect type task scheduling callback operator, the normal execution of the data task is not affected, and the time points of the data task can be accurately and real-timely obtained.

[0044] The operator execution order of the first data task is the first callback operator, the dependency check operator, the second callback operator, the calculation operator, and the third callback operator. When it is detected that the first data task is executed, and before the dependency check operator of the first data task is executed, the first callback operator is executed. The first callback operator can call the time-consuming monitoring service in the form of an application programming interface. The time-consuming monitoring service can record the current time point as the start time.

[0045] In step 304, the time difference between the dependency check completion time and the start time of the first data task is determined as the external preparation time consumption.

[0046] In the embodiment, the execution subject can further obtain the external preparation time consumption after obtaining the start time. Specifically, after the first data task obtains the start time, the first callback operator is executed, and the dependency check operator is executed. The dependency check operator is used to check whether the upstream data is ready. In the first data task, the dependency check operator is used to check whether the external data is ready. After the dependency check operator is executed, the second callback operator is executed. The second callback operator can call the time-consuming monitoring service in the form of an application programming interface. The time-consuming monitoring service can record the current time point as the dependency check completion time. The time difference between the dependency check completion time and the start time of the first data task is calculated, and the obtained time difference is determined as the external preparation time consumption.

[0047] In step 305, the resource ready time of each of the plurality of data tasks is obtained based on the plurality of resource listeners pre-registered in the plurality of data tasks.

[0048] In the embodiment, the execution subject can obtain the resource ready time of each data task. Specifically, a resource listener can be registered in each data task in advance. When each data task is executed, the first callback operator, the dependency check operator, the second callback operator, the calculation operator, and the third callback operator are sequentially executed. The resource listener is located in the calculation operator of each data task. When a data task starts to be executed, the first callback operator, the dependency check operator, and the second callback operator are sequentially executed. When it is detected that the calculation operator starts to be executed, the resource listener in the calculation operator is activated. The resource listener starts to monitor the number of executors applied by the data task. When it is monitored that the number of applied executors reaches the number of executors required by the data task, the resource listener records the time as the resource ready time of the data task. In this way, the resource ready time of each data task is obtained.

[0049] In some optional implementations of the embodiment, the following resource ready time determination step can be performed on any one of the plurality of data tasks: in response to starting to execute the calculation operator in the data task and detecting the newly added executors, generating an application event; sending the application event to the resource listener; monitoring the number of accumulated applied executors by the resource listener; and in response to monitoring that the number of accumulated applied executors is greater than or equal to a preset threshold, recording the resource ready time.

[0050] Specifically, for any one data task, after the first callback operator, the dependency check operator and the second callback operator are executed in turn, when it is detected that the calculation operator starts to be executed, the executor starts to be applied for execution, so as to perform calculation. When it is detected that the new executor is added, an application event can be generated, which is used to notify how many executors are added. The application event can be sent to the listener bus in advance, wherein the listener bus is a listener container that can accommodate multiple different types of listeners. After receiving the application event, the listener bus can broadcast the application event to each listener in the listener bus. The pre-registered resource listener responds to the application event, receives and processes the application event. The resource listener can read the information about how many executors are added in the application event, and then add the number of newly added executors to the stored cumulative number of applied executors to obtain an updated cumulative number of applied executors. When the updated cumulative number of applied executors is greater than or equal to a preset threshold, for example, the preset threshold is 30% of the number of executors required by the data task, for example, 20, the time at this moment is recorded as the resource readiness time of the data task, so as to obtain the resource readiness time of each of the multiple data tasks. It should be noted that the running mechanism of any one data task does not need to apply for enough executors to start execution. Therefore, the time when the preset threshold number of executors are in place can be regarded as the resource readiness time, wherein the preset threshold number of executors can be 30% of the total number of executors required by a data task, or 50%, 70%, or 90% of the total number of executors.

[0051] Step 306, obtaining the dependency check completion time and the end time of each of the multiple data tasks.

[0052] In the embodiment, the execution subject can further obtain the dependency check completion time and the end time of each of the multiple data tasks. The dependency check completion time is the time when the upstream task is ready, which is equivalent to the time when the dependency check operator of the data task to be executed is executed. Specifically, a time recording program can be added in the dependency check operator and the calculation operator of each task in advance. When the execution of the dependency check operator is completed, the time recording program is triggered to record the time at this moment as the dependency check completion time of the task, and the calculation operator continues to be executed. When the execution of the calculation operator is completed, the time recording program is triggered to record the time at this moment as the end time of the task, so as to obtain the dependency check completion time and the end time of each of the multiple data tasks.

[0053] In some optional implementations of the embodiment, the following dependency check completion time and end time determination steps can be performed on any one of the plurality of data tasks: performing a dependency check operator in the data task, in response to completion of the dependency check operator, performing a second callback operator added in advance in the data task to obtain the dependency check completion time; performing a calculation operator in the data task, in response to completion of the calculation operator, performing a third callback operator added in advance in the data task to obtain the end time.

[0054] Specifically, each data task can sequentially perform the first callback operator, the dependency check operator, the second callback operator, the calculation operator, and the third callback operator when the data task is executed, wherein the first callback operator, the second callback operator, and the third callback operator are operators for recording time added in advance in each data task, and when the program reaches the position of any one of the callback operators, the callback operator can call the time consumption monitoring service in the form of an application programming interface, and the time consumption monitoring service can record the current time point, wherein the time consumption monitoring service is a time recording service independent of any one of the data tasks. When a data task starts to be executed, the first callback operator and the dependency check operator are sequentially executed first, and when it is detected that the dependency check operator is executed, the second callback operator is started to be executed. The second callback operator calls the time consumption monitoring service in the form of an application programming interface, and the time consumption monitoring service records the current time point as the dependency check completion time of the data task, and continues to execute the calculation operator of the data task. When it is detected that the calculation operator is executed, the third callback operator is started to be executed. The third callback operator calls the time consumption monitoring service in the form of an application programming interface, and the time consumption monitoring service records the current time point as the end time of the data task, thereby obtaining the dependency check completion time and the end time of each of the plurality of data tasks.

[0055] In step 307, based on the resource readiness time, the dependency check completion time, and the end time, a plurality of sets of resource consumption time and running consumption time are obtained.

[0056] In the embodiment, after the resource readiness time, the dependency check completion time, and the end time are obtained, the execution subject can further obtain a plurality of sets of resource consumption time and running consumption time. Specifically, the resource readiness time, the dependency check completion time, and the end time of a data task can be input into a consumption time calculation model, and a set of resource consumption time and running consumption time of the data task is output from the output end of the consumption time calculation model, thereby obtaining a plurality of sets of resource consumption time and running consumption time corresponding to the plurality of data tasks.

[0057] In some optional implementations of the embodiment, for any one of the plurality of data tasks, a set of resource time consumption and running time consumption can be determined by the following steps: determining the time difference between the resource ready time of the data task and the dependency check completion time as the resource time consumption of the data task; and determining the time difference between the end time of the data task and the resource ready time as the running time consumption of the data task.

[0058] Specifically, for any one data task, the resource ready time of the data task can be subtracted from the dependency check completion time, and the time length obtained is taken as the resource time consumption of the data task; the end time of the data task can be subtracted from the resource ready time, and the time length obtained is taken as the running time consumption of the data task; the resource time consumption and the running time consumption of the same data task are taken as a set of resource time consumption and running time consumption, so as to obtain a plurality of sets of resource time consumption and running time consumption corresponding to the plurality of data tasks.

[0059] Step 308: taking the external preparation time consumption as the first in series.

[0060] In the embodiment, after obtaining the external preparation time consumption, the plurality of sets of resource time consumption and running time consumption, the execution subject can concatenate the external preparation time consumption, the plurality of sets of resource time consumption and running time consumption. Specifically, the external preparation time consumption is the time required to prepare the external input data required by the data task before calculating any one data task on the same task link, so the external data preparation is the first step of executing the plurality of data tasks, and therefore the external preparation time consumption can be taken as the first in series.

[0061] Step 309: according to the task dependency relationship, concatenating the plurality of sets of resource time consumption and running time consumption in turn after the external preparation time consumption to obtain the time consumption result.

[0062] In the embodiment, the execution subject can further concatenate the resource consumption time and the running consumption time. Specifically, each set of resource consumption time and running consumption time is generated by the same data task, and in a set of resource consumption time and running consumption time, the resource consumption time is generated first, and the running consumption time is generated later. The concatenation order of the multiple sets of resource consumption time and running consumption time can be arranged based on the task dependency relationship between the corresponding multiple data tasks. For example, the multiple data tasks include three data tasks: data task A, data task B, and data task C. The task dependency relationship between the three data tasks is to execute the data task A first, execute the data task C based on the calculation result of the data task A, and execute the data task B based on the calculation result of the data task C. Therefore, the concatenation order of the three sets of resource consumption time and running consumption time generated by the three data tasks is: the resource consumption time generated by the data task A, the running consumption time, the resource consumption time generated by the data task B, the running consumption time, the resource consumption time generated by the data task C, and the running consumption time. The three sets of resource consumption time and running consumption time are concatenated after the external preparation consumption time, and the obtained concatenation result is used as the consumption time result.

[0063] From Figure 3 It can be seen that, compared with the embodiment corresponding to Figure 2 The consumption time statistical method in the embodiment acquires the start time, the dependency check completion time, and the end time of each data task based on the aspect-based callback method, does not affect the normal execution of the data task, can monitor the stage consumption time information of the data task in real time, acquires the resource readiness time by registering the resource listener in the calculation operator of each data task and monitoring the part executor placement time, can accurately collect the resource consumption time information of the data task, provides data support for the time efficiency optimization of the data task, and greatly reduces the cost of the overall time efficiency of the complex data task link in the data management process.

[0064] Further continuing to refer to FIG. 4(a) and FIG. 4(b), which show a schematic diagram 400 of the time-consuming statistics method according to the present disclosure. As can be seen from FIG. 4(a), any one data task includes a first callback operator, a dependency check operator, a second callback operator, a calculation operator, and a third callback operator, and when executing a data task, the first callback operator, the dependency check operator, the second callback operator, the calculation operator, and the third callback operator are executed in sequence, wherein when the first callback operator, the second callback operator, and the third callback operator are executed, the time-consuming monitoring service is called in the form of an application program interface, and the time-consuming monitoring service records and stores the current time point as the start time, the dependency check completion time, and the end time of the data task, and when the dependency check operator and the calculation operator are executed, the dependency check service and the calculation cluster are called to complete, respectively. FIG. 4(b) is a schematic diagram of a resource listener in the calculation operator in any one data task. Before executing the data task, the designed resource listener is first registered to the listener bus in the calculation operator, and when the calculation operator starts to be executed, the resource listener is activated to listen to the number of newly added executors, and when the accumulated number of applied executors reaches a preset threshold, the resource listener can record the resource readiness time of the data task.

[0065] Further continuing to refer to FIG. 5(a) and FIG. 5(b), another schematic diagram 500 of the time-consuming statistical method according to the present disclosure is shown. As can be seen from FIG. 5(a), each data task obtains four time points, namely, a start time, a dependency check completion time, a resource readiness time, and an end time, and each data task obtains three time consumptions based on the four obtained time points, namely, a dependency check time consumption, a resource time consumption, and a running time consumption, wherein the dependency check time consumption is the time consumed for checking the readiness of an upstream task, in the first data task on the same task link, the dependency check time consumption is the time consumed for checking the readiness of external data, therefore, the dependency check time consumption of the first data task is the external preparation time consumption, and the dependency check time consumption of each remaining data task is the time consumed for checking the readiness of all upstream tasks, the resource time consumption of each data task is the time consumed for applying to a preset threshold number of executors, and the running time consumption is the time consumed for executing the calculation task in the calculation operator. As can be seen from FIG. 5(b), after obtaining the dependency check time consumption, the resource time consumption, and the running time consumption of all data tasks, all the obtained time consumptions can be concatenated, the external preparation time consumption is the time consumed for the first step of executing multiple data tasks, therefore, it is concatenated first, the dependency check time consumption of each remaining data task is the same as the total time consumption of the upstream task, therefore, it is no longer repeated in concatenation, and the dependency check time consumption of each remaining data task is ignored, and based on the task dependency relationship between the data tasks, the resource time consumption and the running time consumption of each remaining data task are concatenated after the external preparation time consumption, respectively, to obtain a time consumption result, and based on the concatenated time consumption result, the time consumption of each stage can be intuitively analyzed, thereby providing data support for time efficiency optimization.

[0066] Further referring to Figure 6 , as an implementation of the time-consuming statistical method, the present disclosure provides an embodiment of a time-consuming statistical device, which corresponds to the method embodiment shown in Figure 2 , and the device can be applied in various electronic devices.

[0067] As shown in Figure 6 , the time-consuming statistical device 600 of the present embodiment can include a determination module 601, a first acquisition module 602, a second acquisition module 603, and a concatenation module 604. The determination module 601 is configured to determine multiple data tasks on the same task link based on a task dependency relationship; the first acquisition module 602 is configured to obtain an external preparation time consumption generated by executing the multiple data tasks; the second acquisition module 603 is configured to obtain multiple sets of resource time consumption and running time consumption generated by executing the multiple data tasks based on multiple resource listeners pre-registered in the multiple data tasks; and the concatenation module 604 is configured to concatenate the external preparation time consumption, the multiple sets of resource time consumption, and the running time consumption according to the task dependency relationship to obtain a time consumption result.

[0068] In the embodiment, the time-consuming statistics device 600: the specific process of the determining module 601, the first obtaining module 602, the second obtaining module 603, and the concatenating module 604 and the technical effects brought by the specific process can be referred to the related description of steps 201-204 in the corresponding embodiment respectively. Figure 2 The related description of steps 201-204 in the corresponding embodiment will not be repeated here.

[0069] In some optional implementation manners of the embodiment, the second obtaining module 603 includes: a first obtaining sub-module, configured to obtain the resource readiness time of each of the plurality of data tasks based on a plurality of resource listeners pre-registered in the plurality of data tasks; a second obtaining sub-module, configured to obtain the dependency check completion time and the end time of each of the plurality of data tasks; and a third obtaining sub-module, configured to obtain a plurality of sets of resource time consumption and running time consumption based on the resource readiness time, the dependency check completion time, and the end time.

[0070] In some optional implementation manners of the embodiment, the first obtaining module 602 includes, for any one of the plurality of data tasks: a generating unit, configured to generate an application event in response to starting to execute a computing operator in the data task and detecting an added executor; a sending unit, configured to send the application event to a resource listener; a listening unit, configured to listen to the number of accumulatively applied executors through the resource listener; and a recording unit, configured to record the resource readiness time in response to listening to that the number of accumulatively applied executors is greater than or equal to a preset threshold.

[0071] In some optional implementation manners of the embodiment, the second obtaining sub-module includes, for any one of the plurality of data tasks: a first executing unit, configured to execute a dependency check operator in the data task, and in response to the dependency check operator being executed, execute a second callback operator pre-added in the data task to obtain the dependency check completion time; and a second executing unit, configured to execute a computing operator in the data task, and in response to the computing operator being executed, execute a third callback operator pre-added in the data task to obtain the end time.

[0072] In some optional implementation manners of the embodiment, the third obtaining sub-module includes, for any one of the plurality of data tasks: a first determining unit, configured to determine, as the resource time consumption of the data task, the time difference between the resource readiness time of the data task and the dependency check completion time; and a second determining unit, configured to determine, as the running time consumption of the data task, the time difference between the end time of the data task and the resource readiness time.

[0073] In some optional implementations of the embodiment, the first obtaining module 602 includes: a determination sub-module, configured to determine, based on the task dependency relationship, the first data task on the same task link; a fourth obtaining sub-module, configured to, in response to executing a dependency check operator in the first data task, execute a first callback operator added in the first data task in advance to obtain a start time; and a fifth obtaining sub-module, configured to determine, as the external preparation time consumption, a time difference between a dependency check completion time of the first data task and the start time.

[0074] In some optional implementations of the embodiment, the concatenation module 604 includes: a first concatenation sub-module, configured to concatenate the external preparation time consumption as a concatenation head; and a second concatenation sub-module, configured to concatenate, in sequence, the multiple groups of resource time consumptions and the running time consumptions after the external preparation time consumption according to the task dependency relationship, to obtain the time consumption result.

[0075] According to embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium, and a computer program product.

[0076] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0077] As shown in Figure 7 The device 700 includes a computing unit 701 that can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. Various programs and data required for the operation of the device 700 can also be stored in the RAM 703. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0078] A number of the components in device 700 are connected to I / O interface 705, including: input unit 706, e.g., keyboard, mouse, etc.; output unit 707, e.g., various types of displays, speakers, etc.; storage unit 708, e.g., disk, optical disk, etc.; and communication unit 709, e.g., network card, modem, wireless communication transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices over computer networks, such as the Internet, and / or various telecommunication networks.

[0079] Computing unit 701 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of computing unit 701 include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. Computing unit 701 performs various methods and processes described above, e.g., the time-consuming statistical method. For example, in some embodiments, the time-consuming statistical method can be implemented as a computer software program, which is tangibly embodied in a machine-readable medium, e.g., storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded onto RAM 703 and executed by computing unit 701, one or more steps of the time-consuming statistical method described above can be performed. Alternatively, in other embodiments, computing unit 701 can be configured to perform the time-consuming statistical method by other any appropriate means, e.g., by means of firmware.

[0080] Various implementations of the systems and techniques described above herein can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0081] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package, or entirely on a remote machine or server.

[0082] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0083] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0084] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0085] The computer system can include clients and servers. The clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server can arise by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The servers can be servers of a distributed system, or servers combined with a blockchain. The servers can also be cloud servers, or intelligent cloud computing servers or intelligent cloud hosts with artificial intelligence technology.

[0086] It should be understood that various forms of flow shown above can be used, with steps reordered, added, or removed. For example, the steps recited in the present disclosure can be performed in parallel, in series, or in a different order, without limitation herein, so long as the desired results of the technology disclosed in the present disclosure are achieved.

[0087] The specific embodiments described above do not constitute an exhaustive list of all possible embodiments as can be made within the scope of the present disclosure. It should be understood that various modifications, combinations, sub-combinations, and alternatives can be practiced with the scope of the present disclosure. Any modifications, equivalent replacements, and improvements, etc. made within the spirit and principles of the present disclosure should be included in the scope of the present disclosure.

Claims

1. A time-consuming statistical method, comprising: determining a plurality of data tasks on a same task link based on a task dependency relationship; executing a first callback operator pre-added in a first data task on the task link before executing a dependency check operator in the first data task to obtain a start time; obtaining an external preparation time-consuming generated by executing the plurality of data tasks, the external preparation time-consuming being a time difference between a dependency check completion time of the first data task and the start time; obtaining a plurality of sets of resource time-consumings and running time-consumings generated by executing the plurality of data tasks based on a plurality of resource listeners pre-registered in the plurality of data tasks, wherein for any one of the plurality of data tasks: monitoring an amount of executors applied by the resource listener to the data task, and recording a resource ready time in response to monitoring that the cumulative amount of applied executors is greater than or equal to a preset threshold; determining a time difference between the resource ready time and the dependency check completion time of the data task as the resource time-consuming; determining a time difference between an end time of the data task and the resource ready time as the running time-consuming; concatenating the external preparation time-consuming, the plurality of sets of resource time-consumings and running time-consumings according to the task dependency relationship to obtain a time-consuming result.

2. The method of claim 1, wherein, The determination of the resource ready time further comprises: generating an application event in response to starting to execute a calculation operator in the data task and detecting a newly added executor; sending the application event to the resource listener.

3. The method of claim 1, wherein, The determination of the resource time-consuming and the running time-consuming further comprises: executing a dependency check operator in the data task, and executing a second callback operator pre-added in the data task in response to the execution of the dependency check operator being completed to obtain the dependency check completion time; executing a calculation operator in the data task, and executing a third callback operator pre-added in the data task in response to the execution of the calculation operator being completed to obtain the end time.

4. The method of claim 1, wherein, The concatenating the external preparation time-consuming, the plurality of sets of resource time-consumings and running time-consumings according to the task dependency relationship to obtain a time-consuming result comprises: concatenating the external preparation time-consuming as the first one; concatenating the plurality of sets of resource time-consumings and running time-consumings in turn after the external preparation time-consuming according to the task dependency relationship to obtain the time-consuming result.

5. A time-consuming statistical device, comprising: a determination module configured to determine a plurality of data tasks on a same task link based on a task dependency relationship; a first obtaining module configured to execute a first callback operator pre-added in a first data task on the task link before executing a dependency check operator in the first data task to obtain a start time, and obtain an external preparation time-consuming generated by executing the plurality of data tasks, the external preparation time-consuming being a time difference between a dependency check completion time of the first data task and the start time; The second obtaining module is configured to obtain a plurality of sets of resource consumption time and running consumption time generated by executing the plurality of data tasks based on a plurality of resource listeners pre-registered in the plurality of data tasks, wherein for any one data task in the plurality of data tasks: The resource listener is configured to monitor the number of executors applied for by the data task, and in response to monitoring that the cumulative number of applied executors is greater than or equal to a preset threshold, record a resource readiness time; The time difference between the resource readiness time and the dependency check completion time of the data task is determined as the resource consumption time; The time difference between the end time of the data task and the resource readiness time is determined as the running consumption time; The series connection module is configured to connect the external preparation consumption time, the plurality of sets of resource consumption time and running consumption time in series according to the task dependency relationship to obtain a consumption time result.

6. The apparatus of claim 5, wherein, The second obtaining module For the determination of the resource readiness time, the second obtaining module is further configured to: In response to starting to execute a calculation operator in the data task and detecting that a new executor is added, generate an application event; Send the application event to the resource listener.

7. The apparatus of claim 5, wherein, The second obtaining module For the determination of the resource consumption time and the running consumption time, the second obtaining module is configured to: Execute a dependency check operator in the data task, and in response to the completion of the execution of the dependency check operator, execute a second callback operator pre-added in the data task to obtain the dependency check completion time; Execute a calculation operator in the data task, and in response to the completion of the execution of the calculation operator, execute a third callback operator pre-added in the data task to obtain the end time.

8. The apparatus of claim 5, wherein, The series connection module is further configured to: Take the external preparation consumption time as the first one in series connection; According to the task dependency relationship, sequentially connect the plurality of sets of resource consumption time and running consumption time after the external preparation consumption time to obtain the consumption time result.

9. An electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-4.

10. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method of any one of claims 1-4.

11. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-4.

Images