A data output timeliness processing method and device, a storage medium and an equipment

Abstract

Embodiments of the present application provide a data output timeliness processing method and device, a storage medium and equipment. In the method, the expected output time of a target job is obtained, the information of all jobs on the full link is obtained through data blood relationship layer-by-layer recursion, the upstream and downstream relationships are determined, then based on the historical execution records of each job within a preset time period, the historical average start time and the historical average time consumption are calculated, and the expected output time and the expected start time of each job are automatically configured based on the blood relationship full link in combination with the expected output time of the target job. In this way, the automation configuration of data output timeliness is realized, the time and labor costs are reduced, and the rationality is improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically, to a method, apparatus, storage medium, and device for processing data output timeliness. Background Technology

[0002] Currently, enterprises have increasingly higher requirements for the timeliness of data output. Most enterprises now use monitoring and scheduling platforms to ensure service delivery based on output time, supporting lineage tracing, monitoring configuration, and SMS alerts. However, these tools only provide monitoring services in the entire service delivery process; the baseline timeliness of operations relies on manual configuration, which consumes significant time and manpower and is prone to issues of insufficient configuration adequacy. Summary of the Invention

[0003] The purpose of this application is to provide a data output timeliness processing method, apparatus, storage medium and device, which aims to solve the problem that the configuration scheme for the benchmark timeliness of operations is heavily dependent on manual labor, requires a lot of time and manpower, and has insufficient rationality.

[0004] In a first aspect, an embodiment of this application provides a data output timeliness processing method, comprising:

[0005] Obtain the expected output time of the target job from the business side, and obtain information on all jobs in the entire chain of the target job through recursive data lineage;

[0006] Based on the historical execution records of each job within a preset time period, determine the historical average start time and historical average execution time of each job.

[0007] Based on the expected output time of the target task, the historical average start time and historical average consumption time of each task, the expected output time and expected start time of each task are configured. The expected output time of the current layer task is calculated based on the historical average start time of all first-level downstream tasks of the current layer, and the expected start time of the current layer task is calculated based on the expected output time of the current layer task and the historical average consumption time.

[0008] In the above implementation process, the expected output time of the target job is collected from the business. Through recursive data lineage, information on all jobs across the entire chain is obtained to determine upstream and downstream relationships. Then, based on the historical execution records of each job within a preset time period, the historical average start time and historical average execution time are calculated. Combined with the expected output time of the target job, the expected output time and expected start time of each job are automatically configured based on the entire lineage. This achieves automated configuration of data output timeliness, reducing time and labor costs while improving rationality.

[0009] Furthermore, in some embodiments, the historical average time consumption is the value that is in the 95th percentile after arranging the historical time consumption of the job in ascending order within a preset time period.

[0010] In the above implementation process, taking the time spent on P95 for each job in the past month as the historical average time can determine the accurate and effective expected start time, which is conducive to optimizing the management of data output timeliness.

[0011] Furthermore, in some embodiments, the target job is the job corresponding to core data, which is obtained in the following manner:

[0012] The data assets are scored based on the weights corresponding to the target influencing factors; the target influencing factors include the number of user departments, the number of users accessing the platform, and the number of visits by management.

[0013] Based on the scores, the data assets are divided into core data, important data, and general data.

[0014] In the above implementation process, by analyzing the lineage of core data assets, the lineage of application assets, application library, and big data platform can be connected, i.e., the end-to-end lineage, so as to carry out differentiated and refined time-sensitive management.

[0015] Furthermore, in some embodiments, the method includes:

[0016] Monitor the actual output time of each task; if the actual output time is later than the corresponding expected output time, determine that the task has been delayed.

[0017] When a delay occurs in the task corresponding to the core data, a problem registration list is generated and sent to the person responsible for the task.

[0018] In the above implementation process, when core data is delayed, the system can automatically generate a problem registration list for the job development owner, so as to automatically assess the impact of the delay, follow up on the resolution of timeliness issues, register the causes of the problems, and manage timeliness issues.

[0019] Furthermore, in some embodiments, the method further includes:

[0020] When any task is delayed, determine whether the task is in an error state or meets a first condition; the first condition is that the difference between the actual time consumed by the task and the historical average time consumed is greater than the difference between the actual output time and the historical average end time.

[0021] If so, determine that the delay is an anomaly caused by the job; otherwise, determine that the delay is an anomaly caused by the target upstream job; the target upstream job is the upstream job with the latest actual output time.

[0022] If the target upstream job is the top-level job in the entire chain, and the job meets the second condition, it is determined that the delay is an anomaly caused by the platform; the second condition is that the difference between the actual output time of the job and the historical average end time is less than a preset value.

[0023] In the above implementation process, an automatic anomaly location scheme is provided, which can automatically locate anomalies and provide data support for optimized management.

[0024] Furthermore, in some embodiments, the method further includes:

[0025] When a delay is detected in the upstream operation of the target operation, it is determined whether the upstream operation affects the output of the target operation. If so, an early warning is output.

[0026] In the above implementation process, intelligent prediction of the impact of delays on target operations is achieved through reverse deduction of automatic abnormal problem location.

[0027] Furthermore, in some embodiments, the method further includes:

[0028] Count the number of delays and errors for each job, and determine the jobs to be optimized based on the statistical results;

[0029] The statistical results of the task to be optimized are pushed to the corresponding person in charge so that the person in charge can optimize the task.

[0030] In the above process, the system automatically identifies jobs that need to be optimized and then pushes them to the data manager for evaluation and optimization, thus achieving timeliness optimization management.

[0031] Secondly, an embodiment of this application provides a data output timeliness processing device, comprising:

[0032] The acquisition module is used to obtain the expected output time of the target job from the business side, and to obtain the information of all jobs in the entire chain of the target job through data lineage recursion.

[0033] The determination module is used to determine the historical average start time and historical average execution time of each job based on the historical execution records of each job within a preset time period.

[0034] The configuration module is used to configure the expected output time and expected start time of each job based on the expected output time of the target job, the historical average start time and historical average consumption time of each job. The expected output time of the current layer job is calculated based on the historical average start time of all first-level downstream jobs of the current layer, and the expected start time of the current layer job is calculated based on the expected output time of the current layer job and the historical average consumption time.

[0035] Thirdly, an electronic device provided in this application includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method as described in any of the first aspects.

[0036] Fourthly, embodiments of this application provide a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the method described in any of the first aspects.

[0037] Fifthly, embodiments of this application provide a computer program product that, when run on a computer, causes the computer to perform the method described in any of the first aspects.

[0038] Other features and advantages disclosed in this application will be set forth in the following description, or some features and advantages may be inferred from the description or determined without doubt, or may be learned by practicing the above-described technology disclosed in this application.

[0039] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0040] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 A flowchart illustrating a data output timeliness processing method provided in this application embodiment;

[0042] Figure 2 A schematic diagram illustrating the workflow of an end-to-end automated intelligent control scheme for data output timeliness provided in an embodiment of this application;

[0043] Figure 3A block diagram of a data output timeliness processing device provided in an embodiment of this application;

[0044] Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0045] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0046] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0047] As described in the background section, existing configuration schemes for baseline timeliness of operations heavily rely on manual labor, consuming significant time and manpower, and often lack sufficient rationality in their configuration. Therefore, this application provides a data output timeliness processing scheme to address these issues.

[0048] The embodiments of this application will be described below:

[0049] like Figure 1 As shown, Figure 1 This is a flowchart of a data output timeliness processing method provided in an embodiment of this application. The method can be applied to the management system of internal operations of an enterprise.

[0050] The method includes:

[0051] In step 101, the expected output time of the target job is obtained from the business side, and information on all jobs in the entire link of the target job is obtained through recursive data lineage.

[0052] The tasks mentioned in this embodiment refer to data generated within an enterprise, such as reports, models, software code, etc. The expected output time of a task can characterize the baseline timeliness of the task. In practical applications, when the task is a report, the expected output time can be considered as the latest deadline configured for the completion time of the daily batch run of that report.

[0053] The target task mentioned in this step is a task that requires timeliness management. In some embodiments, the target task may be a task corresponding to core data, which is obtained in the following way: The data asset is scored according to the weights corresponding to the target influencing factors; these target influencing factors include the number of user departments, the number of users accessing the platform, and the number of visits by management personnel; based on the score, the data asset is divided into core data, important data, and general data. In other words, the data asset is scored comprehensively based on the weights corresponding to influencing factors such as the number of user departments, the number of users accessing the platform, and the number of visits by management personnel, and then divided into core data, important data, and general data. Analyzing the lineage of data assets for core data can connect application assets, application libraries, and big data platforms—that is, the end-to-end lineage—thereby enabling differentiated and refined timeliness management.

[0054] The expected output time of the target task is collected from the business side; that is, the target task can be considered the lowest-level task in the entire chain. Here, the entire chain refers to a chain of tasks interconnected with the business to which the target task belongs, which can be obtained through data lineage. Data lineage is the origin and development of data, mainly including the data source, data processing method, mapping relationship, and data exit point. From a business perspective, the dimensions included in data lineage are primarily the business line to which the data belongs. Optionally, this data lineage can be obtained through program parsing. Taking a complete data processing flow as an example, each data processing step is marked with a unique identifier, and each step in the flow records its dependencies. After the program parses the logic of each step, it can generate the data lineage of the entire flow based on the dependencies and the flow. In this embodiment, by using data lineage, information about all jobs in the entire chain corresponding to the target job can be obtained, and the first-level downstream jobs of each job can be determined. For example, the target job is a report generated by the application. This report is obtained by processing data in Oracle Table 1 and Oracle Table 2 of the application library. Oracle Table 1 and Oracle Table 2 are obtained by processing the same Hive table of the big data platform. Then, by using data lineage, it can be known that the first-level downstream jobs of the Hive table include Oracle Table 1 and Oracle Table 2. The first-level downstream jobs of Oracle Table 1 include this report, and the first-level downstream jobs of Oracle Table 2 also include this report.

[0055] In step 102, based on the historical execution records of each job within a preset time period, the historical average start time and historical average execution time of each job are determined.

[0056] After obtaining information on all jobs across the entire supply chain, historical execution records of these jobs within a preset time period can be retrieved to determine the historical average start time and historical average execution time of each job. This preset time period can be the most recent month, the most recent week, the most recent two weeks, etc., and can be set according to the specific needs of the scenario. Taking the most recent month as an example, based on the historical batch processing data of a job over the past month, the historical average start time and historical average execution time of the job can be calculated, and then the historical average execution time of the job can be calculated. It should be noted that during the calculation, outliers can be removed first according to the Raida criterion (3σ criterion). When data is outside the range of (μ-σ, μ+σ), it is determined to be an outlier, thus making the final configured data output timeliness more reasonable.

[0057] In some embodiments, the historical average time mentioned in this step can be the value at the 95th percentile after arranging the historical times of the jobs within a preset time period from smallest to largest. That is, the historical average time of a job is the P95 time of that job within the preset time period. Experiments have shown that using the P95 time of each job over the past month as the historical average time can determine an accurate and effective expected start time, which is beneficial for optimizing the management of data output timeliness.

[0058] In step 103, the expected output time and expected start time of each job are configured according to the expected output time of the target job, the historical average start time and historical average consumption time of each job. The expected output time of the current layer job is calculated based on the historical average start time of all first-level downstream jobs of the current layer job, and the expected start time of the current layer job is calculated based on the expected output time of the current layer job and the historical average consumption time.

[0059] This step refers to the recursive configuration of upstream job baseline timeliness based on the expected output time of the target job, layer by layer, and the entire supply chain. The expected output time of the current-layer job is calculated based on the historical average start time of all first-level downstream jobs of that current-layer job. Optionally, the expected output time of the current-layer job can be the minimum historical average start time among all first-level downstream jobs of that current-layer job. For example, if a job has three first-level downstream jobs with historical average start times of 7:00, 7:15, and 7:20 respectively, then the expected output time of that job can be 7:00. After calculating the expected output time, the expected start time can be obtained by subtracting the corresponding historical average time from the expected output time. For example, if the expected output time of a job is 9:00 and the historical average time of that job is 30 minutes, then the expected start time of that job can be calculated as 8:30. Through this configuration, the orderly connection of each job in the entire supply chain can be ensured.

[0060] In this embodiment, the expected output time of the target task is collected from the business. Through recursive data lineage, information on all tasks across the entire chain is obtained to determine upstream and downstream relationships. Then, based on the historical execution records of each task within a preset time period, the historical average start time and historical average execution time are calculated. Combined with the expected output time of the target task, the expected output time and expected start time of each task are automatically configured based on the entire lineage. This achieves automated configuration of data output timeliness, reducing time and labor costs while improving rationality.

[0061] In some embodiments, the above method may further include: monitoring the actual output time of each task; if the actual output time is later than the corresponding expected output time, determining that the task is delayed; when a task corresponding to core data is delayed, generating a problem registration list and transmitting it to the person responsible for the task. That is, when the actual output time of a task is later than the corresponding expected output time, it is determined that the task is delayed, and users can be reached through notification channels; when core data is delayed, the system can automatically generate a problem registration list for the task development owner to automatically assess the impact of delays, follow up on timeliness issues, register the causes of problems, etc., thereby managing timeliness anomalies.

[0062] Furthermore, some embodiments of this application also provide a logical calculation scheme for automatic location of abnormal problems. Specifically, when any job experiences a delay, it is determined whether the job is in an error state or meets a first condition; the first condition is that the difference between the actual time consumed by the job and the historical average time consumed is greater than the difference between the actual output time and the historical average end time; if so, it is determined that the delay is an abnormality caused by the job, otherwise it is determined that the delay is an abnormality caused by a target upstream job; the target upstream job is the upstream job with the latest actual output time; if the target upstream job is the top-level job in the entire chain, and the job meets a second condition, it is determined that the delay is an abnormality caused by the platform; the second condition is that the difference between the actual output time and the historical average end time of the job is less than a preset value. In other words, when a job delay occurs, the system first determines whether it is due to its own internal factors. If the job is in an error state, the delay is confirmed to be internally caused. Alternatively, if the job meets the first condition, it indicates a performance anomaly, which also confirms the delay is internally caused. If it is not internally caused, it is considered to be due to upstream delays, and the latest upstream arrival time is taken as the output. If tracing back to the top-level job, the difference between its completion time and the historical average completion time is less than a preset value, it indicates that the data supply from the source system has not experienced a significant delay. Therefore, the delay may exist throughout the entire chain, which can be categorized as a platform performance issue. This allows for automatic location of anomalies, providing data support for optimization and management.

[0063] Furthermore, in some other embodiments, the above method may further include: when a delay is detected in an upstream job of the target job, determining whether the upstream job affects the output of the target job; if so, issuing an early warning. That is, when an upstream job is delayed, the system automatically calculates whether the delay affects the output of the target job; if so, the system issues an early warning to the business; otherwise, no warning is issued. In this way, through reverse deduction of automatic anomaly location, intelligent prediction of the impact of delays on the target job is achieved.

[0064] Other embodiments of this application also provide improvements in two aspects: timeliness optimization management and operational effectiveness evaluation. Specifically, the system can count the number of delays and errors for each task, determine the tasks to be optimized based on the statistical results, and then push the statistical results of the tasks to be optimized to the corresponding responsible persons so that they can optimize the tasks. In other words, by comparing the calculated expected output time of the entire chain with the average completion time of existing tasks, combined with the number of task delays and errors, the system can automatically identify the tasks to be optimized and then push them to the data responsible persons for evaluation and optimization, thus achieving timeliness optimization management. At the same time, the system can also calculate indicators such as data on-time rate, number of delays, number of errors, and repair time for business topics to conduct overall operational effectiveness evaluation, analyze the reasons for non-compliance, promote rectification, and form closed-loop management, thus achieving operational effectiveness evaluation.

[0065] To provide a more detailed explanation of the solution in this application, a specific embodiment is described below:

[0066] This embodiment relates to the control of data output timeliness. In related technologies, most industry practices rely on monitoring and scheduling platforms to ensure service delivery, supporting lineage tracing, monitoring configuration, and SMS alerts. However, these tools only provide monitoring services throughout the entire service assurance process; other aspects are fragmented. Core link analysis, upstream and downstream operational benchmark timeliness, and anomaly cause analysis heavily depend on manual intervention. Timeliness issues are managed offline, resulting in significant time and effort being spent on ensuring data output timeliness. Therefore, this embodiment provides an end-to-end automated and intelligent control solution for data output timeliness to address this problem.

[0067] The workflow of this solution is as follows: Figure 2 As shown, it includes:

[0068] S201. Key data identification: Starting from data application, data assets are scored by weighting factors such as the number of user departments, the number of users accessing the platform, and the number of accesses by management. Then, the data is divided into core data (level 1), important data (level 2), and general data (level 3) according to the Pareto principle. For core data, the lineage of data assets is analyzed, and the lineage of application assets (reports) - application library - big data platform is connected, i.e., end-to-end lineage, to carry out differentiated, refined and timely management.

[0069] S202, Data Timeliness Monitoring: Due to the long data links, complex dependencies, and numerous teams involved, there is a serious problem of duplicate monitoring configurations and alarms. Therefore, it is crucial to calculate the expected output time of data at each layer of the link and to promptly issue warnings regarding whether upstream delays affect the output of the target table. Timeliness intelligent monitoring manages this from two aspects: S2021 intelligent calculation of expected time and S2022 timeliness monitoring configuration.

[0070] S2021, Intelligent Calculation of Expected Output Time for the Entire Link, including: Step 1, collecting the target achievement time of core data from the business; Step 2, recursively obtaining information of all tables in the entire link through data lineage; Step 3, obtaining the first-level downstream tables of all tables in the link; Step 4, based on the historical batch processing data of the job over the past month, outliers are removed according to the 3σ criterion. When the data is not within the range of (μ-σ, μ+σ), it is determined to be an outlier. Then, the average end time and average start time of the job are calculated; Step 5, calculating the P95 time consumption of each job over the past month; Step 6: Expected output time of the job = min(start time of all first-level downstream tables); Step 7, Expected start time of the job = expected output time of the job - P95 time consumption over the past month, recursively calculating the expected output time of the target table job.

[0071] S2022, Timeliness Monitoring Configuration: Input the expected output time of the target operation, and automatically configure the upstream operation benchmark time based on the entire lineage. When the operation output time is later than the warning value, the user will be notified through multiple channels such as SMS, email, and WeChat.

[0072] S203, Timeliness Anomaly Management: When core data is delayed, the system automatically generates an issue registration list for the job development owner, and automatically performs timeliness anomaly location, delay impact assessment, follow-up on timeliness issue resolution, registration of the cause of the issue, the module in which it occurred, the solution, and the resolution time, etc., and automatically generates statistical analysis, including S2031 and S2032:

[0073] S2031. Automatic anomaly location logic calculation: When a job delay occurs, first determine if it is due to the job itself. If the job is in an error state, the delay can be determined to be due to the job itself. Alternatively, if the difference between the actual time taken and the historical average time taken is greater than the difference between the actual output time and the historical average end time, it indicates a performance anomaly, and the delay can also be determined to be due to the job itself. If it is not due to the job itself, it is considered to be caused by upstream delay, and the latest upstream delay is taken as the output. If the difference between the job completion time and the historical average end time is less than a preset value when tracing back to the top-level job, it indicates that there is no significant delay in the data supply from the source system. It may be that there is a certain delay in the entire link, which can be classified as a platform performance issue.

[0074] S2032. Delay Impact Target Job Intelligent Prediction Logic Calculation: When an upstream job is delayed, the system must automatically calculate whether the delay affects the output of the business target job. If it does not affect the output, there is no need to issue a warning to the business. The calculation logic is as follows: Based on the baseline timeliness of the target job, calculate the average time consumption (batch processing time) of all upstream jobs, and then deduce the baseline timeliness of the upstream jobs. Any timeliness later than the baseline timeliness is considered a delay: Baseline timeliness of the target job minus the average batch processing time of the job to obtain the first-level upstream baseline timeliness of the job. Then, the first-level upstream baseline timeliness minus the average batch processing time of the job to obtain the next higher-level baseline timeliness, and so on.

[0075] S204, Timeliness Optimization Management, which compares the expected output time of the entire chain calculated based on S2021 with the average completion time of existing jobs, and combines the number of job delays, error counts, etc., to automatically identify jobs that need to be optimized and push them to the data responsible person for evaluation and optimization;

[0076] S205. Operational effectiveness evaluation, which involves monitoring and optimizing the timeliness of data output, calculating indicators such as data timeliness rate, number of delays, number of errors, and repair time for business themes, conducting an overall operational effectiveness evaluation, analyzing the reasons for non-compliance, promoting rectification, and forming a closed-loop management system.

[0077] This embodiment of the solution starts from the classification and categorization of application assets, and integrates key data identification, full-chain lineage sorting, data output timeliness monitoring, abnormal problem management, timeliness optimization management, and operational effect evaluation into a one-stop service platform. It realizes end-to-end full-chain control of data output and data timeliness services, effectively reducing manual analysis and monitoring, and improving operational efficiency.

[0078] Corresponding to the embodiments of the aforementioned methods, this application also provides embodiments of a data output timeliness processing device and a terminal for its application:

[0079] like Figure 3 As shown, Figure 3This is a block diagram of a data output timeliness processing device provided in an embodiment of this application. The device includes:

[0080] The acquisition module 31 is used to obtain the expected output time of the target job from the business side, and obtain the information of all jobs in the entire link of the target job through data lineage recursion.

[0081] The determination module 32 is used to determine the historical average start time and historical average execution time of each job based on the historical execution records of each job within a preset time period.

[0082] The configuration module 33 is used to configure the expected output time and expected start time of each job according to the expected output time of the target job, the historical average start time and historical average consumption time of each job. The expected output time of the current layer job is calculated based on the historical average start time of all first-level downstream jobs of the current layer, and the expected start time of the current layer job is calculated based on the expected output time of the current layer job and the historical average consumption time.

[0083] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0084] This application also provides an electronic device, please refer to [link to application]. Figure 4 , Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of this application. The electronic device may include a processor 410, a communication interface 420, a memory 430, and at least one communication bus 440. The communication bus 440 is used to enable direct communication between these components. In this embodiment, the communication interface 420 of the electronic device is used for signaling or data communication with other node devices. The processor 410 may be an integrated circuit chip with signal processing capabilities.

[0085] The processor 410 described above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor, or the processor 410 can be any conventional processor.

[0086] The memory 430 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc. The memory 430 stores computer-readable instructions. When these computer-readable instructions are executed by the processor 410, the electronic device can perform the aforementioned operations. Figure 1 The various steps involved in the method implementation examples.

[0087] Alternatively, the electronic device may also include a storage controller and an input / output unit.

[0088] The memory 430, storage controller, processor 410, peripheral interface, and input / output unit are electrically connected directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses 440. The processor 410 is used to execute executable modules stored in the memory 430, such as software function modules or computer programs included in electronic devices.

[0089] The input / output unit is used to provide users with the ability to create tasks and to set optional start periods or preset execution times for those tasks, thereby enabling user-server interaction. The input / output unit may be, but is not limited to, a mouse and keyboard.

[0090] Understandable. Figure 4 The structure shown is for illustrative purposes only; the electronic device may also include components that are more advanced than those shown. Figure 4 The more or fewer components shown, or having the same Figure 4 The different configurations shown. Figure 4 The components shown can be implemented using hardware, software, or a combination thereof.

[0091] This application also provides a storage medium storing instructions. When the instructions are run on a computer, the computer program is executed by a processor to implement the method described in the method embodiment. To avoid repetition, the method will not be described again here.

[0092] This application also provides a computer program product that, when run on a computer, causes the computer to perform the method described in the method embodiment.

[0093] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0094] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0095] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0096] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0097] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0098] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for processing data output timeliness, characterized in that, include: Obtain the expected output time of the target job from the business side, and obtain information on all jobs in the entire chain of the target job through recursive data lineage; The entire chain is a chain composed of a series of operations that are interconnected with the business to which the target operation belongs; the data lineage includes the data source, data processing method, mapping relationship, and data exit point; Based on the historical execution records of each job within a preset time period, determine the historical average start time and historical average execution time of each job. Based on the expected output time of the target task, the historical average start time and historical average consumption time of each task, the expected output time and expected start time of each task are configured. The expected output time of the current layer task is calculated based on the historical average start time of all first-level downstream tasks of the current layer, and the expected start time of the current layer task is calculated based on the expected output time of the current layer task and the historical average consumption time.

2. The method according to claim 1, characterized in that, The historical average time consumption is the value at the 95th percentile after arranging the historical time consumption of the task within a preset time period from smallest to largest.

3. The method according to claim 1, characterized in that, The target job is the job corresponding to the core data, which is obtained based on the following methods: The data assets are scored based on the weights corresponding to the target influencing factors; the target influencing factors include the number of user departments, the number of users accessing the platform, and the number of visits by management. Based on the scores, the data assets are divided into core data, important data, and general data.

4. The method according to claim 1, characterized in that, The method includes: Monitor the actual output time of each task; if the actual output time is later than the corresponding expected output time, determine that the task has been delayed. When a delay occurs in the task corresponding to the core data, a problem registration list is generated and sent to the person responsible for the task.

5. The method according to claim 4, characterized in that, The method further includes: When any task is delayed, determine whether the task is in an error state or meets a first condition; the first condition is that the difference between the actual time consumed by the task and the historical average time consumed is greater than the difference between the actual output time and the historical average end time. If so, determine that the delay is an anomaly caused by the job; otherwise, determine that the delay is an anomaly caused by the target upstream job; the target upstream job is the upstream job with the latest actual output time. If the target upstream job is the top-level job in the entire chain, and the job meets the second condition, it is determined that the delay is an anomaly caused by the platform; the second condition is that the difference between the actual output time of the job and the historical average end time is less than a preset value.

6. The method according to claim 1, characterized in that, The method further includes: When a delay is detected in the upstream operation of the target operation, it is determined whether the upstream operation affects the output of the target operation. If so, an early warning is output.

7. The method according to claim 1, characterized in that, The method further includes: Count the number of delays and errors for each job, and determine the jobs to be optimized based on the statistical results; The statistical results of the task to be optimized are pushed to the corresponding person in charge so that the person in charge can optimize the task.

8. A data output timeliness processing device, characterized in that, include: The acquisition module is used to obtain the expected output time of the target job from the business side, and to obtain the information of all jobs in the entire chain of the target job through data lineage recursion. The entire chain is a chain composed of a series of operations that are interconnected with the business to which the target operation belongs; the data lineage includes the data source, data processing method, mapping relationship, and data exit point; The determination module is used to determine the historical average start time and historical average execution time of each job based on the historical execution records of each job within a preset time period. The configuration module is used to configure the expected output time and expected start time of each job based on the expected output time of the target job, the historical average start time and historical average consumption time of each job. The expected output time of the current layer job is calculated based on the historical average start time of all first-level downstream jobs of the current layer, and the expected start time of the current layer job is calculated based on the expected output time of the current layer job and the historical average consumption time.

9. A computer-readable storage medium, characterized in that, It stores a computer program thereon, which, when executed by a processor, implements the method as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method as described in any one of claims 1 to 7.

Images