Logistics job scheduling batch chasing method, device and equipment and storage medium

Abstract

The invention discloses a logistics job scheduling batch chasing method, device and equipment and a storage medium. The method comprises the steps that the number of delayed batches corresponding to each logistics job in a plurality of logistics jobs is acquired, and the number of delayed batches represents the number of times not completed within preset time; obtaining a dependency relationship between each logistics job and the downstream job; taking the logistics operation of which the delayed batch number is greater than a preset number and which does not have a dependency relationship with the downstream operation as a target batch-chasing operation; determining the batch-chasing starting time and the batch-chasing ending time of the target batch-chasing operation; obtaining a target dependent job corresponding to the target batch job and a running state of the target dependent job; and when the running state of the target dependent job is a completion state, calling a target batch function based on the batch starting time and the batch ending time, and performing batch scheduling processing on the target batch job. According to the scheme, the execution efficiency of the delay batch can be improved, and the delay time is reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of logistics and transportation, and specifically relates to a logistics operation scheduling and tracking method, device, equipment and storage medium. Background Art

[0002] With the rapid development of big data technology, logistics data warehouses are becoming increasingly important as an important tool for logistics data integration, analysis and decision support. However, in practical applications, the job scheduling and execution of logistics data warehouses are often affected by a variety of factors, resulting in delays and instability in data processing, which in turn affects the timeliness and accuracy of data.

[0003] First, data delay is one of the main factors affecting logistics data warehouse operations. Performance fluctuations in upstream source systems, network transmission delays, and data quality issues can all cause data to fail to reach the data warehouse in a timely manner. This delay not only affects the real-time nature of the data, but also makes it impossible to generate subsequent analysis and reports in a timely manner, thereby affecting the timeliness of business decisions.

[0004] Secondly, abnormal situations such as shutdown and restart and operation errors are also important reasons for the stagnation of logistics data warehouse operations. During the data processing process, the system may shut down or restart due to hardware failure, software error or human error. When the system is restored, it is often necessary to execute delayed batch tasks one by one, which not only takes a lot of time, but may also cause the front-end report to lack data on delayed batches, further affecting the integrity and accuracy of the data.

[0005] To summarize, when data warehouse operations stagnate due to various other factors such as data delays in the upstream source system, shutdown and restart, or job errors, there will be a large number of delayed batches in the data warehouse operations. Waiting for the delayed job batches to finish one by one will cause the front-end report to lack data for the delayed batches, affecting the timeliness and accuracy of the data. When the above situation occurs, the process of executing the delayed batch tasks one by one will take a lot of time. Summary of the invention

[0006] In view of the deficiencies in the prior art, the purpose of the present invention is to provide a logistics operation scheduling and tracking batch method, device, equipment and storage medium to solve the problems of low execution efficiency and long execution time of delayed batches in the prior art.

[0007] According to one aspect of the present application, a logistics operation scheduling and approval method is disclosed, comprising: Obtaining the number of delayed batches corresponding to each of the multiple logistics operations, wherein the number of delayed batches represents the number of times that the batches are not completed within a predetermined time; Obtaining the dependency relationship between each of the logistics operations and downstream operations; The logistics operations with delayed batches greater than the preset number and no dependency on downstream operations are selected as target batch tracking operations; Determine the start time and end time of the target follow-up approval operation; Obtaining the target dependent job corresponding to the target batch tracking job and the running status of the target dependent job; When the running state of the target dependent job is in the completed state, based on the follow-up start time and the follow-up end time, the target follow-up function is called to perform follow-up scheduling processing on the target follow-up job.

[0008] In some embodiments, obtaining the number of delayed batches corresponding to each of the multiple logistics operations includes: Obtain the current system time, the upstream batch end time, and the time interval corresponding to the operation frequency of each logistics operation; The delayed batch corresponding to each logistics operation is determined based on the following formula: Number of delayed batches = (system current time - upstream batch end time) / time interval corresponding to the operation frequency.

[0009] In some embodiments, determining the start time of the target follow-up operation includes: Acquire the job type of the target follow-up batch job and the loading mode of the target follow-up batch job; When the job type of the target batch tracking job is a sampling task, the loading mode of the target batch tracking job is a full loading mode, and there is no dependency relationship between the target batch tracking job and downstream jobs, obtaining job flag information of the target batch tracking job; If the job flag information of the target follow-up job is not full clear, the start time of the follow-up job is determined based on the following formula: The start time of the follow-up batch = the end time of the upstream batch + (the number of delayed batches - 1) × the time interval corresponding to the operation frequency + 1 second; If the job flag information of the target batch job is not "partial clearing", the start time of the first batch in the queue information table is obtained; The start time of the first batch is used as the start time of the follow-up batch.

[0010] In some embodiments, determining the target follow-up batch operation's end time includes: Obtaining the dependency relationship between the target batch job and the upstream job; If the target batch chasing operation depends on the trigger of the upstream operation, the upstream batch ending time corresponding to the upstream operation is used as the batch chasing ending time; If the target batch follow-up job does not depend on the upstream job trigger, the batch follow-up end time is determined based on the following formula: The end time of the follow-up batch = the end time of the upstream batch + the number of delayed batches × the time interval corresponding to the operation frequency.

[0011] In some embodiments, calling the target follow-up batch function based on the follow-up batch start time and the follow-up batch end time to perform follow-up batch scheduling processing on the target follow-up batch job includes: Obtaining attribute information of the target batch job; Update the status of the target batch tracking job to being executed, and update the status of the trigger to be invalid; Determine a time range for the follow-up approval based on the follow-up approval start time and the follow-up approval end time; The job tracking sub-function in the target tracking function is called to perform tracking scheduling processing on the target tracking job within the tracking time range.

[0012] In some embodiments, calling the job tracking sub-function in the target tracking function to perform tracking scheduling processing on the target tracking job within the tracking time range includes: Based on the attribute information, acquiring execution script information of the target batch job; When the target batch job satisfies the job type of a sampling task and the loading mode of the target batch job is a full loading mode, calling the data loading script in the execution script information to perform batch scheduling processing on the target batch job within the batch time range; When at least one of the conditions that the job type in the target batch job is a sampling task or the loading mode of the target batch job is a full loading mode is not satisfied; Determine whether the target batch tracking job is referenced by other systems; If the target batch job is referenced by other systems, When the target batch-following job is subjected to batch-following scheduling processing, the data loading script of the execution script information is called for each batch in the target batch-following job.

[0013] In some embodiments, after calling the target batch tracking function based on the batch tracking start time and the batch tracking end time and performing batch tracking scheduling processing on the target batch tracking job, the method further includes: After determining that the target skin-tracking operation is completed, the downstream operation is triggered and the state of the trigger is updated to be valid.

[0014] The present invention includes but is not limited to the following beneficial effects: (1) In the present invention, as long as the number of delayed batches is greater than the preset number and there is no dependency relationship with the downstream operation, the operation can be considered as a target batch tracking operation, and the batch tracking function can be called to perform batch tracking scheduling processing on the target batch tracking operation. Furthermore, when there are multiple target batch tracking operations, multiple target batch tracking operations can be bundled together to complete the batch tracking processing. Compared with the execution of delayed operations one by one, the delay time is shortened, the timeliness of the overall data of the data warehouse is improved, the delay time is reduced, the operating efficiency of the data warehouse is improved, the failure probability of the server running at high load for a long time is reduced, and the backlog of unfinished operations can be effectively reduced, thereby improving the overall operating efficiency; (2) The present invention obtains the number of delayed batches of each logistics operation and its dependency relationship with downstream operations, which can provide clear data support for logistics management and enhance the transparency of the operation process. Based on these data, it can identify and prioritize the scheduling of logistics operations that are not affected by downstream operations, so that resources can be more reasonably allocated and used, and resource idleness can be reduced; further, Taking into account the dependency between jobs, the resource competition caused by job conflicts can be effectively avoided, thereby improving the smoothness of the overall job; (3) The present invention can effectively reduce the overall delay risk of subsequent jobs caused by the delay of the predecessor job by real-time monitoring of job delays and timely follow-up scheduling; (4) The present invention dynamically determines the follow-up time according to different job types and loading modes, making the scheduling more flexible, able to adapt to the needs of different situations, and optimize the job sequence and time arrangement; (5) The present invention can prevent the follow-up process from conflicting with the scheduling generation queue program by setting the trigger to an invalid state when performing follow-up scheduling processing on the target follow-up job; (6) The present invention can adjust the follow-up strategy in a targeted manner by distinguishing the type and loading mode of the job, thereby improving the intelligence level of job scheduling; (7) When the target follow-up job is a job referenced by other systems, the present invention calls the data loading script for each batch of each target follow-up job, thereby improving the integrity and consistency of the job and ensuring the accuracy and reliability of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0016] Figure 1 This is a flow chart of the logistics operation scheduling and approval method of the embodiment of the present application; Figure 2 is a flow chart for obtaining the number of delayed batches corresponding to each logistics operation in a plurality of logistics operations according to an embodiment of the present application; Figure 3 A flowchart for determining a start time for a target follow-up approval operation according to an embodiment of the present application; Figure 4 A flow chart for determining the end time of a target follow-up approval operation according to an embodiment of the present application; Figure 5 It is a flow chart of an embodiment of the present application, based on the start time of the follow-up approval and the end time of the follow-up approval, calling the target follow-up approval function, and performing follow-up approval scheduling processing on the target follow-up approval job; Figure 6 It is a flow chart of calling the job tracking sub-function in the target tracking function in the embodiment of the present application to perform tracking scheduling processing on the target tracking job within the tracking time range; Figure 7 It is a structural block diagram of a logistics operation scheduling and approval tracking device according to an embodiment of the present application; Figure 8 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] An embodiment of the present invention provides a method for scheduling and tracing logistics operations, the method comprising: obtaining the number of delayed batches corresponding to each logistics operation in a plurality of logistics operations, wherein the number of delayed batches represents the number of times that the operations are not completed within a predetermined time; obtaining the dependency relationship between each logistics operation and a downstream operation; taking a logistics operation whose number of delayed batches is greater than a preset number and has no dependency relationship with the downstream operation as a target tracing operation; determining a tracing start time and a tracing end time of the target tracing operation; obtaining a target dependent operation corresponding to the target tracing operation and the running status of the target dependent operation; when the running status of the target dependent operation is a completed state, based on the tracing start time and the tracing end time, calling a target tracing function to perform tracing scheduling processing on the target tracing operation. In this method, as long as the number of delayed batches is greater than the preset number and there is no dependency relationship with the downstream jobs, the job can be considered as the target batch chasing job, and the batch chasing function can be called to perform batch chasing scheduling processing on the target batch chasing job. Furthermore, when there are multiple target batch chasing jobs, multiple target batch chasing jobs can be bundled together to complete the batch chasing processing. Compared with the execution of delayed jobs one by one, the delay time is shortened, the timeliness of the overall data of the data warehouse is improved, the delay time is reduced, the operating efficiency of the data warehouse is improved, the failure probability of the server running at high load for a long time is reduced, and the backlog of unfinished jobs can be effectively reduced, thereby improving the overall operating efficiency.

[0018] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0019] For ease of understanding, the specific process of the embodiment of the present invention is described below. Specifically, Figure 1 A flow chart of the logistics operation scheduling and approval method of this application, the steps include: S100. Obtain the number of delayed batches corresponding to each logistics operation among multiple logistics operations.

[0020] Specifically, the number of delayed batches represents the number of times that they are not completed within the scheduled time.

[0021] In one example, Figure 2 A preferred method process for obtaining the number of delayed batches corresponding to each logistics operation in multiple logistics operations is given, see Figure 2 The method flow includes the following steps: S200, obtaining the current system time, the upstream batch end time, and the time interval corresponding to the operation frequency of each logistics operation.

[0022] S202. Determine the delayed batch corresponding to each logistics operation based on the following formula: Number of delayed batches = (system current time - upstream batch end time) / time interval corresponding to the operation frequency.

[0023] For example, suppose we have a logistics operation A, the frequency of which corresponds to a time interval of once a day, the upstream batch end time is October 1, and the current system time is October 4. The upstream batch end time may refer to the execution end time of the last batch of the upstream operation.

[0024] The number of delayed batches = (system current time - upstream batch end time) / time interval corresponding to the job frequency. That is, in this example, the number of delayed batches = (October 4 - October 1) / 1 day = 3.

[0025] S102. Obtain the dependency relationship between each logistics operation and downstream operations.

[0026] Specifically, each current logistics operation may have a certain association relationship with the upstream or downstream operation. For example, the current logistics operation can only be executed after the upstream operation is completed, which means that there is a dependency relationship between the upstream operation and the current logistics operation. If the triggering of the downstream operation depends on the end time of the current logistics operation, it means that there is a dependency relationship between the current logistics operation and the downstream operation. In one example, the dependency relationship between the logistics operation and the downstream operation can be pre-stored in the system, or it can be determined in real time based on the execution order or execution conditions.

[0027] S104. The logistics operations whose delayed batch number is greater than a preset number and have no dependency relationship with the downstream operations are selected as target batch tracking operations.

[0028] Specifically, in this step, it can be determined whether the number of delayed batches corresponding to each of the multiple logistics operations is greater than a preset number, wherein the preset number can be 2, 3, 5, etc., and the specific number can be set based on actual needs. In this example, the preset number is 2, that is, the logistics operations with a delayed batch number greater than 2 and no dependency relationship with downstream operations are used as target batch tracking operations.

[0029] S106: Determine the start time and end time of the target follow-up approval operation.

[0030] In one example, Figure 3 A preferred method flow for determining the start time of the target follow-up job is shown, referring to Figure 3 As shown, the method flow includes the following steps: S300: Obtain the job type of the target follow-up batch job and the loading mode of the target follow-up batch job.

[0031] S302: When the job type of the target batch tracking job is a sampling task, the loading mode of the target batch tracking job is a full loading mode, and there is no dependency relationship between the target batch tracking job and the downstream job, obtain job flag information of the target batch tracking job.

[0032] S304: If the job flag information of the target follow-up job is not full-volume clear, the start time of the follow-up job is determined based on the following formula: The start time of the follow-up batch = the end time of the upstream batch + (the number of delayed batches - 1) × the time interval corresponding to the operation frequency + 1 second.

[0033] S306: If the job flag information of the target batch job is not "partial clear", the start time of the first batch in the queue information table is obtained.

[0034] S308. The start time of the first batch is used as the start time of the follow-up batch.

[0035] In this example, based on the above example, if logistics operation A is the target batch tracking operation, and the operation type of the target logistics operation is a snapshot task (STG), the loading mode is full (F), and there is no downstream operation that needs to wait for it. Assume that the frequency of this operation corresponds to a time interval of once a day, the end time of the upstream batch is October 1, and the current system time is October 4, and the number of delayed batches is determined to be 3 based on the above calculation. If the operation flag information of the target batch tracking operation is non-full clearing, then based on the formula: batch tracking start time = upstream batch end time + (delayed batch number - 1) × operation frequency corresponding time interval + 1 second, it can be determined that the batch tracking start time in this example = October 1 + (3 -1) × 1 day + 1 second = October 3 00:00:01. Among them, the snapshot task (STG, Snapshot Task) refers to the operation of extracting data from the source system. Its main purpose is to extract data from one system (usually a database) to another system (such as a data warehouse or data lake) for subsequent analysis or processing. Application scenarios: commonly used in scenarios such as data migration, data synchronization, and data analysis. Full load means that each time an extraction task is executed, the entire data set is extracted instead of only the data that has changed since the last load. This method ensures the integrity of the extracted data, but may result in higher resource consumption and longer execution time.

[0036] Furthermore, if the job flag of the target follow-up batch job is not non-full clear, then the first batch start time of the queue information table is further obtained, that is, the batch start time corresponding to the earliest batch in the queue information table is obtained. Assuming that this time is October 2, then at this time, the first batch start time is used as the follow-up batch start time, that is, the follow-up batch start time is October 2.

[0037] Furthermore, in one example, Figure 4 A preferred method flow for determining the end time of the target follow-up job is shown, referring to Figure 4 As shown, the method flow includes the following steps: S400: Obtain the dependency relationship between the target batch tracking job and the upstream job.

[0038] Specifically, the dependency relationship between the target batch tracking job and the upstream job includes that the target batch tracking job depends on the upstream result to trigger the execution, and at this time it is considered that there is a dependency relationship between the target batch tracking job and the upstream job; further, the dependency relationship between the target batch tracking job and the upstream job also includes that the target batch tracking job is executed independently and does not depend on the upstream result, and at this time it is considered that there is no dependency relationship between the target batch tracking job and the upstream job.

[0039] S402: If the target batch tracking operation depends on the triggering of the upstream operation, the upstream batch end time corresponding to the upstream operation is used as the batch tracking end time.

[0040] Specifically, if the target batch tracking job depends on the triggering of the upstream job, that is, there is a dependency relationship between the target batch tracking job and the upstream job, the upstream batch end time corresponding to the upstream job is used as the batch tracking end time.

[0041] S404: If the target follow-up job does not depend on the upstream job trigger, the end time of the follow-up job is determined based on the following formula: The end time of the follow-up batch = the end time of the upstream batch + the number of delayed batches × the time interval corresponding to the operation frequency.

[0042] Continuing with the above example, if the target follow-up job does not depend on the upstream job trigger, that is, there is no dependency between the target follow-up job and the upstream job, then based on the formula: follow-up end time = upstream batch end time + number of delayed batches × time interval corresponding to job frequency, it can be obtained that the follow-up end time = October 1 + 3 × 1 day = October 4.

[0043] It can be understood that the technical solution of the present invention dynamically determines the chasing time (chasing start time and chasing end time) according to different job types and loading modes, making the scheduling more flexible, able to adapt to the needs of different situations, and optimize the job sequence and time arrangement.

[0044] S108: Obtain the target dependent job corresponding to the target batch tracking job and the running status of the target dependent job.

[0045] It can be understood that in this technical solution, as long as the number of delayed batches is greater than the preset number and there is no dependency relationship with the downstream jobs, the job can be considered as a target batch chasing job, and the batch chasing function can be called to perform batch chasing scheduling processing on the target batch chasing job. Furthermore, when there are multiple target batch chasing jobs, multiple target batch chasing jobs can be bundled together to complete the batch chasing processing. Compared with the execution of delayed jobs one by one, the delay time is shortened, the timeliness of the overall data of the data warehouse is improved, the delay time is reduced, the operating efficiency of the data warehouse is improved, the failure probability of the server running at high load for a long time is reduced, and the backlog of unfinished jobs can be effectively reduced, thereby improving the overall operating efficiency; further, The present invention obtains the number of delayed batches of each logistics operation and its dependency with downstream operations, can provide clear data support for logistics management, enhance the transparency of the operation process, and based on these data, can identify and prioritize those logistics operations that are not affected by downstream operations, so that resources can be more reasonably allocated and used, reducing idle resources; further, considering the dependency between operations, can effectively avoid resource competition caused by operation conflicts, and improve the smoothness of the overall operation; furthermore, the present invention can effectively reduce the overall delay risk of subsequent operations due to delays in predecessor operations by monitoring the delays of operations in real time and performing follow-up batch scheduling in a timely manner.

[0046] Specifically, after determining the target batch job, it is necessary to first determine whether the target batch job has a target dependent job. If the target batch job has a target dependent job, it is necessary to determine the current running status of the target dependent job. Only when the running status of the target dependent job is completed can the batch scheduling processing of the target batch job be started.

[0047] S110. When the running status of the target dependent job is completed, based on the start time and the end time of the follow-up batch, the target follow-up batch function is called to perform follow-up batch scheduling processing on the target follow-up batch job.

[0048] Specifically, the batch scheduling of target batch jobs refers to adding jobs that are not originally in the current scheduling batch (target batch jobs) to the subsequent scheduling batches during the job scheduling process. This is to ensure that jobs can be reasonably scheduled and processed according to their priority or other scheduling criteria, which helps to make more reasonable use of system resources and ensure that important or urgent jobs can be processed in a timely manner.

[0049] In one example, the target batch function may be a batch main function (main). Figure 5 As shown in the figure, a flow chart of calling the target batch function and scheduling the target batch job based on the batch start time and batch end time is given. Figure 5 , the process includes the following steps: S500: Obtain attribute information of a target batch tracking job.

[0050] The attribute information of the target batch job includes, but is not limited to, the execution script name, file path, file name, file suffix, loading mode, loading table, loading type, job flag, file name to be loaded, etc. The execution script name may be the name of the script used to execute the batch job, the file path may be the storage path of the file to be loaded, the file name may be the name of the file to be loaded, the file suffix may be the suffix of the file to be loaded (such as .csv, .txt, etc.), the loading mode may include full load or incremental load, the loading table may be the target table for specifying the loaded data, the loading type may include batch load or real-time load, the job flag is used to identify the uniqueness or characteristics of the job, and the file name to be loaded may contain a date or other dynamic information.

[0051] S502: Update the status of the target batch job to being executed, and update the status of the trigger to be invalid.

[0052] Specifically, the status of the target batch job needs to be updated to executing to mark the job as running to prevent other processes from operating the same job, and the status of the trigger needs to be updated to invalid to avoid conflicts between the target batch job and the program that schedules the generation queue in the batch project.

[0053] S504: Determine the time range for the follow-up approval based on the start time and the end time of the follow-up approval.

[0054] Specifically, the time range for the follow-up approval is the time range formed by the start time of the follow-up approval to the end time of the follow-up approval.

[0055] S506: Call the job approval sub-function in the target approval function to schedule the target approval job within the approval time range.

[0056] The job tracking sub-function may be an invkJob script. In this example, the invkJob script is a custom script used to call or execute a target tracking job. It contains some logic, such as checking the status of the target tracking job, determining when and how to start the target tracking job, and how to handle the cleanup work after the target tracking job is completed.

[0057] In one example, if Figure 6 As shown in the figure, a method flow chart of calling the job tracking sub-function in the target tracking function to schedule the target tracking job within the tracking time range is given. Figure 6 The method flow includes the following steps: S600: Based on the attribute information, obtain the execution script information of the target batch job.

[0058] Specifically, in this step, based on the attribute information of the target batch job, the name of the script of the target batch job is determined. This script may contain some job execution logic, such as data processing, file operation, etc.

[0059] S602: When the target batch tracking job satisfies the job type of sampling task and the loading mode of the target batch tracking job is full loading mode, the data loading script in the execution script information is called to perform batch tracking scheduling processing on the target batch tracking job within the batch tracking time range.

[0060] Specifically, the data loading script may be a LoadFromFile data loading program, which is used to load the data of the target batch job into the target system.

[0061] S604: When at least one of the conditions that the job type in the target batch tracking job is a sampling task or the loading mode of the target batch tracking job is a full loading mode is not satisfied, determine whether the target batch tracking job is referenced by other systems.

[0062] S606: If the target batch tracking job is referenced by other systems, when the target batch tracking job is being processed for batch tracking, the data loading script of the execution script information is called for each batch in the target batch tracking job.

[0063] Specifically, when the job type in the target batch job is a sampling task or the loading mode of the target batch job is a full loading mode, at least one condition is not met, it is further determined whether the target batch job is referenced by other systems. If so, when the target batch job is scheduled for batch tracking, the data loading script of the execution script information is called for each batch in the target batch job. It is understandable that in order to ensure the data integrity and consistency of each batch and reduce the error rate of any missing or inconsistent data to the downstream system, it is necessary to call the data loading script of the execution script information for each batch. By calling the data loading script for each batch of each target batch job, the integrity and consistency of the job can be improved, and the accuracy and reliability of the data can be ensured.

[0064] Furthermore, based on the start time and the end time of the follow-up batch, the target follow-up batch function is called, and the target follow-up batch job is scheduled for follow-up batch. The method also includes: after determining that the follow-up batch of the target follow-up batch job is completed, triggering the downstream job, and updating the trigger status to valid.

[0065] Specifically, it is possible to determine whether the target batch tracking job is completed by checking the returned status code, the status field in the log file or the database. For example, when the returned status code is 0, it is considered that the batch tracking is completed, and when the returned status code is 1, it is considered that the batch tracking is not completed. Further, after determining that the batch tracking is completed, the downstream job is triggered, for example, the execution script of the downstream job can be directly called, or it can be determined whether to trigger the downstream job based on certain conditions (such as data volume, data integrity, etc.).

[0066] You can also directly call the script of the downstream job through system commands or scheduling tools. For example, you can use Shell commands or the API of the scheduling tool to execute downstream jobs. You can also send message notifications. For example, in some architectures, you can send notifications through message queues (such as Kafka, RabbitMQ, etc.) to inform the downstream system to start processing data. This method is more suitable for microservice architecture and can achieve loose coupling.

[0067] Further, in an example, calling the data loading script in the execution script information may specifically be calling a data file processing (PreData) script. Further, calling the data loading script in the execution script information to perform batch scheduling processing on the target batch job within the batch time range may specifically be: Call the data file processing (PreData) script to preprocess the target batch job to ensure that the quality and format of the data meet the requirements of subsequent loading. After the data processing is completed, update the status of the target batch job to reflect the current execution status and reset the end time of the data so that subsequent jobs can correctly identify the validity of the data. Furthermore, start the Gpfdist service so that the external table can access the preprocessed data file. Furthermore, after the data loading is completed, stop the Gpfdist service to release the occupied port.

[0068] Furthermore, in one example, the data file processing (PreData) script is called to pre-process the target batch job. The data files that have been processed historically can also be deleted, and 5 minutes of jobs are loaded every hour to obtain the last 55.csv file time. In other cases, the last file time is obtained normally. Furthermore, the batch end time is reset with the last file time, and the batch end time is reset according to different job types; the data file list is looped and copied to the processing path. By resetting the end time, the consistency of the data can be improved.

[0069] Further, according to one aspect of the present application, a logistics operation scheduling and approval tracking device is also disclosed, such as Figure 7 As shown, the device comprises: A delayed batch number acquisition module is used to obtain the number of delayed batches corresponding to each logistics operation in multiple logistics operations, wherein the delayed batch number represents the number of times that the batches are not completed within the scheduled time; The dependency acquisition module is used to obtain the dependency relationship between each logistics operation and downstream operations; A target batch tracking operation determination module is used to select logistics operations with a delayed batch number greater than a preset number and no dependency relationship with downstream operations as target batch tracking operations; A follow-up approval time determination module is used to determine the follow-up approval start time and the follow-up approval end time of the target follow-up approval job; The target dependent job acquisition module is used to acquire the target dependent job corresponding to the target batch job and the running status of the target dependent job; The batch scheduling process is used to call the target batch function based on the batch start time and batch end time to perform batch scheduling on the target batch job when the running status of the target dependent job is completed.

[0070] Furthermore, the delayed batch number acquisition module includes: The first acquisition unit is used to obtain the current system time, the end time of the upstream batch, and the time interval corresponding to the operation frequency of each logistics operation; The determination unit is used to determine the delayed batch corresponding to each logistics operation based on the following formula: Number of delayed batches = (system current time - upstream batch end time) / time interval corresponding to the operation frequency.

[0071] Furthermore, the module for determining the time of follow-up approval includes: A second acquisition unit, used to acquire the job type of the target follow-up batch job and the loading mode of the target follow-up batch job; The third acquisition unit is used to acquire the job flag information of the target batch tracking job when the job type of the target batch tracking job is a sampling task, the loading mode of the target batch tracking job is a full loading mode, and there is no dependency relationship between the target batch tracking job and the downstream job; The first determination unit for the start time of the follow-up approval is used to determine the start time of the follow-up approval based on the following formula if the job flag information of the target follow-up approval job is not full-volume full-clearance: The start time of the follow-up batch = the end time of the upstream batch + (the number of delayed batches - 1) × the time interval corresponding to the operation frequency + 1 second; A first batch start time determination unit is used to obtain the first batch start time in the queue information table if the job flag information of the target batch job is not non-full clear; The second determination unit of the start time of the follow-up approval is used to use the start time of the first batch as the start time of the follow-up approval; A fourth acquisition unit, used to acquire the dependency relationship between the target batch job and the upstream job; The first unit for determining the end time of the follow-up batch is used to determine the end time of the upstream batch corresponding to the upstream job as the end time of the follow-up batch if the target follow-up batch job depends on the trigger of the upstream job; The second determination unit for the end time of the follow-up batch is used to determine the end time of the follow-up batch based on the following formula if the target follow-up batch operation does not depend on the triggering of the upstream operation: The end time of the follow-up batch = the end time of the upstream batch + the number of delayed batches × the time interval corresponding to the operation frequency.

[0072] Furthermore, the batch scheduling processing module includes: A fifth acquisition unit, used to acquire attribute information of a target batch job; An updating unit, used to update the status of the target batch job to being executed, and to update the status of the trigger to be invalid; A time range determination unit for the follow-up approval, used to determine the time range for the follow-up approval based on the start time and the end time of the follow-up approval; The job approval sub-function determining unit is used to call the job approval sub-function in the target approval function to perform approval scheduling processing on the target approval job within the approval time range.

[0073] Furthermore, the job tracking sub-function determination unit includes: An execution script information acquisition subunit is used to acquire the execution script information of the target batch job based on the attribute information; The data loading script calling subunit is used to call the data loading script in the execution script information to perform batch scheduling processing on the target batch job within the batch time range when the target batch job satisfies the job type of the sampling task and the loading mode of the target batch job is the full loading mode; The target batch tracking job reference determination subunit is used to determine whether the target batch tracking job is referenced by other systems when at least one condition is not met in the target batch tracking job when the job type is a sampling task or the loading mode of the target batch tracking job is a full loading mode; The sub-units are called one by one, and are used to call the data loading script of the execution script information for each batch in the target batch job when the target batch job is referenced by other systems and the target batch job is scheduled for batch scheduling.

[0074] The application introduction of the relevant modules of the device in this example can refer to the relevant introduction of the principle of the above method, which will not be repeated here.

[0075] According to another aspect of the present application, the present application also discloses an electronic device, which includes a memory and at least one processor, wherein instructions are stored in the memory; at least one processor calls the instructions in the memory to enable the electronic device to execute each step of the above-mentioned logistics operation scheduling and tracking method.

[0076] above Figure 7 The grid warehouse billing device based on operating status in the embodiment of the present invention is described in detail from the perspective of modular functional entities, and the electronic device in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0077] Figure 8 8 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device 800 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 810 (for example, one or more processors) and a memory 820, and one or more storage media 830 (for example, one or more mass storage devices) storing application programs 833 or data 832. Among them, the memory 820 and the storage medium 830 can be short-term storage or permanent storage. The program stored in the storage medium 830 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the electronic device 800. Furthermore, the processor 810 may be configured to communicate with the storage medium 830 to execute a series of instruction operations in the storage medium 830 on the electronic device 800.

[0078] The electronic device 800 may also include one or more power supplies 840, one or more wired or wireless network interfaces 850, one or more input and output interfaces 850, and / or one or more operating systems 831, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. It will be appreciated by those skilled in the art that Figure 8 The structure of the electronic device shown does not constitute a limitation on the electronic device, and may include more or less components than shown in the figure, or combine some components, or arrange the components differently.

[0079] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the logistics operation scheduling and tracking method.

[0080] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device, or unit can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0081] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0082] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A logistics operation scheduling and tracking method, characterized in that: The method comprises: Obtaining the number of delayed batches corresponding to each of the multiple logistics operations, wherein the number of delayed batches represents the number of times that the batches are not completed within a predetermined time; Obtaining the dependency relationship between each of the logistics operations and downstream operations; The logistics operations with delayed batches greater than the preset number and no dependency on downstream operations are selected as target batch tracking operations; Determine the start time and end time of the target follow-up approval operation; Obtaining the target dependent job corresponding to the target batch tracking job and the running status of the target dependent job; When the running state of the target dependent job is in the completed state, based on the follow-up start time and the follow-up end time, the target follow-up function is called to perform follow-up scheduling processing on the target follow-up job.

2. The method for scheduling and approving logistics operations according to claim 1, characterized in that: The obtaining of the number of delayed batches corresponding to each of the plurality of logistics operations comprises: Obtain the current system time, the upstream batch end time, and the time interval corresponding to the operation frequency of each logistics operation; The delayed batch corresponding to each logistics operation is determined based on the following formula: Number of delayed batches = (system current time - upstream batch end time) / time interval corresponding to the operation frequency.

3. The method for scheduling and approving logistics operations according to claim 1, characterized in that: Determining the start time of the target follow-up approval operation includes: Acquire the job type of the target follow-up batch job and the loading mode of the target follow-up batch job; When the job type of the target batch tracking job is a sampling task, the loading mode of the target batch tracking job is a full loading mode, and there is no dependency relationship between the target batch tracking job and downstream jobs, obtaining job flag information of the target batch tracking job; If the job flag information of the target follow-up job is not full clear, the start time of the follow-up job is determined based on the following formula: The start time of the follow-up batch = the end time of the upstream batch + (the number of delayed batches - 1) × the time interval corresponding to the operation frequency + 1 second; If the job flag information of the target batch job is not "partial clearing", the start time of the first batch in the queue information table is obtained; The start time of the first batch is used as the start time of the follow-up batch.

4. The method for scheduling and approving logistics operations according to claim 1, characterized in that: Determining the end time of the target follow-up approval operation includes: Obtaining the dependency relationship between the target batch job and the upstream job; If the target batch chasing operation depends on the trigger of the upstream operation, the upstream batch ending time corresponding to the upstream operation is used as the batch chasing ending time; If the target batch follow-up job does not depend on the upstream job trigger, the batch follow-up end time is determined based on the following formula: The end time of the follow-up batch = the end time of the upstream batch + the number of delayed batches × the time interval corresponding to the operation frequency.

5. The method for scheduling and approving logistics operations according to claim 3 is characterized in that: The calling of the target follow-up approval function based on the follow-up approval start time and the follow-up approval end time to perform follow-up approval scheduling processing on the target follow-up approval job includes: Acquire attribute information of the target batch job; Update the status of the target batch tracking job to being executed, and update the status of the trigger to be invalid; Determine a time range for the follow-up approval based on the follow-up approval start time and the follow-up approval end time; The job tracking sub-function in the target tracking function is called to perform tracking scheduling processing on the target tracking job within the tracking time range.

6. The method for scheduling and approving logistics operations according to claim 5, characterized in that: The calling of the job approval sub-function in the target approval function and performing approval scheduling processing on the target approval job within the approval time range includes: Based on the attribute information, acquiring execution script information of the target batch job; When the target batch job satisfies the job type of a sampling task and the loading mode of the target batch job is a full loading mode, calling the data loading script in the execution script information to perform batch scheduling processing on the target batch job within the batch time range; When at least one of the conditions that the job type in the target batch job is a sampling task or the loading mode of the target batch job is a full loading mode is not satisfied; Determine whether the target batch tracking job is referenced by other systems; If the target batch job is referenced by other systems, When the target batch-following job is subjected to batch-following scheduling processing, the data loading script of the execution script information is called for each batch in the target batch-following job.

7. The method for scheduling and approving logistics operations according to claim 1, characterized in that: After calling the target follow-up approval function based on the follow-up approval start time and the follow-up approval end time and performing follow-up approval scheduling processing on the target follow-up approval job, the method further includes: After determining that the target skin-tracking operation is completed, the downstream operation is triggered and the state of the trigger is updated to be valid.

8. A logistics operation scheduling and tracking device, characterized in that: The device comprises: A delayed batch number acquisition module is used to acquire the number of delayed batches corresponding to each of the multiple logistics operations, wherein the delayed batch number represents the number of times that the batches are not completed within a predetermined time; A dependency acquisition module, used to acquire the dependency between each of the logistics operations and downstream operations; A target batch tracking operation determination module is used to select logistics operations with a delayed batch number greater than a preset number and no dependency relationship with downstream operations as target batch tracking operations; A follow-up approval time determination module, used to determine the follow-up approval start time and the follow-up approval end time of the target follow-up approval job; A target dependent job acquisition module, used to acquire the target dependent job corresponding to the target batch tracking job and the running status of the target dependent job; The batch follow-up scheduling process is used to call the target batch follow-up function based on the batch follow-up start time and the batch follow-up end time to perform batch follow-up scheduling process on the target batch follow-up job when the running state of the target dependent job is the completion state.

9. An electronic device, characterized in that: The electronic device includes a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the electronic device executes each step of the logistics operation scheduling and tracking method as claimed in claims 1-7.

10. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the various steps of the logistics operation scheduling and batch tracking method as described in any one of claims 1 to 7 are implemented.

Images