A data batch running system, method, device and storage medium

Abstract

The application discloses a data batch running system, method and device and a storage medium, relates to the technical field of data batch running, and comprises a task judgment interface, a first data batch running module, a judgment module, a second data batch running module and a third data batch running module. The task judgment interface is used for judging whether a current target batch running task has a downstream dependent task. The first data batch running module is used for executing a batch running process of corresponding metadata when the target batch running task has no downstream dependent task, and marking the batch running process as preliminary batch running completion. The judgment module is used for judging whether a data query request is acquired. The second data batch running module is used for executing the target batch running task corresponding to the preliminary batch running completion through a preset cache queue to filter out target data from corresponding batch running data when the data query request is acquired. The third data batch running module is used for performing batch running on the target batch running task corresponding to the preliminary batch running completion based on a preset data batch running time period when the data query request is not acquired. In this way, the situation of resource contention during a batch running peak period is alleviated, and the batch running efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of batch data processing technology, and in particular to a batch data processing system, method, device and storage medium. Background Technology

[0002] With the advent of the digital age, the company's data volume is growing exponentially every year. The number of batch processing tasks requiring nighttime runs in the data warehouse is increasing daily, consuming ever more resources. The growth in tasks has exceeded resource constraints, making it difficult to complete new, important tasks within the stipulated timeframe. Many existing tasks consume significant resources daily, but the users may not necessarily use the new data every day, leading to resource waste. Currently, common approaches are twofold: First, for situations with rapid data growth and limited resources, increasing servers or optimizing batch processing logic; the former increases batch processing queue resources, while the latter reduces resource usage per task. However, there are limits to increasing server resources, and optimizing logic requires substantial human resources, each representing a significant cost investment. While increasing servers ensures overall resources exceed the batch processing requirements, it still cannot meet the demands during peak periods, causing some jobs to fail to complete within the stipulated time. Second, for batch processing needs to be completed within a specified timeframe, precise time and dependency control are commonly used. This ensures that batch processing begins immediately upon completion of time and dependency requirements. However, during periods of intense resource contention, this guarantee fails, and jobs may still fail to complete within the stipulated time.

[0003] Therefore, it is evident that how to rationally utilize batch processing queue resources and improve data batch processing efficiency is a problem that needs to be solved in this field. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a data batch processing system, method, device, and storage medium that can alleviate resource contention during peak data batch processing periods and improve data batch processing efficiency. The specific solution is as follows:

[0005] Firstly, this application provides a data batch processing system, comprising:

[0006] The task judgment interface is used to determine whether the current target batch task has downstream dependent tasks;

[0007] The first data batch processing module is used to perform the corresponding metadata batch processing process based on the target batch processing task when the target batch processing task has no downstream dependent tasks, and to mark the target batch processing task as the initial batch processing completed.

[0008] The judgment module is used to determine whether a data query request for the target batch task has been obtained;

[0009] The second data batch processing module is used to execute the preliminary batch processing to complete the corresponding target batch processing task through a preset cache queue when a data query request for the target batch processing task is obtained, so as to filter out the target data corresponding to the data query request from the corresponding batch processing data.

[0010] The third data batch processing module is used to process the target batch processing task corresponding to the initial batch processing completion based on a preset data batch processing time period when no data query request for the target batch processing task is obtained, so as to complete the batch processing process of the relevant data.

[0011] Optionally, the task determination interface includes:

[0012] The dependent task acquisition unit is used to acquire data on batch tasks with downstream dependent tasks through a preset human-computer interaction interface;

[0013] The list generation unit is used to generate a task list based on the data by running batch tasks, thereby obtaining a target dependent task list.

[0014] The task judgment unit is used to determine whether the target batch task exists in the target dependent task list, so as to determine whether the target batch task has downstream dependent tasks.

[0015] Optionally, the first data batch processing module includes:

[0016] The task identification unit is used to identify whether the target batch processing task is marked as a non-timely viewing type when the target batch processing task has no downstream dependent tasks; the non-timely viewing type is a task type marked through a preset interface;

[0017] The first data batch processing unit is used to perform the corresponding metadata batch processing process based on the target batch processing task when the target batch processing task is marked as not viewable in a timely manner, and to mark the target batch processing task as initially completed.

[0018] Optionally, the second data batch processing module includes:

[0019] The query request acquisition unit is used to acquire data query requests for the target batch processing task.

[0020] The request determination unit is used to determine whether the target batch processing task corresponding to the data query request has been marked as preliminary batch processing completion.

[0021] The second data batch processing unit is used to execute the target batch processing task through a preset cache queue when the target batch processing task is marked as having completed the initial batch processing, and obtain the corresponding first batch processing data.

[0022] The first data query unit is used to filter out the target data corresponding to the data query request from the first batch data.

[0023] Optionally, the system further includes:

[0024] The third data batch processing unit is used to execute the target batch processing task when the target batch processing task has downstream dependent tasks, to obtain the second batch processing data, so as to complete the batch processing process of the relevant data.

[0025] The fourth data batch processing unit is used to execute the target batch processing task when the target batch processing task has no downstream dependent tasks and the target batch processing task is not marked as a non-timely viewing type, to obtain the third batch processing data and complete the batch processing process of the relevant data.

[0026] Optionally, the system further includes:

[0027] The second data query unit is used to filter out the target data corresponding to the data query request from the second batch data or the third batch data.

[0028] Optionally, the third data batch processing module includes:

[0029] The task status identification unit is used to identify whether the target batch running task is currently marked as having completed the initial batch running process based on a preset data batch running time period, in the absence of a data query request for the target batch running task.

[0030] The fifth data batch processing unit is used to execute the target batch processing task when the target batch processing task is currently in the state marked as preliminary batch processing completed, so as to complete the batch processing of relevant data.

[0031] Secondly, this application provides a data batch processing method, including:

[0032] Determine if the current target batch processing task has any downstream dependent tasks;

[0033] When the target batch processing task has no downstream dependent tasks, the batch processing process based on the corresponding metadata is executed, and the target batch processing task is marked as the initial batch processing completed.

[0034] Determine whether a data query request for the target batch processing task has been obtained;

[0035] When a data query request for the target batch running task is obtained, the preliminary batch running task is executed through a preset cache queue to complete the target batch running task, so as to filter out the target data corresponding to the data query request from the corresponding batch running data.

[0036] If no data query request is obtained for the target batch processing task, the target batch processing task corresponding to the initial batch processing completion is processed based on the preset data batch processing time period to complete the batch processing of relevant data.

[0037] Thirdly, this application provides an electronic device, comprising:

[0038] Memory, used to store computer programs;

[0039] A processor for executing the computer program to implement the data batch processing method described above.

[0040] Fourthly, this application provides a computer-readable storage medium for storing a computer program, which, when executed by a processor, implements the data batch processing method described above.

[0041] Therefore, the task judgment interface in this application is used to determine whether the current target batch processing task has downstream dependent tasks; the first data batch processing module is used to execute the corresponding metadata batch processing process based on the target batch processing task when the target batch processing task has no downstream dependent tasks, and mark the target batch processing task as initially completed; the judgment module is used to determine whether a data query request for the target batch processing task has been obtained; the second data batch processing module is used to execute the target batch processing task corresponding to the initially completed batch processing task through a preset cache queue when a data query request for the target batch processing task has been obtained, so as to filter out the target data corresponding to the data query request from the corresponding batch processing data; the third data batch processing module is used to run the target batch processing task corresponding to the initially completed batch processing task based on a preset data batch processing time period when no data query request for the target batch processing task has been obtained, so as to complete the batch processing process of the relevant data. In this way, this application can perform metadata batching according to the actual needs of the target batching task. When a relevant data query request is received in the future, the target batching task can be executed through a preset cache queue. This can reduce resource contention during peak data batching periods and improve the execution efficiency of other batching tasks. Furthermore, based on the preset data batching time period, the target batching task can be run during relatively idle time, which can reasonably allocate batching queue resources and improve data batching efficiency. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0043] Figure 1 This is a schematic diagram of the data batch processing system structure disclosed in this application;

[0044] Figure 2 This is a flowchart of a data batch processing method disclosed in this application;

[0045] Figure 3 This is a flowchart of a specific data batch processing method disclosed in this application;

[0046] Figure 4 This is a flowchart of another specific data batch processing method disclosed in this application;

[0047] Figure 5 This application discloses a specific method for batch data querying;

[0048] Figure 6 This application discloses a specific data supplementation batch processing method flowchart;

[0049] Figure 7 This is a schematic diagram of data batch processing resource allocation disclosed in this application;

[0050] Figure 8 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] See Figure 1 As shown in the figure, an embodiment of the present invention discloses a data batch processing system, including:

[0053] Task Judgment Interface 11 is used to determine whether the current target batch task has downstream dependent tasks.

[0054] In this application, during data batch processing, a task judgment interface is used to determine whether the current target batch processing task has downstream tasks. It is understood that batch processing tasks with downstream tasks have higher data utilization rates, while batch processing tasks without downstream dependent tasks may not be used for a period of time. Therefore, this application can distinguish the current target batch processing task based on the presence or absence of downstream dependent tasks. In a specific embodiment, the task judgment interface may include: a dependent task acquisition unit, used to acquire data batch processing tasks with downstream dependent tasks through a preset human-computer interaction interface; a list generation unit, used to generate a task list based on the data batch processing tasks to obtain a target dependent task list; and a task judgment unit, used to determine whether the target batch processing task exists in the target dependent task list, thereby determining whether the target batch processing task has downstream dependent tasks. Specifically, the dependency task acquisition unit can obtain data batch processing tasks with downstream dependency tasks through a preset human-computer interaction interface. Then, the list generation unit can integrate these data batch processing tasks with downstream dependency tasks into a task list, which serves as the target dependency task list. When a data batch processing task needs to be executed later, the task judgment unit can determine whether the target batch processing task to be executed exists in the aforementioned target dependency task list. If it exists, the target batch processing task has downstream dependency tasks, indicating that the relevant batch processing data has a timely need and needs to be completed promptly. Conversely, if the target batch processing task has no downstream dependency, it means that the relevant batch processing data does not have a timely need.

[0055] The first data batch processing module 12 is used to perform a batch processing process based on the corresponding metadata of the target batch processing task when the target batch processing task has no downstream dependent tasks, and to mark the target batch processing task as having completed the initial batch processing.

[0056] Correspondingly, when the target batch processing task has no downstream dependent tasks, the first data batch processing module can execute the batch processing process of the metadata corresponding to the target batch processing task and mark the target batch processing task as initially completed. It is understood that when the target batch processing task has no downstream dependent tasks, it indicates that the demand for the batch processing data corresponding to the target batch processing task is not high, or its importance is not high. In this case, only the metadata of the target batch processing task can be batch processed, reserving the saved batch processing queue resources for other batch processing tasks. In a specific embodiment, the first data batch processing module may include: a task identification unit, used to identify whether the target batch processing task is marked as a non-timely viewing type when the target batch processing task has no downstream dependent tasks; the non-timely viewing type is a task type marked through a preset interface; and a first data batch processing unit, used to execute the batch processing process of the corresponding metadata based on the target batch processing task when the target batch processing task is marked as a non-timely viewing type, and mark the target batch processing task as initially completed. Specifically, even if a data batch processing task has no downstream dependent tasks, it may still have high usage requirements, such as a high demand for timely feedback of query results. In this case, the corresponding batch processing task can be marked as an "immediate viewing" type, indicating that it needs to be processed promptly to meet the feedback time requirement. Conversely, batch processing tasks with low usage requirements can be marked as "non-immediate viewing" types. Staff can mark data batch processing tasks with low usage requirements through a preset interface. Then, the task identification unit can identify the type of the current target batch processing task. If the target batch processing task is marked as a "non-immediate viewing" type, the first data batch processing unit can execute the metadata batch processing process corresponding to the target batch processing task, and simultaneously mark the target batch processing task that has completed the metadata batch processing process as having completed the initial batch processing.

[0057] The judgment module 13 is used to determine whether a data query request for the target batch task has been obtained.

[0058] It should be noted that the data obtained after batch processing may not be used for querying. Since this application includes data batch processing tasks that only perform metadata batch processing, it is necessary to use a judgment module to determine whether a data query request has been received for these batch processing tasks marked as having completed preliminary batch processing (corresponding to the data that needs to be batch processed).

[0059] The second data batch processing module 14 is used to execute the preliminary batch processing to complete the corresponding target batch processing task through a preset cache queue when a data query request for the target batch processing task is obtained, so as to filter out the target data corresponding to the data query request from the corresponding batch processing data.

[0060] In this application, when a data query request for the aforementioned target batch processing task is received, since the target batch processing task has not been executed and only the related metadata has been batch processed (marked as preliminary batch processing complete), it is necessary to perform actual batch processing on the target batch processing task through a preset cache queue to obtain actual batch processing data. Then, the corresponding target data can be filtered out from the batch processing data according to the data query request. In a specific embodiment, the second data batch processing module may include: a query request acquisition unit, used to acquire a data query request for the target batch processing task; a request judgment unit, used to determine whether the target batch processing task corresponding to the data query request has been marked as preliminary batch processing complete; a second data batch processing unit, used to execute the target batch processing task through a preset cache queue when the target batch processing task is marked as preliminary batch processing complete, to obtain the corresponding first batch processing data; and a first data query unit, used to filter out the target data corresponding to the data query request from the first batch processing data. Specifically, the query request acquisition unit can obtain data query requests for the target batch processing task, and then the request judgment unit can determine whether the target batch processing task corresponding to the data query request has been marked as initially completed. It's understandable that within a workday, there may be multiple query requests for a portion of data. When the first data query is performed on a target batch processing task marked as initially completed (the corresponding data that needs to be batch processed), the target batch processing task can be executed through a preset cache queue, obtaining the first batch of data corresponding to that target batch processing task, thus completing the data batch processing process. It's understood that this preset cache queue can be a separate high-speed cache queue, only executing target batch processing tasks marked as initially completed. This allows for timely execution of the corresponding target batch processing task upon receiving a data query request, obtaining the target data corresponding to the query request.

[0061] In one specific embodiment, it may further include: a third data batching unit, used to execute the target batching task when the target batching task has downstream dependent tasks, to obtain second batching data, thereby completing the batching process of the relevant data; and a fourth data batching unit, used to execute the target batching task when the target batching task does not have downstream dependent tasks and the target batching task is not marked as a non-timely viewing type, to obtain third batching data, thereby completing the batching process of the relevant data. Specifically, during data batching, if the current target batching task has downstream dependent tasks, the third data batching unit can directly execute the target batching task to obtain second batching data, thus completing the batching process of the target batching task. Correspondingly, if the current target batching task does not have downstream dependent tasks, but the target batching task is not marked as a non-timely viewing type or is marked as a timely viewing type, the fourth data batching unit can execute the target batching task to obtain third batching data, thus completing the batching process of the target batching task. It is understood that, in another specific embodiment, a second data query unit may also be included, configured to filter out target data corresponding to the data query request from the second batch data or the third batch data. Specifically, when a subsequent data query request is made for these second batch data or the third batch data, the corresponding target data can be directly filtered out.

[0062] The third data batch processing module 15 is used to process the target batch processing task corresponding to the initial batch processing completion based on a preset data batch processing time period when no data query request for the target batch processing task is obtained, so as to complete the batch processing process of the relevant data.

[0063] In this application, it is understood that for data batching tasks marked as initially completed in the above steps and not used (queried) within a preset data batching time period, a third data batching module needs to perform batching uniformly at the end of the period (or during idle time) to ensure the continuity between all batching data. In a specific embodiment, the third data batching module may include: a task status identification unit, used to identify whether the target batching task is currently marked as initially completed based on a preset data batching time period when no data query request for the target batching task is obtained; and a fifth data batching unit, used to execute the target batching task when the target batching task is currently marked as initially completed to complete the batching process of the relevant data. Specifically, when no data query request for the target batching task is obtained, the task status identification unit can identify all data batching tasks currently in the initially completed state based on the preset data batching time period, and then the fifth data batching unit can execute these data batching tasks to complete the batching process of the relevant data.

[0064] Therefore, this application can perform only the corresponding metadata processing for the current target batch processing task with downstream dependent tasks, saving queue resources during peak batch processing periods. Furthermore, when the batch processing data corresponding to the target batch processing task is needed, it can be processed in a timely manner through a preset cache queue without affecting the normal use of the batch processing data. Subsequently, according to the preset data batch processing time cycle, the target batch processing task marked as having completed the initial batch processing can be executed during idle time, ensuring data continuity.

[0065] like Figure 2 As shown in the figure, this application discloses a data batch processing method, including:

[0066] Step S11: Determine whether the current target batch processing task has downstream dependent tasks.

[0067] In this application, when performing batch data processing, it is necessary to distinguish the actual usage requirements of different batch processing tasks. The need to determine whether the current target batch processing task needs to be processed promptly can be based on whether there are downstream dependent tasks. In other words, since the results of many batch data processing tasks may not be used on the same day—for example, for reporting requirements—if it falls on a weekend, there may be no business activity to view the data. Furthermore, the timeliness of query results is not a high requirement when such data is needed. Therefore, it is possible to distinguish the actual usage requirements of different batch data processing tasks.

[0068] Step S12: When the target batch processing task has no downstream dependent tasks, execute the batch processing process of the corresponding metadata based on the target batch processing task, and mark the target batch processing task as the initial batch processing completed.

[0069] In this application, when the current target batch processing task has no downstream dependent tasks, the current target batch processing task is less important than data batch processing tasks that have downstream dependent tasks. Therefore, during peak batch processing periods, only the metadata corresponding to the target batch processing task can be batch processed, and the target batch processing task can be marked as initially completed.

[0070] Step S13: Determine whether a data query request for the target batch task has been obtained.

[0071] Furthermore, in terms of the business use of batch processing data, in most cases, the data corresponding to the target batch processing task marked as the initial batch processing completed after processing according to the aforementioned steps will not be viewed; correspondingly, there may be data query requests for the data corresponding to the aforementioned target batch processing task. In this case, it is possible to determine at any time whether the obtained data query request is for the target batch processing task marked as the initial query request.

[0072] Step S14: When a data query request for the target batch running task is obtained, the target batch running task corresponding to the preliminary batch running is executed through a preset cache queue, so as to filter out the target data corresponding to the data query request from the corresponding batch running data.

[0073] In this application, if a target batch running task marked as having completed preliminary batch running is obtained, since the target batch running task has only completed the batch running process of metadata and has not carried out the batch running of real data, the target batch running task can be executed through a preset cache queue to complete the batch running process of relevant data and obtain the corresponding batch running data. Then, based on the data query request, the corresponding target data can be filtered out from the batch running data.

[0074] Step S15: If no data query request for the target batch running task is obtained, the target batch running task corresponding to the initial batch running completion is run based on the preset data batch running time period to complete the batch running process of the relevant data.

[0075] Furthermore, if the business use of the batch data does not involve the use of those batch data tasks marked as initially completed, that is, if no data query request for the target batch data task is obtained, then the batch data tasks marked as initially completed can be batch-run with real data according to the preset batch data running time period to complete the batch running process of the relevant tasks.

[0076] In one specific embodiment, such as Figure 3As shown, when an upstream dependent task is completed and the batch processing condition for the target task is triggered, it first determines whether the target task has a downstream dependent task. If so, it directly performs batch processing on the actual data for the target task, completing the batch processing task directly. Conversely, if the target task has no downstream dependent tasks, it can perform "lazy" batch processing (batch processing only on metadata) and record it as "lazy" batch processing (adding a preliminary batch processing completion marker). Then, if a data query request for the target task is received before 6 PM that day (based on a preset data batch processing time period), indicating whether the data corresponding to the target task needs to be batch processed, the batch processing status query interface can be triggered. If the target task is in a "lazy" state, it performs batch processing on the actual data for the target task (through a preset cache queue) and returns the query data corresponding to the data query request based on the batch processing data. Correspondingly, if the data required for batch processing corresponding to the target task is not used before 6 PM on the same day, a supplementary batch processing can be performed at 6 PM. That is, the target task marked as having completed preliminary batch processing is processed with real data to supplement the relevant data that needs to be processed. It should be noted that the efficiency of batch processing the target task through the preset cache queue (a resource queue that is always idle) should be sufficient to meet business requirements.

[0077] In another specific embodiment, such as Figure 4 As shown, if the current batch processing time and dependency conditions (upstream dependent tasks) of the target task are met, it is determined whether the target task has downstream dependent tasks. If so, the batch processing of real data is performed directly. Correspondingly, if the target task does not have downstream dependent tasks, it is further determined whether the target task is marked as a non-timely viewing type. If not, the batch processing of real data is performed directly. If so, the target task is "lazy" batch processing is performed, and only the relevant metadata is batch processed, skipping the batch processing of real data.

[0078] In yet another specific embodiment, such as Figure 5 As shown, this application uses UDFs (User-defined functions) to encapsulate the call to the batch query interface. When querying data using SQL (Structured Query Language), the batch status query is triggered through the UDF. First, it is determined whether the target task corresponding to the data to be queried is in a "lazy" batch running state. If not, the data result corresponding to the current query request is returned based on the current result set (which stores all batch-run data). Conversely, if the target task corresponding to the data to be queried is in a "lazy" batch running state, the task is submitted to a cache queue (preset cache queue) for data batch running, and a prompt is displayed to wait for the query result. Then, the data that has been batch-run is written to the result set to filter out the data result corresponding to the current query request from the result set.

[0079] Furthermore, such as Figure 6 As shown, this application can supplement the data batching task according to the preset data batching time cycle. Specifically, at 6 pm every day, all batching tasks can be traversed to check if there are any "lazy" batching jobs. If so, it can be further determined whether the actual batching process of the batching task has been completed before. If not, it means that the batching task at the current time point has only completed the metadata batching and has not carried out the actual data batching process. At this time, the task needs to be batched during the idle time to supplement all the data of the data batching task.

[0080] like Figure 7 As shown, this application arranges the batch processing queue resources and processing time, executing data batch processing tasks at night. This implies that some of these tasks are "lazy" batch processing tasks. During the day, high-speed resources can be reserved for timely batch processing of these "lazy" tasks (when there is a query requirement). Then, during the evening's idle time, data can be supplemented and batch processed for the "lazy" tasks (those without query requirements during the day). This approach rationally utilizes data batch processing queue resources and improves data batch processing efficiency.

[0081] Therefore, this application can perform "lazy" batch processing or real data batch processing according to different needs of data batch processing tasks. This can reduce the pressure of resource queue contention during peak data batch processing periods. Subsequently, the preset cache queue can be used to perform real data batch processing on the "lazy" batch processing tasks without affecting the normal use of batch processing data. This can make reasonable use of the data batch processing queue resources and improve the efficiency of data batch processing without affecting the use of batch processing data.

[0082] Furthermore, embodiments of this application also disclose an electronic device, Figure 8 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0083] Figure 8 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the data batch processing method disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0084] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0085] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0086] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the data batch processing method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0087] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned data batch processing method. The specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0088] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0089] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0090] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0091] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0092] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data batch processing system, characterized in that, include: The task judgment interface is used to determine whether the current target batch task has downstream dependent tasks; The first data batch processing module is used to perform the corresponding metadata batch processing process based on the target batch processing task when the target batch processing task has no downstream dependent tasks, and to mark the target batch processing task as the initial batch processing completed. The judgment module is used to determine whether a data query request for the target batch task has been obtained; The second data batch processing module is used to execute the preliminary batch processing to complete the corresponding target batch processing task through a preset cache queue when a data query request for the target batch processing task is obtained, so as to filter out the target data corresponding to the data query request from the corresponding batch processing data. The third data batch processing module is used to process the target batch processing task corresponding to the initial batch processing completion based on a preset data batch processing time period when no data query request for the target batch processing task is obtained, so as to complete the batch processing process of the relevant data.

2. The data batch processing system according to claim 1, characterized in that, The task judgment interface includes: The dependent task acquisition unit is used to acquire data on batch tasks with downstream dependent tasks through a preset human-computer interaction interface; The list generation unit is used to generate a task list based on the data by running batch tasks, thereby obtaining a target dependent task list. The task judgment unit is used to determine whether the target batch task exists in the target dependent task list, so as to determine whether the target batch task has downstream dependent tasks.

3. The data batch processing system according to claim 1, characterized in that, The first data batch processing module includes: The task identification unit is used to identify whether the target batch processing task is marked as a non-timely viewing type when the target batch processing task has no downstream dependent tasks; the non-timely viewing type is a task type marked through a preset interface; The first data batch processing unit is used to perform the corresponding metadata batch processing process based on the target batch processing task when the target batch processing task is marked as non-timely viewing type, and to mark the target batch processing task as preliminary batch processing completed.

4. The data batch processing system according to claim 1, characterized in that, The second data batch processing module includes: The query request acquisition unit is used to acquire data query requests for the target batch processing task. The request determination unit is used to determine whether the target batch processing task corresponding to the data query request has been marked as preliminary batch processing completion. The second data batch processing unit is used to execute the target batch processing task through a preset cache queue when the target batch processing task is marked as having completed the initial batch processing, and obtain the corresponding first batch processing data. The first data query unit is used to filter out the target data corresponding to the data query request from the first batch data.

5. The data batch processing system according to claim 3, characterized in that, Also includes: The third data batch processing unit is used to execute the target batch processing task when the target batch processing task has downstream dependent tasks, to obtain the second batch processing data, so as to complete the batch processing process of the relevant data. The fourth data batch processing unit is used to execute the target batch processing task when the target batch processing task has no downstream dependent tasks and the target batch processing task is not marked as a non-timely viewing type, to obtain the third batch processing data and complete the batch processing process of the relevant data.

6. The data batch processing system according to claim 5, characterized in that, Also includes: The second data query unit is used to filter out the target data corresponding to the data query request from the second batch data or the third batch data.

7. The data batch processing system according to any one of claims 1 to 6, characterized in that, The third data batch processing module includes: The task status identification unit is used to identify whether the target batch running task is currently marked as having completed the initial batch running process based on a preset data batch running time period, in the absence of a data query request for the target batch running task. The fifth data batch processing unit is used to execute the target batch processing task when the target batch processing task is currently in the state marked as preliminary batch processing completed, so as to complete the batch processing of relevant data.

8. A data batch processing method, characterized in that, include: Determine if the current target batch processing task has any downstream dependent tasks; When the target batch processing task has no downstream dependent tasks, the batch processing process based on the corresponding metadata is executed, and the target batch processing task is marked as the initial batch processing completed. Determine whether a data query request for the target batch processing task has been obtained; When a data query request for the target batch running task is obtained, the preliminary batch running task is executed through a preset cache queue to complete the target batch running task, so as to filter out the target data corresponding to the data query request from the corresponding batch running data. If no data query request is obtained for the target batch processing task, the target batch processing task corresponding to the initial batch processing completion is processed based on the preset data batch processing time period to complete the batch processing of relevant data.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the data batch processing method as described in claim 8.

10. A computer-readable storage medium, characterized in that, Used to store computer programs, which, when executed by a processor, implement the data batch processing method as described in claim 8.

Images