A method and system for batch processing and monitoring of large files

Abstract

This invention discloses a method and system for batch processing and monitoring large files, relating to the field of computer technology, and particularly to a method and system for batch processing and monitoring large files. The method includes: sending a file to a batch processor via a file transfer platform to obtain a first sent file; triggering a first creation instruction when the file arrives; creating a first batch processing task execution graph through task relationship analysis; obtaining a first distributed execution task, wherein the first distributed execution task includes multiple subtasks; obtaining a first invocation instruction; invoking the logical module service configured corresponding to the first distributed execution task for parallel task execution according to a distributed scheduling framework and the first invocation instruction; and obtaining the first file batch processing result by managing the task status through a callback scheduling center when the task execution is completed. This solves the technical problems of long processing times and the inability to promptly identify and address anomalies in existing technologies for batch processing of large files.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method and system for batch processing and monitoring large files. Background Technology

[0002] Batch processing of large files is a common technical challenge for many enterprises in daily operations. Many systems frequently require batch processing to handle large amounts of data with relatively low real-time requirements, thereby completing corresponding business processing and archiving tasks. For example, banks process and archive historical customer transaction data. Current batch processing technologies are complex and time-consuming, especially when dealing with complex business processes and massive amounts of data. The timeliness issue of batch processing becomes even more prominent, potentially impacting daily business operations. Furthermore, the lack of a complete and standardized batch processing solution often results in fragmented, uncontrollable, and inefficient designs, particularly for processing extremely large files. Excessive server load often leads to long total batch processing times and potential system resource risks. When anomalies occur during batch processing, it's difficult to readily rerun the process to resolve the issues, thus affecting normal business operations. Therefore, researching how to optimize batch processing design, scheduling, and task management is crucial for improving the efficiency of large file batch processing and enhancing overall work quality.

[0003] However, the existing technology for batch processing large files is time-consuming, resulting in low work efficiency. At the same time, the processing of large files increases the risk to system resources. The existing technology cannot identify and deal with anomalies in the processing in a timely manner, resulting in a lack of flexibility. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for batch processing and monitoring of large files, in order to solve the problems of long processing time and low work efficiency in the prior art for batch processing of large files, the increased risk to system resources due to the large file data processing, and the inability of the prior art to identify and deal with anomalies in the processing in a timely manner, resulting in insufficient flexibility.

[0005] In view of the above problems, the present invention provides a method and system for batch processing and monitoring of large files.

[0006] In a first aspect, the present invention provides a method for batch processing and monitoring large files. The method is implemented through a system for batch processing and monitoring large files, comprising: sending a file to a batch processor via a file transfer platform to obtain a first sent file; monitoring the directory of the first sent file and triggering a first creation instruction when the file arrives; performing task relationship analysis on the first sent file according to the first creation instruction to create a first batch processing task execution graph; performing distributed processing based on the first batch processing task execution graph to obtain a first distributed execution task, wherein the first distributed execution task includes multiple subtasks; obtaining a first invocation instruction based on the first distributed execution task; invoking the logical module service configured corresponding to the first distributed execution task for parallel task execution according to a distributed scheduling framework and the first invocation instruction; and obtaining the first file batch processing result by managing the task status through a callback scheduling center when the tasks in the first distributed execution task are completed.

[0007] On the other hand, the present invention also provides a system for batch processing and monitoring large files, used to execute a method for batch processing and monitoring large files as described in the first aspect, wherein the system includes: a first obtaining unit: the first obtaining unit is used to send a file to a batch processor through a file transfer platform to obtain a first sent file; a first triggering unit: the first triggering unit monitors the directory of the first sent file and triggers a first creation instruction when the file arrives; a first creation unit: the first creation unit performs task relationship analysis on the first sent file according to the first creation instruction to create a first batch processing task execution graph; a second obtaining unit: the second obtaining unit performs distributed processing based on the first batch processing task execution graph to obtain a first distributed execution task, wherein the first distributed execution task includes multiple sub-tasks; a third obtaining unit: the third obtaining unit obtains a first calling instruction according to the first distributed execution task; a first execution unit: the first execution unit calls the logical module service configured corresponding to the first distributed execution task for parallel task execution according to the distributed scheduling framework and the first calling instruction; a fourth obtaining unit: when the task in the first distributed execution task is completed, the fourth obtaining unit manages the task status through a callback scheduling center to obtain the first file batch processing result.

[0008] Thirdly, the present invention also provides a system for batch processing and monitoring of large files, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method described in the first aspect above.

[0009] Fourthly, an electronic device, comprising a processor and a memory;

[0010] This memory is used for storage;

[0011] The processor is used to execute the method described in any one of the first aspects above by means of a call.

[0012] Fifthly, a computer program product comprising a computer program and / or instructions that, when executed by a processor, implement the steps of the method described in any of the first aspects above.

[0013] One or more technical solutions provided in this invention have at least the following technical effects or advantages:

[0014] 1. Through a file transfer platform, files to be processed are sent to the batch processor. Simultaneously, the system monitors the transmission status of these files in real time. When a file arrives at the batch processor, an instruction is automatically triggered to perform task relationship analysis on that file, resulting in the first batch of processing task execution graphs. Based on this graph, the system analyzes and identifies the first distributed execution task, which contains multiple sub-tasks. The corresponding logic module service is then invoked, performing a series of processing steps on the first distributed execution task, including splitting, calculation, merging, reconciliation, and end-of-day processing. Finally, a callback to the scheduling center service is made to obtain the status information of each task, thus determining the processing result for the corresponding file. By employing distributed computing principles, large pipeline files from the upstream system are split into smaller files. Leveraging the advantages of distributed system microservices, each smaller file is processed independently and in parallel. Finally, the processing results of the smaller files are aggregated to obtain the processing result for the entire large pipeline file. This achieves the technical goal of improving the efficiency of large file batch processing, enhancing both the efficiency of large file processing and the flexibility of the processing system.

[0015] 2. Through the abnormal data detection model, intelligent detection of real-time monitoring data is realized, and then based on the detected abnormal data, corresponding early warnings and reminders are issued intelligently, achieving the technical effect of intelligent early warning and abnormal handling.

[0016] 3. By monitoring the entire batch processing process in real time, timely location of anomalies is achieved, and reprocessing is performed based on the located anomalies. This achieves the technical effect of flexibly reprocessing anomalies and thus improving file processing efficiency.

[0017] 4. By calling the system callback center, the status of each task is changed in real time, and new tasks are created, thereby achieving the goal of intelligent batch processing of large files and obtaining batch processing results.

[0018] 5. Through the system of batch processing and monitoring of large files, larger data volumes of files can be processed in batches, thereby saving historical data of files. At the same time, the query response time is short, and the inventory history details will not affect the query performance, thus achieving the technical effect of effectively improving the user experience.

[0019] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating a method for batch processing and monitoring large files according to the present invention.

[0022] Figure 2 This is a flowchart illustrating the process of using a visual batch processing monitoring platform to issue an alarm for the first early warning information in a method for batch processing and monitoring large files according to the present invention.

[0023] Figure 3 This is a flowchart illustrating the process of rerunning the first abnormal job according to the first rerun instruction in a method for batch processing and monitoring large files according to the present invention.

[0024] Figure 4 This is a flowchart illustrating the process of generating the first file batch processing result in a method for batch processing and monitoring large files according to the present invention.

[0025] Figure 5 This is a schematic diagram of the structure of a system for batch processing and monitoring of large files according to the present invention;

[0026] Figure 6 This is a schematic diagram of the structure of an exemplary electronic device of the present invention.

[0027] Explanation of reference numerals in the attached figures:

[0028] First acquisition unit 11, first trigger unit 12, first creation unit 13, second acquisition unit 14, third acquisition unit 15, first execution unit 16, fourth acquisition unit 17, bus 300, receiver 301, processor 302, transmitter 303, memory 304, bus interface 305. Detailed Implementation

[0029] This invention provides a method and system for batch processing and monitoring large files. Existing technologies for batch processing large files are time-consuming, leading to low efficiency. Furthermore, large file data processing increases system resource risks. This invention addresses the technical problem of existing technologies failing to promptly identify and address anomalies during processing, resulting in insufficient flexibility. By employing distributed computing principles, large files are broken down into smaller files. The advantages of distributed system microservices are leveraged to process each smaller file independently and in parallel. Finally, the processing results of the smaller files are aggregated to obtain the processing result of the entire large file, achieving the technical goal of improving the efficiency of large file batch processing. Through distributed principles, this invention achieves the technical effect of improving the efficiency of large file processing while simultaneously enhancing the flexibility of the processing system.

[0030] The acquisition, storage, use, and processing of data in the technical solution of this invention all comply with the relevant provisions of national laws and regulations.

[0031] The technical solutions of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. It should be understood that the present invention is not limited to the exemplary embodiments described herein. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention. It should also be noted that, for ease of description, only the parts related to the present invention are shown in the accompanying drawings, not all of them.

[0032] This invention provides a method for batch processing and monitoring large files. The method is applied to a system for batch processing and monitoring large files. The method includes: sending a file to a batch processor via a file transfer platform to obtain a first sent file; monitoring the directory of the first sent file and triggering a first creation instruction when the file arrives; performing task relationship analysis on the first sent file according to the first creation instruction to create a first batch processing task execution graph; performing distributed processing based on the first batch processing task execution graph to obtain a first distributed execution task, wherein the first distributed execution task includes multiple subtasks; obtaining a first invocation instruction based on the first distributed execution task; invoking the logical module service configured corresponding to the first distributed execution task for parallel task execution according to a distributed scheduling framework and the first invocation instruction; and obtaining the first file batch processing result by managing the task status through a callback scheduling center when the tasks in the first distributed execution task are completed.

[0033] After introducing the basic principles of the present invention, various non-limiting embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0034] Example 1

[0035] Please see the appendix Figure 1 This invention provides a method for batch processing and monitoring large files, wherein the method is applied to a system for batch processing and monitoring large files, and the method specifically includes the following steps:

[0036] Step S100: Send the file to the batch processor via the file transfer platform to obtain the first sent file;

[0037] Specifically, the method for batch processing and monitoring large files is applied to the system for batch processing and monitoring large files. It utilizes distributed computing principles to split large files into smaller files, leveraging the advantages of microservices in a distributed system to process each smaller file independently and in parallel. Finally, the processing results of the smaller files are aggregated to obtain the processing result of the entire large file. The file transfer platform connects the batch processor in the system to the upstream system. The upstream system then transmits the large files to be batch processed to the batch processor in the system via the file transfer platform. The large file to be batch processed is the first sent file. The batch processor is the server in the system used for batch processing large files. By obtaining the first sent file, the technical effect of acquiring information about the large files to be batch processed is achieved, providing a file foundation for subsequent system batch processing.

[0038] Step S200: By monitoring the directory of the first distributed file, when the file arrives, a first creation instruction is triggered;

[0039] Specifically, when the upstream system transmits the first distributed file to the batch processor in the large file batch processing and monitoring system via a file transfer platform, the directory monitoring service in the large file batch processing and monitoring system monitors the arrival time and transmission status of the first distributed file in real time. Once the successful transmission of the first distributed file is detected, the first distributed file is moved to the executable directory, and the task management service in the large file batch processing and monitoring system is invoked to create the instruction for the processing task corresponding to the first distributed file, i.e., the first creation instruction. By automatically monitoring the transmission process of the first distributed file and automatically triggering the task creation instruction for processing the first distributed file after successful transmission, the technical effect of automatically creating batch processing task instructions is achieved.

[0040] Step S300: Perform task relationship analysis on the first issued file according to the first creation instruction, and create the first batch of processing task execution diagrams;

[0041] Specifically, based on the first creation instruction automatically triggered by the system for large file batch processing and monitoring, the task management service automatically creates a task to process the first issued file. Further, task relationship analysis is performed on the first issued file, including the correspondence between task types and task chains, thereby obtaining the first batch processing task execution graph corresponding to the first issued file. Here, the task chain is the complete task process of batch processing the first issued file. By obtaining the first batch processing task execution graph, the technical goal of visualizing the complete processing process of the first issued file is achieved, realizing the technical effect of real-time monitoring of the batch processing progress of the first issued file.

[0042] Step S400: Based on the first batch of processing task run graphs, perform distributed processing to obtain a first distributed execution task, wherein the first distributed execution task includes multiple sub-tasks;

[0043] Specifically, based on the concept of distributed computing, the execution graph of the first batch of processing tasks corresponding to the first issued file is distributed across various processing stages to obtain a first distributed execution task. This first distributed execution task is any one of the execution tasks in the execution graph of the first batch of processing tasks, and it includes multiple sub-tasks. By processing the execution graph of the first batch of processing tasks corresponding to the first issued file, the execution task information for each step of the first issued file batch processing is obtained. This achieves the technical effect of refining the file batch processing process, distributing the entire processing task, and thus improving the overall batch processing efficiency of the file.

[0044] Step S500: Execute the task according to the first distribution to obtain the first invocation instruction;

[0045] Step S600: Based on the distributed scheduling framework and the first invocation instruction, invoke the logical module service configured corresponding to the first distributed execution task to perform parallel task execution;

[0046] Specifically, based on the first distributed execution task, the system for large file batch processing and monitoring automatically issues a corresponding first invocation instruction. This first invocation instruction is used to invoke the logical module service configured corresponding to the first distributed execution task. In other words, based on the first batch of processing task execution graphs corresponding to the first issued file, the distributed scheduling framework periodically retrieves tasks to be executed and invokes the corresponding logical module service. Furthermore, through the logical module service, the various execution tasks corresponding to the first issued file are processed and executed in parallel. The logical module service includes modules for splitting, calculating, merging, reconciliation, and end-of-day processing, etc. For example, a bank batch-processes all user transfer data, including splitting transfer time and transfer amount, calculating the total number of transfers and total transfer amount within a specific time period. Through the logical module service, the parallel processing of each sub-task in the first distributed execution task corresponding to the first issued file is achieved, thereby improving the efficiency of processing the first issued file.

[0047] Step S700: When the task in the first distributed execution task is completed, the task status is managed by calling back the scheduling center to obtain the batch processing result of the first file.

[0048] Specifically, by employing the concept of distributed computing, the goal of parallel processing of multiple subtasks within the first distributed execution task is achieved. Furthermore, by invoking the scheduling center service in the large file batch processing and monitoring system, the completion status of each subtask is checked. Based on the completion status of each subtask, the completion status of the first distributed execution task can be determined. Consequently, based on the completion status of all distributed execution tasks, the batch processing status of the first distributed file can be determined, thus obtaining the corresponding batch processing result for the first file. Through the concept of distributed computing, the large pipeline file from the upstream system is split into smaller files. Leveraging the advantages of microservices in the distributed system, each smaller file is processed independently and in parallel. Finally, by aggregating the processing results of the smaller files, the processing result of the entire large pipeline file is obtained. This achieves the technical goal of improving the efficiency of large file batch processing, thereby enhancing both the efficiency of large file processing and the flexibility of the processing system.

[0049] Furthermore, step S300 of the present invention also includes:

[0050] Step S310: Obtain file type information by performing file type analysis on all files in the first issued file;

[0051] Step S320: Obtain task chain information by performing task chain relationship analysis on all files in the first issued file;

[0052] Step S330: Create the first batch of processing task execution graphs based on the file type information and the task chain information.

[0053] Specifically, based on the first issued file, the file types corresponding to each file in the first issued file are analyzed to obtain the file type information. Further, a task chain relationship analysis is performed on all files in the first issued file, that is, the order in which tasks are processed during the first issued file processing, thereby obtaining the task chain information. Finally, by comprehensively analyzing the file type information and the task chain information, the first batch processing task execution diagram corresponding to the first issued file is obtained. Based on the first batch processing task execution diagram, the order of each batch processing task in the batch processing of the first issued file, as well as the file type information processed by each batch processing task, can be clearly defined, thereby visualizing the batch processing process of the first issued file and providing a foundation for subsequent orderly system processing. This achieves the technical effect of real-time monitoring of batch processing scheduling tasks and job execution details, and real-time monitoring of the system's batch processing task execution status based on a visualized page.

[0054] Furthermore, step S600 of the present invention also includes:

[0055] Step S610: According to the first configuration instruction, configure the task module for the first distributed execution task to obtain a task configuration module, wherein the task configuration module includes multiple task modules;

[0056] Step S620: Configure the parameters of the task configuration module according to the second configuration instruction to obtain the module configuration parameters;

[0057] Step S630: Execute the first distributed execution task according to the task configuration module and the module configuration parameters.

[0058] Furthermore, step S630 of the present invention also includes:

[0059] The distributed scheduling framework processes all subtasks of the first distributed execution task periodically.

[0060] Specifically, the first configuration instruction is used to configure task modules for the first distributed execution task, thereby obtaining the task configuration module. The task configuration module includes multiple task modules, each corresponding to a subtask in the first distributed execution task. The second configuration instruction is used to configure parameters for the task configuration module, thereby obtaining the parameters of the task configuration module, i.e., the module configuration parameters. Finally, based on the task configuration module and the module configuration parameters, each subtask in the first distributed execution task is processed in parallel. Furthermore, the distributed scheduling framework is configured with a self-developed resource scheduling algorithm, which can be deployed in a cluster and supports dynamic scaling.

[0061] By leveraging the advantages of microservices in distributed systems, each subtask in the first distributed execution task is processed independently, thereby saving processing time for the first distributed execution task and achieving the technical effect of improving the efficiency of large file batch processing.

[0062] Further details are attached. Figure 2 As shown, the present invention further includes step S800:

[0063] Step S810: Perform batch monitoring of the first issued file through a visual interface to obtain real-time monitoring data;

[0064] Step S820: Input the real-time monitoring data into the abnormal data detection model, perform abnormal detection according to the abnormal data detection model, and obtain the first abnormal data;

[0065] Step S830: Generate a first warning message based on the first abnormal data;

[0066] Step S840: Send the first warning information to an alarm through the visual batch processing monitoring platform.

[0067] Specifically, the system for batch processing and monitoring large files includes a monitoring platform service. Based on the visual interface of this monitoring platform service, batch processing monitoring of the first distributed file can be performed, obtaining real-time monitoring data of the batch processing process. Furthermore, the real-time monitoring data obtained by the monitoring platform service is used as input information, and an abnormal data detection model is used for intelligent analysis to identify abnormal monitoring data, i.e., the first abnormal data. The abnormal data detection model is a neural network model trained on a large amount of batch processing monitoring data, capable of intelligently identifying abnormal data information during batch processing. Finally, based on the first abnormal data, the system automatically generates a corresponding first warning message and sends this warning message through the visual batch processing monitoring platform, thus achieving the technical effect of intelligent anomaly warning.

[0068] Further details are attached. Figure 3 As shown, step S820 of the present invention further includes:

[0069] Step S821: Obtain first real-time location information by performing anomaly location on the first abnormal data;

[0070] Step S822: Generate a first rerun instruction based on the first real-time positioning information;

[0071] Step S823: Reprocess the first abnormal data according to the first rerun instruction.

[0072] Specifically, after the abnormal data detection model intelligently analyzes and obtains the first abnormal data, the large file batch processing and monitoring system automatically locates the abnormal data based on the first abnormal data, thereby obtaining the location information of the job that handles the abnormality, i.e., the first real-time location information. Further, based on the first real-time location information, the large file batch processing and monitoring system issues a first rerun instruction to reprocess the processing job under the first real-time location. Finally, the first abnormal data is reprocessed based on the first rerun instruction. Through real-time monitoring of the entire batch processing process, timely location of abnormalities is achieved, and rerun processing is performed based on the located abnormalities, achieving the technical effect of flexible rerunning for abnormalities and thus improving file processing efficiency.

[0073] Further details are attached. Figure 4 As shown, step S700 of the present invention further includes:

[0074] Step S710: After each subtask of the first distributed execution task is completed, the task status is changed by calling back the scheduling center to manage the task status, and the first real-time task status is obtained.

[0075] Step S720: Based on the first real-time task state, obtain the parallel real-time task state of the first distributed execution task;

[0076] Step S730: Summarize the file processing based on the status of the parallel real-time tasks to generate the first file batch processing result.

[0077] Specifically, after all subtasks in the first distributed execution task have been completed, the system automatically updates the task status information of each subtask in the scheduling center, thereby forming the first real-time task status corresponding to the first distributed execution task. The scheduling center manages each task in the file batch processing, configuring and processing tasks based on real-time queries that meet the criteria. Furthermore, based on the first real-time task status, i.e., the execution status of each subtask, the parallel real-time task status of the first distributed execution task can be determined. Finally, based on the parallel real-time task status of the first distributed execution task, the processed files corresponding to each subtask in the first distributed execution task are summarized, thus obtaining the first file batch processing result. By calling the system callback center, real-time changes to task status and the creation of new tasks are achieved, thereby realizing the goal of intelligent batch processing of large files and achieving the technical effect of obtaining file batch processing results.

[0078] In summary, the method for batch processing and monitoring of large files provided by this invention has the following technical effects:

[0079] 1. Through a file transfer platform, files to be processed are sent to the batch processor. Simultaneously, the system monitors the transmission status of these files in real time. When a file arrives at the batch processor, an instruction is automatically triggered to perform task relationship analysis on that file, resulting in the first batch of processing task execution graphs. Based on this graph, the system analyzes and identifies the first distributed execution task, which contains multiple sub-tasks. The corresponding logic module service is then invoked, performing a series of processing steps on the first distributed execution task, including splitting, calculation, merging, reconciliation, and end-of-day processing. Finally, a callback to the scheduling center service is made to obtain the status information of each task, thus determining the processing result for the corresponding file. By employing distributed computing principles, large pipeline files from the upstream system are split into smaller files. Leveraging the advantages of distributed system microservices, each smaller file is processed independently and in parallel. Finally, the processing results of the smaller files are aggregated to obtain the processing result for the entire large pipeline file. This achieves the technical goal of improving the efficiency of large file batch processing, enhancing both the efficiency of large file processing and the flexibility of the processing system.

[0080] 2. Through the abnormal data detection model, intelligent detection of real-time monitoring data is realized, and then based on the detected abnormal data, corresponding early warnings and reminders are issued intelligently, achieving the technical effect of intelligent early warning and abnormal handling.

[0081] 3. By monitoring the entire batch processing process in real time, timely location of anomalies is achieved, and reprocessing is performed based on the located anomalies. This achieves the technical effect of flexibly reprocessing anomalies and thus improving file processing efficiency.

[0082] 4. By calling the system callback center, the status of each task is changed in real time, and new tasks are created, thereby achieving the goal of intelligent batch processing of large files and obtaining batch processing results.

[0083] 5. Through the system of batch processing and monitoring of large files, larger data volumes of files can be processed in batches, thereby saving historical data of files. At the same time, the query response time is short, and the inventory history details will not affect the query performance, thus achieving the technical effect of effectively improving the user experience.

[0084] Example 2

[0085] Based on the method for batch processing and monitoring large files described in the foregoing embodiments, and using the same inventive concept, this invention also provides a system for batch processing and monitoring large files. Please refer to the appendix. Figure 5 The system includes:

[0086] First obtaining unit 11: The first obtaining unit 11 is used to obtain the first downloaded file by sending a file to the batch processor through the file transfer platform;

[0087] First triggering unit 12: The first triggering unit 12 monitors the directory of the first distributed file, and triggers the first creation instruction when the file arrives;

[0088] First creation unit 13: The first creation unit 13 performs task relationship analysis on the first issued file according to the first creation instruction, and creates the first batch of processing task operation diagram;

[0089] Second obtaining unit 14: The second obtaining unit 14 performs distributed processing based on the first batch of processing task operation graph to obtain a first distributed execution task, wherein the first distributed execution task includes multiple sub-tasks;

[0090] Third obtaining unit 15: The third obtaining unit 15 executes a task according to the first distribution to obtain the first calling instruction;

[0091] First execution unit 16: The first execution unit 16 calls the logical module service configured corresponding to the first distributed execution task to perform parallel execution of the task according to the distributed scheduling framework and the first call instruction;

[0092] Fourth obtaining unit 17: When the task in the first distributed execution task is completed, the fourth obtaining unit 17 obtains the batch processing result of the first file by calling back the scheduling center to manage the task status.

[0093] Furthermore, the system also includes:

[0094] The fifth obtaining unit is used to obtain file type information by performing file type analysis on all files in the first issued file;

[0095] The sixth obtaining unit is used to obtain task chain information by performing task chain relationship analysis on all files in the first issued file;

[0096] The second creation unit is used to create the first batch of processing task execution graphs based on the file type information and the task chain information.

[0097] Furthermore, the system also includes:

[0098] The seventh obtaining unit is configured to configure the task module of the first distributed execution task according to the first configuration instruction, and obtain the task configuration module, wherein the task configuration module includes multiple task modules;

[0099] The eighth obtaining unit is used to configure the parameters of the task configuration module according to the second configuration instruction and obtain the module configuration parameters;

[0100] The second execution unit is used to execute the first distributed execution task according to the task configuration module and the module configuration parameters.

[0101] Furthermore, the system also includes:

[0102] The first setting unit is used by the distributed scheduling framework to periodically process all subtasks in the first distributed execution task.

[0103] Furthermore, the system also includes:

[0104] The ninth obtaining unit is used to perform batch processing monitoring of the first issued file through a visual interface to obtain real-time monitoring data;

[0105] The tenth obtaining unit is used to input the real-time monitoring data into the abnormal data detection model, perform abnormal detection according to the abnormal data detection model, and obtain the first abnormal data.

[0106] The first generation unit is configured to generate a first warning message based on the first abnormal data;

[0107] The third execution unit is used to send the first warning information as an alarm through a visual batch processing monitoring platform.

[0108] Furthermore, the system also includes:

[0109] The eleventh obtaining unit is used to obtain first real-time location information by performing anomaly location on the first abnormal data;

[0110] The second generation unit is used to generate a first rerun instruction based on the first real-time positioning information;

[0111] The fourth execution unit is used to reprocess the first abnormal data according to the first rerun instruction.

[0112] Furthermore, the system also includes:

[0113] The twelfth obtaining unit is used to obtain the first real-time task status by changing the task status through the callback scheduling center after each subtask of the first distributed execution task has been completed.

[0114] The thirteenth obtaining unit is used to obtain the parallel real-time task status of the first distributed execution task based on the first real-time task status.

[0115] The third generation unit is used to summarize the file processing based on the status of the parallel real-time task and generate the first file batch processing result.

[0116] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Figure 1 The method and specific example of large file batch processing and monitoring in Embodiment 1 are also applicable to the system of large file batch processing and monitoring in this embodiment. Through the foregoing detailed description of the method of large file batch processing and monitoring, those skilled in the art can clearly understand the system of large file batch processing and monitoring in this embodiment; therefore, for the sake of brevity, it will not be described in detail here. As for the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant details can be found in the method section.

[0117] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0118] Exemplary electronic devices

[0119] The following is for reference. Figure 6 The electronic device described herein is used to illustrate this invention.

[0120] Figure 6 The figure shows a schematic diagram of the structure of an electronic device according to the present invention.

[0121] Based on the inventive concept of a method for batch processing and monitoring large files in the foregoing embodiments, the present invention also provides a system for batch processing and monitoring large files, wherein a computer program is stored thereon, which, when executed by a processor, implements the steps of any of the methods of the aforementioned method for batch processing and monitoring large files.

[0122] Among them, Figure 6In this document, a bus architecture (represented by bus 300) is used. Bus 300 may include any number of interconnected buses and bridges, linking various circuits including one or more processors represented by processor 302 and memory represented by memory 304. Bus 300 may also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. Bus interface 305 provides an interface between bus 300 and receiver 301 and transmitter 303. Receiver 301 and transmitter 303 may be the same element, i.e., a transceiver, providing a unit for communicating with various other devices over a transmission medium.

[0123] The processor 302 is responsible for managing the bus 300 and general processing, while the memory 304 can be used to store the data used by the processor 302 when performing operations.

[0124] This invention provides a method for batch processing and monitoring large files. The method is applied to a system for batch processing and monitoring large files. The method includes: sending a file to a batch processor via a file transfer platform to obtain a first sent file; monitoring the directory of the first sent file and triggering a first creation instruction when the file arrives; performing task relationship analysis on the first sent file according to the first creation instruction to create a first batch processing task execution graph; performing distributed processing based on the first batch processing task execution graph to obtain a first distributed execution task, wherein the first distributed execution task includes multiple subtasks; obtaining a first invocation instruction based on the first distributed execution task; invoking the logical module service configured corresponding to the first distributed execution task for parallel task execution according to a distributed scheduling framework and the first invocation instruction; and obtaining the first file batch processing result by managing the task status through a callback scheduling center when the tasks in the first distributed execution task are completed. Existing technologies for batch processing large files are time-consuming, resulting in low work efficiency. Furthermore, large file data processing increases system resource risks. This invention addresses the technical problem of existing technologies being unable to promptly identify and address anomalies during processing, resulting in insufficient flexibility. By employing the principles of distributed computing, large files are broken down into smaller files. The advantages of microservices in a distributed system are then leveraged to process each smaller file independently and in parallel. Finally, the processing results of the smaller files are aggregated to obtain the final result for the entire large file, thus achieving the technical goal of improving the efficiency of large file batch processing. Through distributed computing, this approach effectively enhances the efficiency of large file processing while simultaneously increasing the flexibility of the processing system.

[0125] The present invention also provides an electronic device, wherein the device includes a processor and a memory;

[0126] This memory is used for storage;

[0127] The processor is used to execute the method described in any one of the above embodiments by means of a call.

[0128] The present invention also provides a computer program product, including a computer program and / or instructions, which, when executed by a processor, implement the steps of the method described in any of the above embodiments.

[0129] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely software embodiment, a completely hardware embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention is in the form of a computer program product executable on one or more computer-usable storage media containing computer-usable program code. Such computer-usable storage media include, but are not limited to, various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), disk storage, compact disc read-only memory (CD-ROM), and optical storage.

[0130] This invention is described with reference to flowchart illustrations and / or block diagrams of the methods, apparatus (systems), and computer program products of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.

[0131] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction set implemented in a process. Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0132] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps of the functions specified in one or more boxes. Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments.

[0133] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of this invention and its equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for batch processing and monitoring large files, characterized in that, The method includes: The first file is sent to the batch processor via a file transfer platform. By monitoring the directory of the first distributed file, a first creation command is triggered when the file arrives. Based on the first creation instruction, perform task relationship analysis on the first issued file and create the first batch of processing task execution diagrams; Based on the first batch of processing task execution graph, distributed processing is performed to obtain a first distributed execution task, wherein the first distributed execution task includes multiple sub-tasks. Execute the task according to the first distribution to obtain the first invocation instruction; Based on the distributed scheduling framework and the first invocation instruction, the logical module service configured corresponding to the first distributed execution task is invoked to perform parallel task execution; When the task in the first distributed execution task is completed, the task status is managed through the callback scheduling center to obtain the batch processing result of the first file. The callback scheduling center also performs task configuration and processing based on real-time query of tasks that meet the conditions, and changes the task status in real time to create new tasks. The method further includes: According to the first configuration instruction, the first distributed execution task is configured with task modules to obtain a task configuration module, wherein the task configuration module includes multiple task modules; According to the second configuration instruction, the task configuration module is configured with parameters to obtain the module configuration parameters; The first distributed execution task is executed according to the task configuration module and the module configuration parameters; The method of analyzing the first issued file according to the first creation instruction to create the first batch of processing task execution diagrams further includes: File type information is obtained by analyzing the file types of all files in the first issued file. Task chain information is obtained by analyzing the task chain relationships of all files in the first issued file; Based on the file type information and the task chain information, create the first batch of processing task execution graphs.

2. The method as described in claim 1, characterized in that, The distributed scheduling framework processes all subtasks of the first distributed execution task periodically.

3. The method as described in claim 1, characterized in that, The method further includes: The first distributed file is batch-processed and monitored through a visual interface to obtain real-time monitoring data. The real-time monitoring data is input into the anomaly detection model, and anomaly detection is performed according to the anomaly detection model to obtain the first anomaly data. Based on the first abnormal data, a first early warning message is generated; The first warning information will be sent out as an alert through a visual batch processing monitoring platform.

4. The method as described in claim 3, characterized in that, After inputting the real-time monitoring data into the anomaly detection model, performing anomaly detection based on the anomaly detection model, and obtaining the first anomaly data, the method further includes: By performing anomaly localization on the first abnormal data, first real-time location information is obtained; Based on the first real-time positioning information, a first rerun instruction is generated; The first abnormal data is reprocessed according to the first rerun instruction.

5. The method as described in claim 1, characterized in that, When the task in the first distributed execution task is completed, the task status is managed through a callback scheduling center to obtain the batch processing result of the first file. The method further includes: After each subtask of the first distributed execution task is completed, the task status is changed by calling back the scheduling center to manage the task status and obtain the first real-time task status. Based on the first real-time task status, obtain the parallel real-time task status of the first distributed execution task; Based on the status of the parallel real-time tasks, the file processing is summarized to generate the first file batch processing result.

6. A system for batch processing and monitoring large files, characterized in that, The system includes: First obtaining unit: The first obtaining unit is used to send a file to the batch processor through the file transfer platform to obtain the first sent file; First triggering unit: The first triggering unit monitors the directory of the first distributed file, and triggers the first creation instruction when the file arrives; First creation unit: The first creation unit performs task relationship analysis on the first issued file according to the first creation instruction, and creates the first batch of processing task execution diagrams; Second obtaining unit: The second obtaining unit performs distributed processing based on the first batch of processing task operation graph to obtain a first distributed execution task, wherein the first distributed execution task includes multiple sub-tasks; Third obtaining unit: The third obtaining unit executes a task according to the first distribution to obtain the first calling instruction; First execution unit: The first execution unit calls the logical module service configured corresponding to the first distributed execution task to perform parallel execution of the task according to the distributed scheduling framework and the first invocation instruction; Fourth obtaining unit: When the task in the first distributed execution task is completed, the fourth obtaining unit manages the task status through the callback scheduling center and obtains the batch processing result of the first file. The callback scheduling center also performs task configuration and processing based on real-time query of tasks that meet the conditions, and changes the task status in real time, thereby creating new tasks. The system also includes: The fifth obtaining unit is used to obtain file type information by performing file type analysis on all files in the first issued file; The sixth obtaining unit is used to obtain task chain information by performing task chain relationship analysis on all files in the first issued file; The second creation unit is used to create the first batch of processing task execution graphs based on the file type information and the task chain information. The seventh obtaining unit is configured to configure the task module of the first distributed execution task according to the first configuration instruction, and obtain the task configuration module, wherein the task configuration module includes multiple task modules; The eighth obtaining unit is used to configure the parameters of the task configuration module according to the second configuration instruction and obtain the module configuration parameters; The second execution unit is used to execute the first distributed execution task according to the task configuration module and the module configuration parameters.

7. An electronic device, characterized in that, Including processor and memory; The memory is used for storage; The processor is configured to execute the method of any one of claims 1 to 5 by means of a call.

8. A computer program product, comprising a computer program and / or instructions, characterized in that, When the computer program and / or instructions are executed by a processor, they implement the steps of the method described in any one of claims 1 to 5.

Images