Task processing server, method and system based on asynchronous distributed processing technology

Abstract

The application provides a task processing server, method and system based on asynchronous distributed processing technology, comprising a data collection server, a data processing server and a data output server; the data collection server comprises a file import module; the file import module stores the imported file by using a file server after receiving the file from a client, and the file is queued to wait for processing by a file splitting module after being stored; the data processing server comprises the file splitting module, a file distributed processing module and a file integration module; the data output server comprises a file output module; the file output module obtains a result file and transmits the result file to a client. When a large number of tasks need to be processed, the application realizes asynchronous and distributed rapid processing of the tasks through data import, asynchronous splitting, distributed processing and summarization and integration.

Description

Technical Field

[0001] This invention relates to the technical field of batch task processing, and more specifically, to a task processing server, method, and system based on asynchronous distributed processing technology. Background Technology

[0002] When dealing with a large number of tasks, existing technologies use relatively traditional methods, which are time-consuming and labor-intensive. Therefore, current traditional methods need to be improved.

[0003] A patent document with publication number CN112100227A discloses a big data processing method based on multi-level heterogeneous data storage, belonging to the field of data processing technology, and particularly relating to a big data processing method based on multi-level heterogeneous data storage. This invention provides a big data processing method based on multi-level heterogeneous data storage, which overcomes the problems of inconsistent data standards across disciplines and departments, relatively independent business systems, inconsistent data storage application standards, difficulty in cross-disciplinary information resource integration, and high business integration difficulty in existing methods. It also solves the problems of numerous data types, large volumes, difficulty in aggregation, and lack of effective processing methods for diverse data types. A hierarchical data integration architecture is constructed to achieve efficient aggregation of multi-level heterogeneous data throughout the entire power industry process, realizing horizontal data sharing and vertical integration, and eliminating the barriers of fragmented enterprise information.

[0004] The inventor believes that the processing power of the existing technology needs to be improved, which leads to low work efficiency. Therefore, a new technical solution needs to be proposed. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the purpose of this invention is to provide a task processing server, method, and system based on asynchronous distributed processing technology.

[0006] According to the present invention, a task processing server based on asynchronous distributed processing technology includes a data acquisition server, a data processing server, and a data output server.

[0007] The data acquisition server includes a file import module; after receiving a file from the client, the file import module stores the imported file on the file server, and the file is queued for processing by the file splitting module after being stored.

[0008] The data processing server includes a file segmentation module, a file distribution processing module, and a file integration module. The file segmentation module reads files from the file server and performs file segmentation based on the file content size. The file distribution processing module obtains the sub-file it is currently processing, analyzes the file content, merges data with similar characteristics, generates a list to be processed, processes each item on the list, generates a result file after processing, and notifies the file integration module. The file integration module continuously queries the processing results of the file distribution processing modules. After all distribution processing modules have completed their processing, it obtains the result files generated by each distribution processing module, integrates multiple result files into a single file, and sends it to the file output module.

[0009] The data output server includes a file output module; the file output module acquires the result file and transmits the result file to the client.

[0010] Preferably, the server is used for processing bank batch deduction files; after the customer submits the batch deduction file, it is summarized and filtered based on the deduction account as a feature value. The same deduction account is summarized together to generate a sub-file, and the sub-tasks are distributed to summarize and merge the deduction amounts corresponding to the same deduction account for accounting.

[0011] This invention provides a task processing method based on asynchronous distributed processing technology. The method utilizes the aforementioned task processing server based on asynchronous distributed processing technology and includes the following steps:

[0012] Step S1: Create online data import transactions, online data export transactions, and background asynchronous tasks 1, 2, and 3;

[0013] Step S2: The client imports the data to be processed into the server through online import transactions, saves the file name and record count, and notifies the background asynchronous task 1 to start;

[0014] Step S3: Background asynchronous task 1 reads the file name and record count, and determines whether the starting number for splitting has been reached based on the record count;

[0015] Step S4: Background asynchronous task 2 starts multiple processes simultaneously based on the total number of sub-files, with each process responsible for processing one sub-file;

[0016] Step S5: After the background asynchronous task 3 starts, it continuously scans the execution status of task 2. After all tasks 2 have been completed, it reads the result files generated by each task 2, merges the result files, and generates the final result file for this processing.

[0017] Step S6: The client exports the transaction data online and reads the final result file generated by the background asynchronous task 3. Based on the result file, it determines whether the processing was successful.

[0018] Preferably, after step S3 reaches the initial number of segments, the file is sorted according to the account code field in the file content to obtain a new file. Then, the file is read line by line from the beginning of the file and saved to the new file 1. After reaching the standard number of segments, it is first determined whether the next line has the same account code as the previous line.

[0019] Preferably, if they are the same, read another line, and then loop to check if the next line is the same as the previous line, until they are different. Save the file as subfile 1, and record the subfile name and the number of records in the subfile. Loop through the splitting until the file is processed. After processing is complete, notify task 2 and task 3 to start.

[0020] Preferably, in step S4:

[0021] After background asynchronous task 2 starts, it reads the sub-file and the number of records in the sub-file. After verifying the correctness of the sub-file, it processes it line by line. During each processing, it reads the account code of the current line and then reads the account code of the next line. If they are the same, the accounting amount is merged. The process is repeated until the account codes are no longer the same. Then, the accounting module is called to merge the accounting deduction and the accounting result is recorded to the result file. The above steps are repeated to process all the records in the sub-file.

[0022] The present invention also provides a task processing system based on asynchronous distributed processing technology, wherein the system applies the task processing server based on asynchronous distributed processing technology described above, and the system includes the following modules:

[0023] Module M1: Creates online data import transactions, online data export transactions, and background asynchronous tasks 1, 2, and 3;

[0024] Module M2: The client imports the data to be processed into the server through online import transactions, saves the file name and record count, and notifies the background asynchronous task 1 to start;

[0025] Module M3: Background asynchronous task 1 reads the file name and record count, and determines whether the starting quantity for splitting has been reached based on the record count;

[0026] Module M4: Background asynchronous task 2 starts multiple processes simultaneously based on the total number of sub-files, with each process responsible for processing one sub-file;

[0027] Module M5: After background asynchronous task 3 starts, it continuously scans the execution status of task 2. After all tasks 2 have been completed, it reads the result files generated by each task 2, merges the result files, and generates the final result file for this processing.

[0028] Module M6: The client exports transaction data via online data, reads the final result file generated by the background asynchronous task 3, and determines whether the processing was successful based on the result file.

[0029] Preferably, after module M3 reaches the initial number of splits, it sorts the file according to the account code field in the file content to obtain a new file, and then reads the file line by line from the beginning of the file and saves it to the new file 1. After reaching the standard number of split lines, it first determines whether the next line has the same account code as the previous line.

[0030] Preferably, if they are the same, read another line, and then loop to check if the next line is the same as the previous line, until they are different. Save the file as subfile 1, and record the subfile name and the number of records in the subfile. Loop through the splitting until the file is processed. After processing is complete, notify task 2 and task 3 to start.

[0031] Preferably, in module M4:

[0032] After background asynchronous task 2 starts, it reads the sub-file and the number of records in the sub-file. After verifying the correctness of the sub-file, it processes it line by line. Each time it processes, it reads the account number of the current line and then reads the account number of the next line. If they are the same, it merges the accounting amount. It loops until the account numbers are no longer the same, then it calls the accounting module to merge the accounting deduction and records the accounting result to the result file. The above module is called repeatedly to process all the records in the sub-file.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 1. When a large number of tasks need to be processed, this invention achieves asynchronous and rapid task processing through data import, asynchronous splitting, distributed processing, and aggregation and integration.

[0035] 2. This invention can perform asynchronous distributed processing of large files, improving execution efficiency;

[0036] 3. The feature value filtering and summarizing function in the file segmentation module enables the same processing steps to be summarized and processed without processing each item individually, thus improving execution efficiency. Attached Figure Description

[0037] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0038] Figure 1 This is a flowchart illustrating the principle of the present invention. Detailed Implementation

[0039] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0040] Example 1:

[0041] According to the present invention, a task processing server based on asynchronous distributed processing technology includes a data acquisition server, a data processing server, and a data output server.

[0042] The data acquisition server includes a file import module; after receiving files from the client, the file import module stores the imported files on a file server, and the files are then queued for processing by the file splitting module.

[0043] The data processing server includes a file splitting module, a file distribution processing module, and a file integration module. The file splitting module reads files from the file server and splits them according to their size. The file distribution processing module obtains the sub-file it is currently processing, analyzes its content, merges data with similar characteristics, generates a list of files to be processed, processes each file according to the list, generates a result file after processing, and notifies the file integration module. The file integration module continuously queries the processing results of the file distribution processing modules. After all distribution processing modules have completed their processing, it obtains the result files generated by each module, integrates the multiple result files into a single file, and sends it to the file output module.

[0044] The data output server includes a file output module; the file output module obtains the result file and sends the result file to the client.

[0045] The server is used for processing bank batch deduction files. After the customer submits the batch deduction file, it is summarized and filtered based on the deduction account as a feature value. The same deduction account is summarized together to generate a sub-file. The sub-tasks are processed separately to summarize and merge the deduction amounts corresponding to the same deduction account for accounting.

[0046] Reference Figure 1 This invention provides a task processing method based on asynchronous distributed processing technology. The method utilizes the aforementioned task processing server based on asynchronous distributed processing technology and includes the following steps:

[0047] Step S1: Create online data import transactions, online data export transactions, and background asynchronous tasks 1, 2, and 3;

[0048] Step S2: The client imports the data to be processed into the server through online import transactions, saves the file name and record count, and notifies the background asynchronous task 1 to start;

[0049] Step S3: Background asynchronous task 1 reads the filename and record count, and determines whether the starting number for splitting has been reached based on the record count. After reaching the starting number for splitting, it sorts the file according to the account code field in the file content to obtain a new file. Then, it reads the file line by line from the beginning of the file and saves it to the new file 1. After reaching the standard number of lines for splitting, it first checks whether the account code of the next line is the same as that of the previous line. If they are the same, it reads another line and then loops to check whether the next line is the same as the previous line until they are different. The file is then saved as sub-file 1, and the sub-file name and the number of records in the sub-file are recorded. The splitting is repeated until the file is processed. After processing is complete, task 2 and task 3 are notified to start.

[0050] Step S4: Background asynchronous task 2 starts multiple processes simultaneously based on the total number of sub-files. Each process is responsible for processing one sub-file. After starting, background asynchronous task 2 reads the sub-file and the number of records in the sub-file. After verifying the correctness of the sub-file, it processes it line by line. Each time it processes, it reads the account number of the current line and then reads the account number of the next line. If they are the same, the accounting amount is merged. The process is repeated until the account numbers are no longer the same. Then, the accounting module is called to merge the accounting deduction and record the accounting result to the result file. The above steps are repeated to process all the records in the sub-file.

[0051] Step S5: After the background asynchronous task 3 starts, it continuously scans the execution status of task 2. After all tasks 2 have been completed, it reads the result files generated by each task 2, merges the result files, and generates the final result file for this processing.

[0052] Step S6: The client exports the transaction data online and reads the final result file generated by the background asynchronous task 3. Based on the result file, it determines whether the processing was successful.

[0053] Example 2:

[0054] Example 2 is a preferred embodiment of Example 1, and is used to illustrate the present invention in more detail.

[0055] The present invention also provides a task processing system based on asynchronous distributed processing technology. The system applies the task processing server based on asynchronous distributed processing technology described above, and the system includes the following modules:

[0056] Module M1: Creates online data import transactions, online data export transactions, and background asynchronous tasks 1, 2, and 3;

[0057] Module M2: The client imports the data to be processed into the server through online import transactions, saves the file name and record count, and notifies the background asynchronous task 1 to start;

[0058] Module M3: Background asynchronous task 1 reads the filename and record count, and determines whether the starting number for splitting has been reached based on the record count. After reaching the starting number, it sorts the file according to the account code field in the file content to obtain a new file. Then, it reads the file line by line from the beginning of the file and saves it to the new file 1. After reaching the standard number of lines for splitting, it first checks whether the next line has the same account code as the previous line. If they are the same, it reads another line and then loops to check whether the next line is the same as the previous line until they are different. The file is then saved as sub-file 1, and the sub-file name and the number of records in the sub-file are recorded. The splitting is repeated until the file is processed. After processing is complete, task 2 and task 3 are notified to start.

[0059] Module M4: Background asynchronous task 2 starts multiple processes simultaneously based on the total number of sub-files. Each process is responsible for processing one sub-file. After starting, background asynchronous task 2 reads the sub-file and the number of records in the sub-file. After verifying the correctness of the sub-file, it processes it line by line. During each processing, it reads the account number of the current line and then reads the account number of the next line. If they are the same, the accounting amount is merged. The process is repeated until the account numbers are no longer the same. Then, the accounting module is called to merge the accounting deduction and record the accounting result to the result file. The above module is called repeatedly to process all the records in the sub-file.

[0060] Module M5: After background asynchronous task 3 starts, it continuously scans the execution status of task 2. After all tasks 2 have been completed, it reads the result files generated by each task 2, merges the result files, and generates the final result file for this processing.

[0061] Module M6: The client exports transaction data via online data, reads the final result file generated by the background asynchronous task 3, and determines whether the processing was successful based on the result file.

[0062] Example 3:

[0063] Example 3 is a preferred example of Example 1, and is used to illustrate the present invention in more detail.

[0064] This invention provides a method for processing large tasks based on asynchronous distributed processing technology. When a large number of tasks need to be processed, the method achieves fast asynchronous and distributed processing of tasks through data import, asynchronous splitting, distributed processing, and summary integration.

[0065] A method for handling large tasks based on asynchronous distributed processing technology includes: a data acquisition server, a data processing server, and a data output server.

[0066] The data acquisition server includes a file import module;

[0067] The data processing server includes a file splitting module, a file distribution processing module, and a file integration module;

[0068] The data output server includes a file output module.

[0069] After receiving the file from the client, the file import module stores the imported file on the file server. After the file is stored, it is queued for processing by the file splitting module.

[0070] The file splitting module reads files from the file server and splits them according to their size. This module supports parameterized configuration; splitting occurs when the file size exceeds the configured maximum number of lines per file. The splitting algorithm supports splitting based on data characteristics to simplify subsequent processing. Based on the data processing characteristics, the file data is first sorted, records of the same account are aggregated, and then splitting is performed according to the maximum number of lines per file. If a file is split and the next record has the same characteristics as the previous record, the split is postponed and the next record is retrieved. After file splitting, file distribution processing tasks are started based on the number of sub-files, with each task responsible for processing one sub-file.

[0071] The file distribution processing module obtains the sub-file it is currently responsible for processing, analyzes the file content, merges data with the same characteristics, generates a list to be processed, processes each item on the list, generates a result file after processing is completed, and notifies the file integration module.

[0072] The file integration module cyclically queries the processing results of the file distribution processing module. After all the distribution processing modules have completed their processing, it obtains the result files generated by the file distribution processing modules one by one, integrates the multiple result files into a single file, and sends it to the file output module.

[0073] The file output module obtains the result file and transmits the result file to the client;

[0074] Based on this solution, it can be applied to the processing of batch deduction files for banks. After a customer submits a batch deduction file, it is summarized and filtered based on the deduction account as a feature value. The same deduction accounts are grouped together to generate sub-files, and the sub-tasks are distributed to summarize and merge the deduction amounts corresponding to the same deduction accounts for accounting purposes.

[0075] Specifically as follows:

[0076] Create online data import transactions, online data export transactions, and background asynchronous tasks 1, 2, and 3.

[0077] The client imports the data to be processed into the server via online import transactions, saves the file name and record count, and notifies the background asynchronous task 1 to start.

[0078] Background asynchronous task 1 reads the filename and record count. Based on the record count, it determines whether the starting number for splitting has been reached. Once the starting number is reached, it sorts the files according to the account ID field in the file content, obtaining a new file. Then, it reads the file line by line from the beginning and saves it to the new file 1. After reaching the splitting threshold, it first checks if the next line has the same account ID as the previous line. If they are the same, it reads another line and repeats the process until they are different. This file is then saved as sub-file 1, and the sub-file name and record count are recorded. This process is repeated until all files are processed. After completion, tasks 2 and 3 are notified to start.

[0079] Background asynchronous task 2 starts multiple processes simultaneously based on the total number of sub-files, with each process responsible for processing one sub-file. After starting, background asynchronous task 2 reads the sub-file and its record count, verifies the sub-file's correctness, and processes it line by line. During each processing iteration, it reads the account number from the current line, then the account number from the next line. If they match, the accounting amounts are merged. This process is repeated until the account numbers are no longer identical, at which point the accounting module is called to merge the accounting deductions and record the results in the results file. These steps are repeated until all records in the sub-file are processed.

[0080] After background asynchronous task 3 starts, it continuously scans the execution status of task 2. After all tasks 2 have been completed, it reads the result files generated by each task 2, merges the result files, and generates the final result file for this processing.

[0081] The client exports the transaction data online and reads the final result file generated by the background asynchronous task 3. It then determines whether the processing was successful based on the result file.

[0082] Large-task processing methods based on asynchronous distributed processing technology are widely used in business scenarios that require large file processing. By using appropriate file sorting and splitting in an asynchronous parallel manner, compared with processing a single file line by line, processing capacity can be greatly improved and computer resources can be fully utilized.

[0083] How does the file segmentation module determine and summarize file characteristics before segmenting? This invention enables asynchronous, distributed processing of large files, improving execution efficiency. By using the feature value filtering and summarizing function in the file segmentation module, identical processing steps can be summarized and processed, eliminating the need for processing each file individually, thus improving execution efficiency.

[0084] Those skilled in the art can understand this embodiment as a more specific description of Embodiment 1 and Embodiment 2.

[0085] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0086] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A task processing server based on asynchronous distributed processing technology, characterized by, The server comprises a data collection server, a data processing server and a data output server; The data collection server comprises a file import module; the file import module stores the imported file in a file server after receiving the file from the client, and queues the file for processing by a file segmentation module after the file is stored; The data processing server comprises a file segmentation module, a file dispersion processing module and a file integration module; the file segmentation module reads the file on the file server, and performs file segmentation processing according to the file content size; the file dispersion processing module obtains the sub-file currently responsible for processing, analyzes the file content first, merges the data with the same characteristics, generates a to-be-processed list, processes each item in the list according to the list, generates a result file after the processing is completed, and notifies the file integration module; The file integration module cyclically queries the processing result of the file dispersion processing module, obtains the result file generated by the file dispersion processing module one by one after all the file dispersion processing modules are processed, integrates the multiple result files into a single file, and transmits the single file to the file output module; The data output server comprises a file output module; The file output module obtains the result file and transmits the result file to the client; The server is applied to the processing of bank batch deduction files; After the client submits the batch deduction file, the same deduction account is summarized together to generate a sub-file according to the deduction account as a characteristic value for screening, and the same deduction account corresponding to the deduction amount is summarized and merged for account recording by the sub-task dispersion processing; The task processing method based on the asynchronous dispersion processing technology comprises the following steps: Step S1: creating an online data import transaction, an online data export transaction, a background asynchronous task 1, a task 2 and a task 3; Step S2: the client imports the to-be-processed data into the server through the online import transaction, saves the file name and the record number, and notifies the background asynchronous task 1 to start; Step S3: the background asynchronous task 1 reads the file name and the record number, and determines whether the segmentation starting quantity is reached according to the record number; Step S4: the background asynchronous task 2 simultaneously starts multiple processes according to the total number of sub-files, and each process is responsible for processing a sub-file; Step S5: the background asynchronous task 3 starts to cyclically scan the execution of the task 2, reads the result file generated by each task 2 after the task 2 is completely executed, merges the result files, and generates the final result file of the current processing; Step S6: the client reads the final result file generated by the background asynchronous task 3 through the online data export transaction, and judges whether the processing is successful according to the result file.

2. The task processing server based on asynchronous distributed processing technology according to claim 1, characterized in that, In step S3, after the segmentation starting quantity is reached, the file is sorted according to the account code field in the file content to obtain a new file, the file is read from the starting line of the file to the new file 1, the next line is judged to be the same as the previous line after the segmentation standard line number is reached, and the file is saved as a sub-file 1.

3. The task processing server based on asynchronous distributed processing technology according to claim 2, characterized in that, If the next line is the same as the previous line, one more line is read, and the next line is cyclically judged to be the same as the previous line until the next line is different from the previous line, the file is saved as a sub-file 1, the sub-file name and the sub-file record number are registered, the file is cyclically segmented until the file processing is completed, and the task 2 and the task 3 are notified to start after the processing is completed.

4. The task processing server based on asynchronous distributed processing technology according to claim 1, characterized in that, In step S4, The background asynchronous task 2 reads the sub-file and the sub-file record number after starting, checks the correctness of the sub-file, and processes each row after the correctness is verified. The account code of the current row is read each time the processing is performed, and the account code of the next row is read. If the account codes are the same, the account balance is combined. The account balance is combined and the account balance is deducted by calling the account module after the account codes are read in a loop until the account codes are not the same. The account result is recorded to the result file. The above steps are repeated to process all records of the sub-file.

5. A task processing system based on asynchronous distributed processing technology, characterized by, The system applies the task processing server based on the asynchronous distributed processing technology in any one of claims 1-2, and the system comprises the following modules. Module M1: creating an online data import transaction, an online data export transaction, a background asynchronous task 1, a task 2, and a task 3; Module M2: the client imports the data to be processed into the server through the online import transaction, saves the file name and the record number, and informs the background asynchronous task 1 to start; Module M3: the background asynchronous task 1 reads the file name and the record number, and determines whether the splitting starting number is reached according to the record number; Module M4: the background asynchronous task 2 starts multiple processes according to the total number of sub-files, and each process is responsible for processing a sub-file; Module M5: the background asynchronous task 3 starts to scan the task 2 execution in a loop. After the task 2 is completely executed, the result file generated by each task 2 is read, the result files are combined, and the final result file of this processing is generated; Module M6: the client reads the final result file generated by the background asynchronous task 3 through the online data export transaction, and judges whether the processing is successful according to the result file.

6. The task processing system based on asynchronous distributed processing technology according to claim 5, characterized in that, After the module M3 reaches the splitting starting number, the account code field in the file content is sorted to obtain a new file. The file is read from the starting row of the file to the new file 1. After the splitting standard row number is reached, whether the next row is the same as the account code of the previous row is judged.

7. The task processing system based on asynchronous distributed processing technology according to claim 6, characterized in that, If the same, the next row is read again, and whether the next row is the same as the previous row is judged in a loop until the next row is different from the previous row. The file is saved as a sub-file 1, and the sub-file name and the sub-file record number are registered. The file is split in a loop until the file is processed. After the processing is completed, the task 2 and the task 3 are informed to start.

8. The task processing system based on asynchronous distributed processing technology according to claim 5, characterized in that, In the module M4: The background asynchronous task 2 reads the sub-file and the sub-file record number after starting, checks the correctness of the sub-file, and processes each row after the correctness is verified. The account code of the current row is read each time the processing is performed, and the account code of the next row is read. If the account codes are the same, the account balance is combined. The account balance is combined and the account balance is deducted by calling the account module after the account codes are read in a loop until the account codes are not the same. The account result is recorded to the result file. All records of the sub-file are processed by repeating the above modules.

Images