File task processing method supporting distributed configuration function

Abstract

The invention discloses a file task processing method supporting a distributed configuration function. The file task processing method comprises the following steps: converting service processing from a synchronous mode into asynchronous processing; realizing configuration processing of an export template and dynamic processing of excl header data of a data set; exporting by adopting in-table calculation formula and inter-table calculation formula configuration, and optimizing an excl summation column and inter-table data calculation logic; and establishing multi-tool export configuration. A thread pool technology is introduced to realize a multi-thread and high-concurrency processing import (export) function; a special server is deployed, and the memory of the business server is not consumed. And thread pool, form and formula class configuration operation is supported.

Description

Technical Field

[0001] The present invention relates to the field of bank file processing, and in particular to a file task processing method supporting distributed configuration functions. Background Art

[0002] As the digital transformation of banks continues to deepen, the business logic of banks has gradually become more complex. All major systems of banks need to import and process data in batches and generate a large number of data reports. In high-concurrency businesses, the report import (export) process is frequently used, and single services will have data logic processing occupying a large amount of memory, thus affecting the operational safety of the production environment. Now it is necessary to optimize and adjust this function, implement distributed expansion, introduce configuration functions, and improve the processing efficiency of the system's non-report import (export) function.

[0003] The existing system adopts a monolithic system development model. The application is deployed on the same server. In the entire monolithic function, the import (export) function accounts for a large proportion of the overall business and is used frequently. The amount of data imported (exported) is large and the business processing logic is complex (more than 10,000 data). When multiple requests are concurrent, multiple threads will be opened for export. Due to the large amount of import (export) and the large amount of server memory occupied, the production server memory is tight, thus affecting other business operations.

[0004] Disadvantages of existing technologies: high memory consumption, severe coupling between import (export) and business functions; not suitable for distributed development; no configurable operation, hard-coded data verification. If there is a small form change, a large amount of code needs to be changed; code redundancy. Summary of the invention

[0005] In view of the above problems, the present invention is proposed to provide a file task processing method supporting distributed configuration functions that overcomes the above problems or at least partially solves the above problems.

[0006] According to one aspect of the present invention, a file task processing method supporting distributed configuration function is provided, the file task processing method comprising:

[0007] Convert business processing from synchronous to asynchronous mode;

[0008] Implement configurable export templates and data sets to dynamically process excl header data;

[0009] Export uses calculation formulas within and between tables to optimize the calculation logic of EXCL total columns and inter-table data.

[0010] Create multi-tool export configurations.

[0011] Optionally, converting the business processing from synchronous mode to asynchronous mode specifically includes:

[0012] All report requests are transformed into asynchronous mode. When there is a report export request, the current export report configuration information is obtained, and the export task table data is saved according to the configuration information;

[0013] If the report task is configured for quasi-real-time export, the API interface provided by the report service is called synchronously.

[0014] Optionally, the task table data includes: report ID, report number, report name, export file name, report type, report export method, export request parameters, export batch, report export requester number, report export requester name, report export times, status, export information, creation time and last update time.

[0015] Optionally, the status includes: unprocessed, exporting, export failed, and export successful.

[0016] Optionally, the file task processing method further includes: adding a report export service to be responsible for all large data volume report export functions of the system.

[0017] Optionally, the report export method specifically includes: quasi-real-time export and scheduled task export;

[0018] The quasi-real-time export is to receive the APP service report export batch parameters, obtain the export task table data and the report task configuration table data, determine the current export task status, and if it is in the export process, skip it and directly return success; otherwise, return specific error information according to the processing result;

[0019] The scheduled task export is as follows: obtaining the export task table data and the report task configuration table data, determining the current export task status, if it is in the exporting process, skipping it and directly returning success; otherwise, returning specific error information according to the processing result, and recording the error information in the export task table.

[0020] Optionally, the establishing of the multi-tool export configuration specifically includes: multiple thread pool components and export function components.

[0021] Optionally, the multiple thread pool components include: a new thread pool component responsible for concurrent execution of multiple tasks and supporting arbitrary adjustment of thread pool size;

[0022] Scheduled tasks support configurable days to clean up generated export files;

[0023] Manually trigger garbage collection.

[0024] Optionally, the export function component includes: migrating the existing system export function to the report export service, and upgrading the export technology component.

[0025] The present invention provides a file task processing method that supports distributed configuration functions, and the file task processing method includes: converting business processing from synchronous mode to asynchronous processing; implementing configuration processing to export templates and data sets to dynamically process excl header data; exporting using calculation formulas within the table and calculation formulas between tables, optimizing excl total columns and calculation logic between tables; establishing multi-tool export configuration. Reference thread pool technology to achieve multi-threaded, high-concurrency import (export) functions; deploy dedicated servers without consuming business server memory. Support thread pool, form, and formula class configuration operations.

[0026] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0028] Figure 1 A schematic diagram of the overall export solution flow provided by an embodiment of the present invention;

[0029] Figure 2 A schematic diagram of a thread pool component provided by an embodiment of the present invention;

[0030] Figure 3 A schematic diagram of the overall import solution flow provided for an embodiment of the present invention;

[0031] Figure 4 A schematic diagram of a newly added thread pool component provided in an embodiment of the present invention, which is responsible for concurrent execution of multiple tasks. DETAILED DESCRIPTION

[0032] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0033] The terms "comprises" and "having" and any variations thereof in the description embodiments, claims and drawings of the present invention are intended to cover non-exclusive inclusions, for example, including a series of steps or units.

[0034] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0035] The "configurable import (export) file processing task center" is an important business function that is indispensable to banking business. The "task center" business is frequently used and the import (export) function involves complex business logic. This design non-invasively adds additional extension modules to the business system to achieve unified import (export) of the entire bank system, and the field attributes of the import (export) template can be highly configurable.

[0036] The "Configuration-based Import (Export) File Processing Task Center" project is based on the Spring Boot framework, integrates Alibaba's latest easyExcl and easyPoi frameworks as import (export) methods, and adopts the design ideas of asynchronous callback and synchronous response. This function mainly includes the configuration of import (export) template page fields, import (export) titles, field type configuration, import (export) table calculation formula configuration, import table verification test configuration, import (export) table formula pair menu configuration, import (export) table formula pair branch configuration, error information configuration, import element association configuration, and thread pool configuration.

[0037] Export function-solution ideas

[0038] Synchronous to asynchronous: Change business processing from synchronous to asynchronous, improve transaction concurrent processing capabilities, and perform background sequential multi-threaded processing.

[0039] Implement configurable export templates and data sets to dynamically process EXCL header data.

[0040] The export uses calculation formulas within the table and between tables to greatly improve the calculation ability and optimize the logic of EXCL total columns and inter-table data calculations.

[0041] Export configurations with multiple tools (EasyExcl, poi).

[0042] The overall export solution process design is as follows Figure 1 As shown,

[0043] App service: transform all report requests into asynchronous mode. When there is a report export request, obtain the current export report configuration information and save the export task table data according to the configuration information. The task table saves the following contents: report ID, report number, report name, export file name, report type, report export method (quasi-real-time, scheduled), export request parameters, export batch (unique), report export requester number, report export requester name, report export times, status (unprocessed, exporting, export failed, export successful), export information (record export failure error exception information), creation time, and last update time.

[0044] If the report task is configured for quasi-real-time export, the API interface provided by the report service is called synchronously.

[0045] Report export service:

[0046] Added a new report export service, which is responsible for all large data report export functions of the system.

[0047] Export method:

[0048] Near real-time export:

[0049] Receive the APP service report export batch parameters, obtain the export task table data and report task configuration table data, determine the current export task status (because the scheduled task may be executed first), and if it is in the export process, skip it and return success directly. Otherwise, return specific error information according to the processing result.

[0050] Scheduled task export:

[0051] Get the data of the export task table and the report task configuration table, determine the current export task status, and if it is in the process of exporting, skip it and return success directly. Otherwise, return specific error information according to the processing result and record the error information in the export task table.

[0052] like Figure 2 As shown, the thread pool component:

[0053] 1. Added a new thread pool component, which is responsible for concurrent execution of multiple tasks and supports arbitrary adjustment of the thread pool size;

[0054] 2. Scheduled tasks support configurable days to clean up generated export files;

[0055] 3. Manually trigger garbage collection.

[0056] Export function component: Migrate the existing system export function to the report export service, and upgrade the export technology component to the Alibaba EasyExcel component.

[0057] The import functions include:

[0058] 1. Synchronous to asynchronous: Change the synchronous business processing mode to asynchronous processing, improve transaction concurrency, and perform background sequential multi-threaded processing.

[0059] 2. Implement configurable import form validation, dynamically execute calculation formula configuration within the import table, validation configuration within the import table, menu configuration for formula pairs within the import table, and configuration of the relationship between import elements, generate error excl for data that failed to be imported, and provide the reason for failure for download and query.

[0060] If the report task is configured for quasi-real-time import, the API interface provided by the report service is called synchronously.

[0061] Multi-tool import file configuration (EasyPoi, poi)

[0062] The overall import solution process design is as follows Figure 3 shown.

[0063] App service: 1. Convert all report requests to asynchronous mode. When there is a report import request, obtain the current import report configuration information and save the import task table data according to the configuration information. The task table saves the following contents: report ID, report number, report name, import file name, report type, report import method (quasi-real-time, scheduled), import request parameters, import batch (unique), report import requester number, report import requester name, report import times, status (unprocessed, importing, import failed, import successful), import information (record import failure error exception information), creation time, and last update time.

[0064] 2. If the report task is configured for quasi-real-time import, the API interface provided by the report service is called synchronously.

[0065] Report import service: A new report import service is added, which is responsible for all large data report import functions of the system.

[0066] Import mode: quasi-real-time import and scheduled task import.

[0067] 1.1 Quasi-real-time import: Receive the APP service report export batch parameters, obtain the import task table data and report task configuration table data, determine the current import task status (because the scheduled task may be executed first), and skip if it is in the import process and return success directly. Otherwise, return specific error information according to the processing result.

[0068] 1.2. Scheduled task import: Get the import task table data and report task configuration table data, determine the current import task status, if it is in the import process, skip it and return success directly. Otherwise, return specific error information according to the processing result and record the error information in the import task table.

[0069] Thread pool component: A new thread pool component is added, which is responsible for concurrent execution of multiple tasks and supports arbitrary adjustment of the thread pool size, such as Figure 4 shown.

[0070] Export function component: Migrate the existing online loan import function to the report and upgrade the import technology component to Ali EasyPoi component. Export service and support exporting prompt excl for failed import and verification.

[0071] Beneficial effects: The entire project is developed based on spring cloudAlibaba, which facilitates distributed expansion;

[0072] In terms of technology selection, Alibaba's easyExcl and easyPoi import (export) methods are used, which occupy less system memory;

[0073] Use thread pool technology to achieve multi-threaded and high-concurrency import and export functions;

[0074] Deploy dedicated servers without consuming business server memory;

[0075] Supports thread pool, form, and formula class configuration operations.

[0076] The above specific implementation methods further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A file task processing method supporting distributed configuration functions, It is characterized in that The file task processing method comprises: Convert business processing from synchronous to asynchronous mode; Implement configurable export templates and data sets to dynamically process excl header data; Export uses calculation formulas within and between tables to optimize the calculation logic of EXCL total columns and inter-table data. Create multi-tool export configurations.

2. A file task processing method supporting distributed configuration functions according to claim 1, It is characterized in that The conversion of business processing from synchronous mode to asynchronous mode specifically includes: All report requests are transformed into asynchronous mode. When there is a report export request, the current export report configuration information is obtained, and the export task table data is saved according to the configuration information; If the report task is configured for quasi-real-time export, the API interface provided by the report service is called synchronously.

3. A file task processing method supporting distributed configuration functions according to claim 2, It is characterized in that The task table data includes: report ID, report number, report name, export file name, report type, report export method, export request parameters, export batch, report export requester number, report export requester name, report export times, status, export information, creation time and last update time.

4. A file task processing method supporting distributed configuration functions according to claim 3, It is characterized in that The statuses include: unprocessed, exporting, export failed, and export successful.

5. A file task processing method supporting distributed configuration functions according to claim 1, It is characterized in that The file task processing method also includes: Added a new report export service, which is responsible for all large data report export functions of the system.

6. A file task processing method supporting distributed configuration functions according to claim 3, It is characterized in that The report export method specifically includes: quasi-real-time export and scheduled task export; The quasi-real-time export is to receive the APP service report export batch parameters, obtain the export task table data and the report task configuration table data, determine the current export task status, and if it is in the export process, skip it and directly return success; otherwise, return specific error information according to the processing result; The scheduled task export is as follows: obtaining the export task table data and the report task configuration table data, determining the current export task status, if it is in the exporting process, skipping it and directly returning success; otherwise, returning specific error information according to the processing result, and recording the error information in the export task table.

7. A file task processing method supporting distributed configuration functions according to claim 1, It is characterized in that The establishment of the multi-tool export configuration specifically includes: multiple thread pool components and export function components.

8. A file task processing method supporting distributed configuration functions according to claim 7, It is characterized in that The multiple thread pool components include: a newly added thread pool component, which is responsible for concurrent execution of multiple tasks and supports arbitrary adjustment of the thread pool size; Scheduled tasks support configurable days to clean up generated export files; Manually trigger garbage collection.

9. A file task processing method supporting distributed configuration functions according to claim 7, It is characterized in that The export function component includes: migrating the existing system export function to the report export service, and upgrading the export technology component.

Images