General multi-format file generation method and system based on multi-thread filter processing

Abstract

The application relates to a general multi-format file generation method and system based on a multi-thread filter process, which comprises the following steps: configuring a plurality of independent data acquisition threads and data processing threads, calling all the data acquisition threads according to a judgment result, decomposing a file generation request into a plurality of sub-requests according to thread load, and correspondingly distributing the sub-requests to each data acquisition thread to execute data acquisition operation and subsequent file processing operation, and generating a file item corresponding to to-be-processed data; complex flow and application logic development is converted into configuration file configuration and database configuration, development work and operation and maintenance cost are extremely simplified, thread implementation rules and filter implementation rules can be added as required, the expansibility is extremely strong, the method is suitable for a business system with multiple file generation rules, non-uniform requirements and a large number of file generation demands, and the generation and sending of an account file can be completed only by simple configuration.

Description

Technical Field

[0001] This invention relates to the fields of software system design and big data processing technology, and in particular to a general method and system for generating multi-format files based on multi-threaded filter processing. Background Technology

[0002] Modern electronic services largely rely on data files. For example, payment and collection systems in banking and financial services need to interface with specific functional systems of different local hospitals, utility companies, social security agencies, mobile payment platforms, and lottery sales for payment and end-of-day reconciliation. These end-of-day reconciliations require document-based processing. In practice, because each external system has different requirements regarding the range of data to be queried, the content and style of the documents to be displayed, and the encoding format, current technology necessitates one-to-one development tailored to specific project needs, resulting in a large workload and hindering future maintenance.

[0003] To address the aforementioned application needs, existing technologies have attempted to provide general file generation solutions by developing file generation logic. However, in practical applications, different file formats have different structures and encoding methods, requiring different parsing technologies to read and parse file content. For some uncommon file formats, special parsing libraries or tools may be needed. Moreover, due to the diverse and inconsistent rules for generating files, the system may need to spend more time and computing resources to meet various rules and requirements. This can lead to low processing efficiency and prolong the file generation cycle. For data transmission needs of different target systems, the generated file rules are numerous, and the system may need to support multiple file formats. Different file formats may have different encoding methods, structures, and usage specifications. If it is necessary to convert files to other formats, appropriate conversion tools or libraries are required. However, some file formats may not support all functions, or the converted file may lose some information or have format errors. This can lead to reading or conversion failures due to the different encoding methods and structures of different file formats, requiring additional format conversion and adaptation processing. Furthermore, different devices may have different effects on file display and storage, requiring compatibility processing. As the number of supported target systems increases, the scale of the corresponding generated rules will also grow significantly. For business systems with a large number of generated rules, how to effectively manage and maintain these rules is a key point. Targeted configuration is required to provide flexible rule configuration and management functions so that rules can be easily added, modified, or deleted to meet ever-changing business needs.

[0004] On the other hand, when generating files in batches, a large number of files need to be processed. The system may need to handle a large number of data streams and concurrent operations, which may lead to performance problems or errors. Therefore, it is necessary to have redundant performance and scalability to cope with large-scale file generation tasks. The system needs to have additional data verification and validation mechanisms to ensure that the generated files contain accurate and reliable data, but this will have a negative impact on the need to improve processing efficiency and reduce processing time.

[0005] Therefore, it can be seen that traditional solutions for file generation are basically in a scattered development state, requiring the development of a set of related file generation logic for each specific business requirement; the current attempts to make multi-file generation universal are limited to adding and improving code logic, such as adding multiple if branches to implement the corresponding generation logic selection function, resulting in a large amount of code development and poor readability and maintainability. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes a general multi-format file generation method and system based on multi-threaded filter processing. This method transforms complex process and application logic development into configuration file and database configuration, greatly simplifying development work and reducing maintenance costs. Threads and filters can be added as needed to implement rules and implement rules, making it highly scalable. It is suitable for business systems with many file generation rules, inconsistent requirements, and a large number of file generation needs. Only simple configuration is required to complete the generation and sending of reconciliation files.

[0007] To achieve the above objectives, the technical solution adopted by the present invention includes:

[0008] A general multi-format file generation method based on multi-threaded filter processing, characterized by comprising:

[0009] S1. Configure several independent data acquisition threads and data processing threads respectively;

[0010] S2. Obtain the file generation request and determine whether the file generation request calls multiple data acquisition threads based on the preset first matching condition;

[0011] S3. When it is determined that the file generation request calls multiple data acquisition threads, all data acquisition threads are called and the file generation request is decomposed into several sub-requests according to the thread load and assigned to each data acquisition thread to perform data acquisition operations, generating independent data to be processed respectively.

[0012] S4. Assign a data processing thread to the data to be processed. The data processing thread performs file processing operations on one or more data to be processed according to one or more preset filter rules, and generates file items corresponding to the data to be processed.

[0013] S5. Use the file item to generate the corresponding file generation request data message and send back the file generation request.

[0014] Furthermore, step S2 also includes:

[0015] When it is determined that the file generation request does not invoke multiple data acquisition threads, one available data acquisition thread is selected based on the thread load to perform the data acquisition operation and generate the data to be processed.

[0016] Furthermore, the first matching condition includes any one or more combinations of data entry number threshold comparison, specified data item identification, and data period range threshold comparison.

[0017] Furthermore, the allocation of data processing threads for the data to be processed includes:

[0018] Determine whether the filter rules invoked for multiple pieces of data to be processed are the same;

[0019] When it is determined that multiple pieces of data to be processed correspond to the same filter rule, the multiple pieces of data to be processed are assigned to the same data processing thread.

[0020] Furthermore, the filter rules include file naming filters, file generation filters, file sending filters, file addition filters, file compression filters, and file sorting filters.

[0021] Furthermore, the step of generating the data message corresponding to the file generation request using the file item includes:

[0022] Identify the corresponding message transaction code based on the file generation request;

[0023] Obtain the corresponding configuration template by matching the message transaction code;

[0024] Data packets are generated by inserting file items using a configuration template. The naming of the data packets includes a message transaction code and / or a corresponding identification field indicating the message transaction code.

[0025] This invention also relates to a general multi-format file generation system based on multi-threaded filter processing, characterized in that it includes:

[0026] The thread management module is used to configure several independent data acquisition threads and data processing threads.

[0027] The thread determination module is used to determine whether the file generation request calls multiple data acquisition threads based on a preset first matching condition.

[0028] The data acquisition module is used to decompose the file generation request into several sub-requests based on the thread load, and to call the data acquisition thread to perform data acquisition operations.

[0029] The file processing module is used to call the data processing thread to perform file processing operations on one or more data to be processed according to one or more preset filter rules, and generate file items for the corresponding data to be processed.

[0030] The message feedback module is used to generate a data message for the corresponding file generation request using the file item and to send back the file generation request.

[0031] The present invention also relates to a computer-readable storage medium, characterized in that the storage medium stores a computer program, which, when executed by a processor, implements the above-described method.

[0032] The present invention also relates to an electronic device, characterized in that it includes a processor and a memory;

[0033] The memory is used to store data to be processed and file items;

[0034] The processor is configured to execute the above-described method by invoking the data to be processed and the file item.

[0035] The present invention also relates to a computer program product, including a computer program and / or instructions, characterized in that the computer program and / or instructions, when executed by a processor, implement the steps of the above-described method.

[0036] The beneficial effects of this invention are as follows:

[0037] The general multi-format file generation method and system based on multi-threaded filter processing described in this invention transforms complex process and application logic development into configuration file and database configuration, greatly simplifying development work and reducing maintenance costs. It makes the file generation steps pluggable, providing common method implementations for each stage. Thread implementation rules and filter implementation rules can be added as needed, resulting in strong scalability. Subsequent file generation only requires modification of the configuration file, database configuration, and file template configuration to complete the generation of relevant files. It is suitable for business systems with many file generation rules, inconsistent requirements, and a large number of file generation needs. Simple configuration is all that's needed to complete the generation and sending of reconciliation files, improving system maintainability and reducing upgrade and maintenance costs. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the general multi-format file generation method based on multi-threaded filter processing according to the present invention.

[0039] Figure 2 This is a schematic diagram of the general multi-format file generation system based on multi-threaded filter processing according to the present invention. Detailed Implementation

[0040] To better understand the content of this invention, it will be described in detail with reference to the accompanying drawings and embodiments.

[0041] The first aspect of this invention relates to a process flow as follows: Figure 1 The general multi-format file generation method based on multi-threaded filter processing shown includes:

[0042] S1. Configure several independent data acquisition threads and data processing threads respectively.

[0043] Preferably, multi-threaded common processing logic plugins can be developed as needed for data acquisition in various ways.

[0044] Specifically, for multiple existing configurations, this may include: designing the architecture of data acquisition threads and data processing threads, determining the role and task of each thread, and how threads communicate and collaborate with each other; creating data acquisition threads, writing code to implement the functionality of each data acquisition thread, which should be able to acquire data from different data sources and pass the data to the data processing threads; creating data processing threads, writing code to implement the functionality of each data processing thread, which should be able to receive data from the data acquisition threads and process, analyze, and store it; developing a multi-threaded common processing logic plugin, developing a general plugin as needed that can be used for various data acquisition methods, the plugin should be able to interact with the data acquisition threads and receive and process data; configuring thread pools, creating thread pools for data acquisition threads and data processing threads to effectively manage and schedule these threads.

[0045] S2. Obtain the file generation request and determine whether the file generation request calls multiple data acquisition threads based on a preset first matching condition. Specifically, the first matching condition includes any one or more combinations of data item quantity threshold comparison, specified data item identification, and data period range threshold comparison.

[0046] Preferably, when it is determined that the file generation request does not invoke multiple data acquisition threads, one available data acquisition thread is selected based on the thread load to perform the data acquisition operation and generate the data to be processed.

[0047] Specifically, the process involves receiving a file generation request and extracting the data items to be collected; judging the data items to be collected based on a preset first matching condition; if the first matching condition is met, a new data collection thread is created and the data items to be collected are passed to that thread; if the first matching condition is not met, only a default data collection thread is used for processing; the above steps are repeated until all the data items to be collected have been processed.

[0048] S3. When it is determined that the file generation request calls multiple data acquisition threads, all data acquisition threads are called and the file generation request is decomposed into several sub-requests according to the thread load and assigned to each data acquisition thread to perform data acquisition operations, thereby generating independent data to be processed.

[0049] When implementing data collection, a general data acquisition method can be selected as needed for most projects; a personalized data acquisition method can be applied when data acquisition rules are specified in the configuration file; or an upper limit for the number of details per page can be specified and paginated files can be generated. Preferably, regardless of the specific data acquisition method used, aggregated data parameters can be configured as needed, such as obtaining the total aggregated amount and total number of transactions, as well as specifying specific conditions for the size of individual files.

[0050] S4. Assign a data processing thread to the data to be processed. The data processing thread performs file processing operations on one or more pieces of data to be processed according to one or more preset filter rules, generating file items corresponding to the data to be processed. The filter rules include file naming filter, file generation filter, file sending filter, file adding filter, file compression filter, and file sorting filter.

[0051] Preferably, the file naming filter allows you to specify file naming rules using the `rule` tag, supporting both dynamic and static value generation; the file generation filter allows you to specify the file encoding format using the `rule` tag, specifying UTF-8 or GBK; the file addition filter allows you to supplement the data retrieval statement for adding a new file using the `sql` tag; the file compression filter allows you to specify the file compression format using the `rule` tag, supporting tar and zip; and the file sorting filter allows you to specify the sorting field and ascending / descending order using the `rule` tag.

[0052] Preferably, allocating data processing threads to the data to be processed includes: determining whether the filter rules called for multiple data to be processed are the same; when it is determined that the filter rules called for multiple data to be processed are the same, allocating the multiple data to be processed to the same data processing thread.

[0053] S5. Use the file item to generate the corresponding file generation request data message and send back the file generation request.

[0054] In practice, this may include identifying the corresponding message transaction code based on the file generation request; obtaining the corresponding configuration template by matching the message transaction code; and using the configuration template to insert file items to generate data messages, wherein the naming of the data messages includes the message transaction code and / or a corresponding identification field indicating the message transaction code.

[0055] Preferably, data message generation is achieved by configuring three independent attributes of the `rcncl` tag: `txcode`, `type`, and `convert`. `txcode` corresponds to the processing rule code, which must have an execution thread corresponding to `exce` in the `thread-mapping`. The `type` attribute can take the following values: `file` (file processing type), `print` (print file processing type), `callback` (callback message processing type), `request` (request message processing type), and `response` (response message processing type). The `convert` attribute splits the file based on a specified column. A value of `default` means no splitting; a value with the field name in uppercase means splitting the file based on database query data (if a sub-unit has no transactions, no file is generated). A value of `CONVERT` generates a file for the region code (sub-unit) configured in the database (if the sub-unit has no transaction data), and a value of `CONVERT2` generates a file for the region code (sub-unit) in the database account settings table (if the sub-unit has no transactions, an empty file is generated).

[0056] The following specific example, taken within the context of a bank's reconciliation business, further illustrates the execution process of the above method.

[0057] Taking a branch as an example, unit reconciliation is typically an automatically initiated task at the end of the day. The task parameters are the first-level branch number and the accounting date. Based on these parameters, the system first queries the database to retrieve the configuration information of all projects that need to participate in unit reconciliation that day, including transaction rule codes, outbound call interfaces, template numbers, and project numbers. Multiple projects are generated in parallel using a multi-threaded approach. When each project is executed, it first finds the multi-threaded file generation data retrieval rule in the configuration file based on the transaction rule code to obtain the required business data. For example, for T0001, it retrieves the business transaction log data of all successful transactions for that project that day. This data is then placed in a map, and then... Then, the corresponding file filter rule is found based on the transaction rule code. It usually contains three rules. The first is the F_N0001 file naming rule, which generates the corresponding file name according to the rule parameter configuration. The second is the F_B0001 file generation rule, which generates the corresponding file according to the personalized configuration template configured in the database based on the file generation format (such as UTF-8) parameter obtained by the thread rule. The third is the F_S0001 message sending rule, which sends the file name, file path and other information of the generated file to the third party in the form of a message based on the outbound call transaction message code obtained by the transaction rule code.

[0058] Another aspect of the present invention relates to a general multi-format file generation system based on multi-threaded filter processing, the structure of which is as follows: Figure 2 As shown, it includes:

[0059] The thread management module is used to configure several independent data acquisition threads and data processing threads.

[0060] The thread determination module is used to determine whether the file generation request calls multiple data acquisition threads based on a preset first matching condition.

[0061] The data acquisition module is used to decompose the file generation request into several sub-requests based on the thread load, and to call the data acquisition thread to perform data acquisition operations.

[0062] The file processing module is used to call the data processing thread to perform file processing operations on one or more data to be processed according to one or more preset filter rules, and generate file items for the corresponding data to be processed.

[0063] The message feedback module is used to generate a data message for the corresponding file generation request using the file item and to send back the file generation request.

[0064] By using this system, the aforementioned computational processing methods can be executed and the corresponding technical effects can be achieved.

[0065] Embodiments of the present invention also provide a computer-readable storage medium capable of implementing all the steps of the methods in the above embodiments, wherein the computer-readable storage medium stores a computer program that, when executed by a processor, implements all the steps of the methods in the above embodiments.

[0066] Embodiments of the present invention also provide an electronic device for performing the above-described method. As an implementation device for the method, the electronic device includes at least a processor and a memory. In particular, the memory stores data and related computer programs required for performing the method, such as data to be processed and file items. The processor calls the data and programs in the memory to execute all the steps of the method and obtain the corresponding technical effect.

[0067] Preferably, the electronic device may include a bus architecture, which may include any number of interconnected buses and bridges. The bus will include various circuits linked together by one or more processors and memories. The bus may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface between the bus and the receiver and transmitter. The receiver and transmitter may be the same element, i.e., a transceiver, providing a unit for communicating with various other systems over a transmission medium. The processor is responsible for managing the bus and general processing, while the memory may be used to store data used by the processor during operation.

[0068] Additionally, the electronic device may further include components such as a communication module, an input unit, an audio processor, a display, and a power supply. The processor (or controller, operating control) used may include a microprocessor or other processor device and / or logic device, which receives input and controls the operation of various components of the electronic device; the memory may be one or more of a buffer, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices, which can store the aforementioned data information, and may also store programs for executing the information, and the processor can execute the program stored in the memory to achieve information storage or processing, etc.; the input unit is used to provide input to the processor, for example, it can be a button or touch input device; the power supply is used to provide power to the electronic device; the display is used to display images and text, for example, it can be an LCD display. The communication module is a transmitter / receiver that transmits and receives signals via an antenna. The communication module (transmitter / receiver) is coupled to the processor to provide input signals and receive output signals, which can be the same as in conventional mobile communication terminals. Based on different communication technologies, multiple communication modules can be incorporated into the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication module (transmitter / receiver) is also coupled to a speaker and microphone via an audio processor to provide audio output through the speaker and receive audio input from the microphone, thereby enabling typical telecommunications functions. The audio processor can include any suitable buffer, decoder, amplifier, etc. Furthermore, the audio processor is coupled to a central processing unit, enabling on-device recording via the microphone and on-device playback of stored sound via the speaker.

[0069] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0070] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.

[0071] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction set implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0072] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the functions specified in one or more boxes. Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0073] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A general multi-format file generation method based on multi-threaded filter processing, characterized in that, include: S1. Configure several independent data acquisition threads and data processing threads respectively; S2. Obtain the file generation request and determine whether the file generation request calls multiple data acquisition threads based on the preset first matching condition. If the first matching condition is met, create a new data acquisition thread and pass the data items to be collected to the thread. When it is determined that the file generation request does not call multiple data acquisition threads, select an idle data acquisition thread according to the thread load to perform data acquisition operations and generate data to be processed. Repeat the above steps until all the data items that need to be collected have been processed; S3. When it is determined that the file generation request calls multiple data acquisition threads, all data acquisition threads are called and the file generation request is decomposed into several sub-requests according to the thread load and assigned to each data acquisition thread to perform data acquisition operations, generating independent data to be processed respectively. S4. Assign a data processing thread to the data to be processed. The data processing thread performs file processing operations on one or more data to be processed according to one or more preset filter rules, and generates file items corresponding to the data to be processed. S5. Use the file item to generate the corresponding file generation request data message and send back the file generation request.

2. The method as described in claim 1, characterized in that, The first matching condition includes any one or more combinations of data entry number threshold comparison, specified data item identification, and data period range threshold comparison.

3. The method as described in claim 1, characterized in that, The data processing thread allocated to the data to be processed includes: Determine whether the filter rules invoked for multiple pieces of data to be processed are the same; When it is determined that multiple pieces of data to be processed correspond to the same filter rule, the multiple pieces of data to be processed are assigned to the same data processing thread.

4. The method as described in claim 1, characterized in that, The filter rules include file naming filters, file generation filters, file sending filters, file addition filters, file compression filters, and file sorting filters.

5. The method as described in claim 1, characterized in that, The process of generating the corresponding file generation request data message using the file item includes: Identify the corresponding message transaction code based on the file generation request; Obtain the corresponding configuration template by matching the message transaction code; Data packets are generated by inserting file items using a configuration template. The naming of the data packets includes a message transaction code and / or a corresponding identification field indicating the message transaction code.

6. A general-purpose multi-format file generation system based on multi-threaded filter processing, characterized in that, include: The thread management module is used to configure several independent data acquisition threads and data processing threads. The thread judgment module is used to determine whether the file generation request calls multiple data acquisition threads based on a preset first matching condition. If the first matching condition is met, a new data acquisition thread is created and the data items to be collected are passed to that thread. When it is determined that the file generation request does not call multiple data acquisition threads, an idle data acquisition thread is selected according to the thread load to perform the data acquisition operation and generate the data to be processed. Repeat the above steps until all the data items that need to be collected have been processed; The data acquisition module is used to decompose the file generation request into several sub-requests based on the thread load, and to call the data acquisition thread to perform data acquisition operations. The file processing module is used to call the data processing thread to perform file processing operations on one or more data to be processed according to one or more preset filter rules, and generate file items for the corresponding data to be processed. The message feedback module is used to generate a data message for the corresponding file generation request using the file item and to send back the file generation request.

7. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method of any one of claims 1 to 5.

8. An electronic device, characterized in that, Including processor and memory; The memory is used to store data to be processed and file items; The processor is configured to execute the method of any one of claims 1 to 5 by invoking data to be processed and file items.

9. A computer program product, comprising a computer program and / or instructions, characterized in that, When the computer program and / or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 5.

Images