A method to improve the batch processing execution efficiency of spring-batch framework

Abstract

The present invention relates to a method for improving the batch processing execution efficiency of spring-batch framework, which is based on file thread database cache interface, creates thread sharding file, then loads and writes the number of paging entries, splits a single large file or a large result set into small files or small result sets according to rules for subsequent processing; developers use business reader interface to write query / read logic according to business needs, loads pending data, and transfers pending data to processor layer; uses file cache entity interface to configure single cache number, reads corresponding row data from thread sharding file at one time, converts it into row data entity, and writes it into redis cache; uses file reading interface to configure basic information of reading file; creates spring-batch framework public class through Reader interface of springframework. Compared with the prior art, the present invention has the advantages of greatly improving batch execution efficiency, reducing business risks of large data volume and high data timeliness, etc.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method for improving the batch processing execution efficiency of a spring-batch framework. Background Art

[0002] In modern enterprise applications, in the face of complex business and massive amounts of data, in addition to various processing through complex human-computer interaction interfaces, there is another processing method, namely batch processing. Batch processing does not require manual intervention, but only requires regular reading of large batches of data, and then completing the corresponding business processing and archiving. Batch processing is an indispensable data processing method in modern enterprise applications, and how to efficiently execute batch processing tasks has become a problem.

[0003] At present, when distributed systems use multithreading, the processing flow of the existing batch processing architecture is to read all the required processing data through the reader layer and generate shard thread files. Each piece of data is obtained separately and transferred to the processor layer for business logic, and then the results are output through the writer until all the data are processed. However, the main disadvantage of this method is that the batch execution time is long when processing large amounts of data, the resource consumption is large, and there is a certain risk for time-sensitive businesses. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a method for improving the batch execution efficiency of the spring-batch framework. The method can greatly improve the batch execution efficiency, reduce the business risks of large data volume and high data timeliness, carry more business functions in a fixed time period, and improve production capacity output.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A method for improving the batch processing execution efficiency of the spring-batch framework, the specific contents of the method are:

[0007] Based on the file thread database cache interface, create a thread shard file, then load and write the paging entries, and split the single large file or large result set into small files or small result sets according to the rules for subsequent processing;

[0008] Developers use the business reader interface to write query / reading logic according to business needs, load the data to be processed through the file thread database cache interface, and transfer the data to be processed to the processor layer;

[0009] Use the file cache entity interface to configure the single cache number, read the corresponding row data from the thread shard file at one time, convert it into row data entities, and write it to the redis cache;

[0010] Use the file reading interface to configure the basic information of the read file;

[0011] Create the spring-batch framework public class through the springframework's Reader interface.

[0012] Furthermore, the basic information of the read file includes but is not limited to a separator, a character encoding, a line break, and a number of skipped lines.

[0013] Furthermore, the file thread database cache interface uses Spring's TaskExecutor to create thread shard files.

[0014] Furthermore, the file thread database cache interface loads and writes the number of paging entries through the batchKeyLoadPageSize parameter, and splits a single large file or a large result set into small files according to the rules.

[0015] Furthermore, the large file is a file larger than a preset size, the small file is a file smaller than or equal to a preset size. The large result set is a result set larger than or equal to a preset result set size threshold, and the small result set is a result set smaller than the result set size threshold.

[0016] Furthermore, the file cache entity interface provides a cacheLineNum parameter to configure the number of single caches.

[0017] Furthermore, the file cache entity interface reads the corresponding row data from the thread shard file at one time, converts it into row data entity through the transform method, and writes it into the redis cache.

[0018] The method for improving the batch processing execution efficiency of the spring-batch framework provided by the present invention has at least the following beneficial effects compared with the prior art:

[0019] 1) The present invention pre-caches the batch data that needs to be processed, and a single interaction can process multiple data, so as to reduce the number of data scanning and database interactions, thereby reducing the total number of shards and thread processing, and shortening the system batch processing execution time from the overall level; reducing the consumption of resources, avoiding the impact of timeliness on the business as much as possible, and improving production capacity output.

[0020] 2) The method of the present invention can be used at both the database level and the file level, and can cover most batch processing scenarios and unify batch processing modes and processes. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram of element relationships of a method for improving the batch processing execution efficiency of the spring-batch framework in an embodiment;

[0022] Figure 2 The figure is a logical flow diagram of a method for improving the batch processing execution efficiency of the spring-batch framework in an embodiment. DETAILED DESCRIPTION

[0023] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work should belong to the scope of protection of the present invention.

[0024] Example

[0025] The present invention relates to a method for improving the batch processing execution efficiency of a spring-batch framework. The basic concepts and principles of the spring-batch framework are as follows: spring-batch is an open source large data volume parallel processing framework. Lightweight and robust parallel processing applications can be built through spring-batch, which supports transactions, concurrency, processes, and monitoring, and provides unified interface management and task management.

[0026] A key issue encountered when using spring batch for data migration is how to ensure memory when the amount of data to be migrated is large. When using spring batch, three things must be configured: reader, processor, and writer. The reader is used to read data from the database. When the amount of data is small, the logic of the reader will not bring too much pressure on the memory, but when the amount of data to be read is very large, you have to consider issues such as memory, because if the amount of data is very large, memory, execution time, etc. will be affected. Writer is used to write data from the Spring Batch application to a specific destination. The processor is a class that contains the processing code for processing the data read into the spring batch. If the application reads n records, the code in the processor will be executed on each record.

[0027] The element relationship according to which the method for improving the batch processing execution efficiency of the spring-batch framework is provided in the present invention is as follows: Figure 1 As shown, where:

[0028] JOB: is the core concept of the spring-batch framework. It includes all operations of batch processing and is a batch task.

[0029] STEP: Each JOB consists of one or more STEPs, which are the task steps in a batch task; STEP is connected to the reader interface, processor interface, and writer interface respectively.

[0030] Reader: data source reading interface;

[0031] processor: business logic processing interface;

[0032] writer: the processed data output interface.

[0033] based on Figure 1 The present invention creates a unified and standardized batch processing public class for independent business readers to inherit. The logic of the method uses cache. The actual process is as follows Figure 2 As shown, where:

[0034] Business reader: Developers only need to write query / read logic according to business needs and transfer the data to be processed to the processor layer. The data to be processed is loaded and processed by the following reader.

[0035] FileThreadDBCacheReader: File thread database cache interface, uses spring's TaskExecutor to create thread shard files, then loads and writes the number of paging entries through the batchKeyLoadPageSize parameter, and splits individual large files or large result sets into small files or small result sets according to the rules for subsequent processing. It should be noted that large files are files larger than the preset size, and small files are files smaller than or equal to the preset size. When judging whether a result set is a large result set or a small result set, if the result set is larger than or equal to the preset result set size threshold, the result set is a large result set; if the result set is smaller than the result set size threshold, the result set is a small result set.

[0036] AbsFileCacheEntityReader: File cache entity interface, provides the cacheLineNum parameter to configure the number of single caches, reads the corresponding row data from the thread shard file at one time, converts it into row data entity through the transform method, and writes it to the redis cache.

[0037] AbsFileReader: File reading interface, configures basic information of the read file, such as: delimiter, character encoding, line break, number of skipped lines, etc.

[0038] XXXReader of springframework: public class of spring-batch framework.

[0039] In summary, the present invention performs pre-caching processing on the data to be processed, and a single interaction can process multiple data, which can reduce the number of data scans and database interactions, thereby reducing the total number of sharding and thread processing. It can be used at both the database level and the file level, covering most batch processing scenarios, unifying batch processing modes and processes, and shortening the system batch processing execution time from an overall level. In actual use, the present invention can shorten the original system account processing batch (taking this batch as an example) from 3 hours to 1 hour, which includes the processing of billions of data and involves database operations of multiple billions of large tables, greatly shortening the batch processing time, reducing resource consumption, and avoiding the impact of timeliness on business as much as possible, and improving production capacity output.

[0040] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person familiar with the technical field can easily think of various equivalent modifications or substitutions within the technical scope disclosed by the present invention, and these modifications or substitutions should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for improving the batch processing execution efficiency of the spring-batch framework, characterized in that: include: Based on the file thread database cache interface, create a thread shard file, then load and write the paging entries, and split the single large file or large result set into small files or small result sets according to the rules for subsequent processing; Use the business reader interface to write query / reading logic according to business needs, load the data to be processed through the file thread database cache interface, and transfer the data to be processed to the processor layer; Use the file cache entity interface to configure the single cache number, read the corresponding row data from the thread shard file at one time, convert it into row data entities, and write it to the redis cache; Use the file reading interface to configure the basic information of the read file; Create the spring-batch framework public class through the springframework's Reader interface; The file cache entity interface provides a cacheLineNum parameter to configure the number of single caches; The file cache entity interface reads the corresponding row data from the thread shard file at one time, converts it into row data entity through the transform method, and writes it into the redis cache.

2. The method for improving the batch processing execution efficiency of the spring-batch framework according to claim 1 is characterized in that: The file thread database cache interface uses spring's TaskExecutor to create thread shard files.

3. The method for improving the batch processing execution efficiency of the spring-batch framework according to claim 2 is characterized in that: The file thread database cache interface loads and writes the number of paging entries through the batchKeyLoadPageSize parameter, splitting a single large file or a large result set into small files according to rules.

4. The method for improving the batch processing execution efficiency of the spring-batch framework according to claim 3 is characterized in that: The large file is a file larger than a preset size, and the small file is a file smaller than or equal to a preset size.

5. The method for improving the batch processing execution efficiency of the spring-batch framework according to claim 3 is characterized in that: The large result set is a result set that is greater than or equal to a preset result set size threshold, and the small result set is a result set that is less than the result set size threshold.

6. The method for improving the batch processing execution efficiency of the spring-batch framework according to claim 1 is characterized in that: The basic information of the read file includes but is not limited to the separator, character encoding, line break, and number of skipped lines.

Images