Method for quickly generating multi-format file based on database large data volume table

Abstract

The invention provides a method for quickly generating a multi-format file based on a large data volume table of a database, and belongs to the field of file generation. The method comprises the following steps: when a file generation request is detected, starting a main thread to assemble a context environment of a file generation task, and initializing a producer thread, a consumer thread and a file generation thread based on the context environment; calling a producer thread to perform streaming query on a database, and pushing target data obtained by the streaming query to a message queue; calling a consumer thread, pulling the target data from the message queue in batches, generating a file generation task according to the target data, and submitting the task to a file generation thread; the file generation thread generates a target file in a corresponding format according to the file generation task, and stores the target file in a temporary directory; all the generated target files are compressed through the main thread, and the compressed files are uploaded to a data medium table. According to the method, the generation efficiency of the large-data-volume file can be improved, and the consumption of memory resources is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of file generation, and in particular to a method for rapidly generating multi-format files based on a large data volume table in a database. Background Art

[0002] With the emergence of the data middle platform, centralized management and integration of data assets have been achieved, the problem of data silos has been solved, and data from various business systems and departments have been centralized to form a unified data view, making it easier for various departments within the enterprise to obtain, share and utilize data resources.

[0003] However, when some tables with relatively large amounts of data need to be shared, they are generally downloaded in the form of library table synchronization or files. Traditional file generation methods often face problems such as low efficiency and high resource consumption, and are difficult to meet the high-speed and high-efficiency requirements of modern data processing. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a method for quickly generating multi-format files based on a large database table, so as to solve the problem that the existing file generation method is inefficient and consumes a lot of resources.

[0005] The technical solution adopted by the present invention is: The present invention provides a method for quickly generating multi-format files based on a large data table in a database, comprising: A file generation request is detected, the main thread is started to assemble the context environment of the file generation task, and the producer thread, consumer thread and file generation thread are initialized based on the context environment; Call the producer thread to perform streaming query on the database, and push the target data obtained by the streaming query to the message queue; Call the consumer thread, pull the target data in batches from the message queue, generate a file generation task based on the target data, and submit the file generation task to the file generation thread; The file generation thread generates a target file of a corresponding format according to the file generation task, and saves the target file to a temporary directory; All generated target files are compressed through the main thread, and the compressed files are uploaded to the data center.

[0006] Further, the detecting of the file generation request, starting the main thread to assemble the context environment of the file generation task, and initializing the producer thread, the consumer thread and the file generation thread based on the context environment, includes: When a file generation request sent by a user is detected, the file generation request is parsed to obtain database information, message queue information, a temporary directory for storing files, a data batch writing threshold, and a generated file format; Start the main thread, encapsulate the database information, message queue information, temporary directory for file storage, data batch write threshold and generated file format, and obtain the context environment of the file generation task; Initialize multiple producer threads based on the database information and message queue information in the context environment to form a producer thread pool; Obtain the data batch write threshold and message queue information from the context environment and initialize multiple consumer threads to form a consumer thread pool; The temporary directory where files are stored and the format of the generated files are read from the context environment and multiple file generation threads are initialized to obtain a file generation thread pool.

[0007] Furthermore, the calling of the producer thread to perform streaming query on the database and push the target data obtained by the streaming query to the message queue includes: Get database information from the context and create a database connection; Call the producer thread from the producer thread pool, access the database based on the database connection, perform streaming queries on the database, and determine whether the streaming query results contain the target data. If so, push the target data to the message queue. If not, close the database connection, encapsulate the end flag and push it to the message queue. The database streaming query ends and the producer thread is recycled.

[0008] Furthermore, the calling of the consumer thread, batch-pulling target data from the message queue and generating a file generation task according to the target data, and submitting the file generation task to the file generation thread, includes: Call the consumer thread from the consumer thread pool, monitor the message queue through the consumer thread, and pull the target data from the message queue when a message is pushed in the message queue; Determine whether the currently pulled data contains an end mark. If not, determine whether the pulled target data reaches the data batch write threshold. If so, write the currently pulled target data into the file generation task and submit the file generation task to the file generation thread pool. If not, continue to pull the target data from the message queue. If the currently pulled data contains an end mark, determine whether there is target data that does not reach the data batch write threshold. If there is corresponding target data, write the corresponding target data into the file generation task and submit it to the file generation thread pool. If there is no corresponding target data, end the data pulling operation and recycle the current consumer thread.

[0009] Furthermore, the file generation thread generates a target file of a corresponding format according to the file generation task, and saves the target file to a temporary directory, including: The file generation thread pool starts multiple file generation threads in sequence according to the received file generation tasks; The file generation thread generates a target file according to the file format corresponding to the file generation task and the target data carried by the file generation task, and saves the target file to a temporary directory, and at the same time recycles the file generation thread that has completed the file generation task.

[0010] In summary, the beneficial effects of the present invention are as follows: The method provided by the present invention for quickly generating multi-format files based on a large database table adopts a combination of batch processing, producer-consumer model, and multi-threaded tasks to generate files in multiple formats required by users, which can greatly improve the efficiency of generating large data files. At the same time, the thread uses a streaming reading method to read the data in the database, and does not load all the data into the memory at one time, which can reduce the resource occupancy rate and the frequency of triggering the garbage collection mechanism, and improve the stability of the file generation task. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the drawings required for use in the embodiment of the present invention will be briefly introduced below. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work, and these are all within the protection scope of the present invention.

[0012] Figure 1 A schematic flow chart of a method for rapidly generating multi-format files based on a large database table of data in the present invention; Figure 2 It is the main thread running flow chart of the present invention; Figure 3 It is a flow chart of producer thread operation of the present invention; Figure 4 A flow chart for the operation of the consumer thread of the present invention; Figure 5 A flow chart of the file generation thread operation of the present invention. DETAILED DESCRIPTION

[0013] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. If there is no conflict, the various features of the present invention and the embodiments can be combined with each other, all within the scope of protection of the present invention.

[0014] Example 1: Reference Figure 1 As shown, Figure 1 The present invention is a method for quickly generating multi-format files based on a large database table, and the method mainly includes the following steps: A file generation request is detected, the main thread is started to assemble the context environment of the file generation task, and the producer thread, consumer thread and file generation thread are initialized based on the context environment; Call the producer thread to perform streaming query on the database, and push the target data obtained by the streaming query to the message queue; Call the consumer thread, pull the target data in batches from the message queue, generate a file generation task based on the target data, and submit the file generation task to the file generation thread; The file generation thread generates a target file of a corresponding format according to the file generation task, and saves the target file to a temporary directory; All generated target files are compressed through the main thread, and the compressed files are uploaded to the data center.

[0015] The embodiment of the present invention adopts the idea of ​​splitting large file tasks into small file tasks, and cooperates with database streaming reading, producer consumer model, batch processing, multi-threading technology, and file compression upload to solve the problems of low file generation efficiency and high resource consumption under large data volumes.

[0016] Further, refer to Figure 1 As shown, when a file generation request is detected, the main thread is started to assemble the context environment of the file generation task, and the producer thread, consumer thread and file generation thread are initialized based on the context environment. The process specifically includes the following steps: When a file generation request sent by a user is detected, the file generation request is parsed to obtain database information, message queue information, a temporary directory for storing files, a data batch writing threshold, and a generated file format.

[0017] Start the main thread, encapsulate the database information, message queue information, temporary directory for file storage, data batch write threshold and generated file format, and obtain the context environment of the file generation task.

[0018] Among them, the above steps are the pre-processing of the entire file generation task, which forms the context environment of this task by packaging basic data such as database information, temporary file storage directory, data batch writing threshold, generated file type, etc.

[0019] Initialize multiple producer threads based on the database information and message queue information in the context environment to form a producer thread pool; Obtain the data batch write threshold and message queue information from the context environment and initialize multiple consumer threads to form a consumer thread pool; The temporary directory where files are stored and the format of the generated files are read from the context environment and multiple file generation threads are initialized to obtain a file generation thread pool.

[0020] Among them, the number of file generation threads in the file generation thread pool is set according to the required file format type. One file type corresponds to one file generation task. For example, txt, excel, json, pdf and other file formats correspond to four file generation tasks respectively.

[0021] Further, refer to Figure 3 As shown, the embodiment of the present invention calls the producer thread to perform streaming query on the database, and pushes the target data obtained by the streaming query to the message queue. The process specifically includes: Get database information from the context and create a database connection.

[0022] Call the producer thread from the producer thread pool, access the database based on the database connection, perform streaming queries on the database, and determine whether the streaming query results contain the target data. If so, push the target data to the message queue. If not, close the database connection, encapsulate the end flag and push it to the message queue.

[0023] Finally, the database streaming query ends and the producer thread is recycled.

[0024] In the embodiment of the present invention, data is read in a streaming reading manner instead of loading all data into the memory at one time, which can effectively reduce the frequency of Full GC, improve the stability of the program, and also reduce the occupation of memory resources.

[0025] Further, refer to Figure 4As shown, the embodiment of the present invention calls the consumer thread, pulls the target data in batches from the message queue, generates a file generation task according to the target data, and submits the file generation task to the file generation thread. The process specifically includes: The consumer thread is called from the consumer thread pool, and the message queue is monitored through the consumer thread. When a pushed message is detected in the message queue, the target data is pulled from the message queue.

[0026] Determine whether the currently pulled data contains an end mark. If not, determine whether the pulled target data reaches the data batch write threshold. If so, write the currently pulled target data into the file generation task and submit the file generation task to the file generation thread pool. If the data batch write threshold is not reached, continue to pull the target data from the message queue.

[0027] If the currently pulled data contains an end mark, determine whether there is target data that does not reach the data batch write threshold. If there is corresponding target data, write the corresponding target data into the file generation task and submit it to the file generation thread pool. If there is no corresponding target data, end the data pulling operation and recycle the current consumer thread.

[0028] In an embodiment of the present invention, a data batch write threshold is used to control the amount of data in a file generation task, thereby splitting a large file generation task into multiple small file generation tasks. This can prevent a single file from being monopolized by an application, and improve file generation efficiency by processing file generation tasks through multi-threading.

[0029] Further, refer to Figure 5 As shown, in the embodiment of the present invention, the file generation thread generates a target file of a corresponding format according to the file generation task, and saves the target file to a temporary directory. The process specifically includes: The file generation thread pool starts multiple file generation threads in sequence according to the received file generation tasks; The file generation thread generates a target file according to the file format corresponding to the file generation task and the target data carried by the file generation task, and saves the target file to a temporary directory, and at the same time recycles the file generation thread that has completed the file generation task.

[0030] In the embodiment of the present invention, through the above-mentioned file generation process, files of various formats can be generated, which reduces time cost and resource cost compared with traditional file generation methods. Under the same resource and time cost, the throughput of file generation tasks is greatly increased, solving the problems of low file generation efficiency and high resource consumption under large data volume.

[0031] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for quickly generating multi-format files based on a large database table, characterized in that: include: A file generation request is detected, the main thread is started to assemble the context environment of the file generation task, and the producer thread, consumer thread and file generation thread are initialized based on the context environment; Call the producer thread to perform streaming query on the database, and push the target data obtained by the streaming query to the message queue; Call the consumer thread, pull the target data in batches from the message queue, generate a file generation task based on the target data, and submit the file generation task to the file generation thread; The file generation thread generates a target file of a corresponding format according to the file generation task, and saves the target file to a temporary directory; All generated target files are compressed through the main thread, and the compressed files are uploaded to the data center.

2. The method for rapidly generating multi-format files based on a large database table according to claim 1, characterized in that: The method detects a file generation request, starts a main thread to assemble a context environment of the file generation task, and initializes a producer thread, a consumer thread, and a file generation thread based on the context environment, including: When a file generation request sent by a user is detected, the file generation request is parsed to obtain database information, message queue information, a temporary directory for storing files, a data batch writing threshold, and a generated file format; Start the main thread, encapsulate the database information, message queue information, temporary directory for file storage, data batch write threshold and generated file format, and obtain the context environment of the file generation task; Initialize multiple producer threads based on the database information and message queue information in the context environment to form a producer thread pool; Obtain the data batch write threshold and message queue information from the context environment and initialize multiple consumer threads to form a consumer thread pool; The temporary directory where files are stored and the format of the generated files are read from the context environment and multiple file generation threads are initialized to obtain a file generation thread pool.

3. The method for rapidly generating multi-format files based on a large database table according to claim 1, characterized in that: The calling producer thread performs streaming query on the database and pushes the target data obtained by the streaming query to the message queue, including: Get database information from the context and create a database connection; Call the producer thread from the producer thread pool, access the database based on the database connection, perform streaming queries on the database, and determine whether the streaming query results contain the target data. If so, push the target data to the message queue. If not, close the database connection, encapsulate the end flag and push it to the message queue. The database streaming query ends and the producer thread is recycled.

4. The method for rapidly generating multi-format files based on a large database table according to claim 1, characterized in that: The calling of the consumer thread, pulling target data in batches from the message queue and generating a file generation task according to the target data, and submitting the file generation task to the file generation thread, includes: Call the consumer thread from the consumer thread pool, monitor the message queue through the consumer thread, and pull the target data from the message queue when a message is pushed in the message queue; Determine whether the currently pulled data contains an end mark. If not, determine whether the pulled target data reaches the data batch write threshold. If so, write the currently pulled target data into the file generation task and submit the file generation task to the file generation thread pool. If not, continue to pull the target data from the message queue. If the currently pulled data contains an end mark, determine whether there is target data that does not reach the data batch write threshold. If there is corresponding target data, write the corresponding target data into the file generation task and submit it to the file generation thread pool. If there is no corresponding target data, end the data pulling operation and recycle the current consumer thread.

5. The method for rapidly generating multi-format files based on a large database table according to claim 1, characterized in that: The file generation thread generates a target file of a corresponding format according to the file generation task, and saves the target file to a temporary directory, including: The file generation thread pool starts multiple file generation threads in sequence according to the received file generation tasks; The file generation thread generates a target file according to the file format corresponding to the file generation task and the target data carried by the file generation task, and saves the target file to a temporary directory, and at the same time recycles the file generation thread that has completed the file generation task.

Images