A file generation method, device, apparatus, and storage medium
By listening to the message queue in real time and storing it in the database, and using asynchronous threads to generate files, the performance consumption problem of file generation in Java web systems is solved, achieving system decoupling and performance improvement.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- PONY TORCH (YUXI) TECH CO LTD
- Filing Date
- 2022-08-29
- Publication Date
- 2026-05-12
AI Technical Summary
In Java web systems, file generation consumes server performance, and byte conversion after parsing leads to unfriendly client responses. Existing technical solutions may result in message loss and insufficient performance on the consumer side under high concurrency.
Data is acquired by listening to the message queue in real time, stored in the database and tagged with information. File generation is completed using an asynchronous thread, thus decoupling the system and avoiding the need to write separate code for file generation.
It reduced maintenance costs, improved system performance and ease of use, and avoided message loss and performance bottlenecks.
Smart Images
Figure CN115390834B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus, device and storage medium for generating files. Background Technology
[0002] In real-world Java web systems, actual file generation is resource-intensive, involving file generation parsing, byte conversion after generation, and finally, writing to the server as a byte stream. Without decoupling file generation from API responses, the client's response will wait until the file is generated, which is inconvenient for users. Most existing solutions use message queues (MQ) internally to send messages, decoupling the process by having another MQ in another service listen for file generation. However, this can lead to message loss if a large number of requests are received simultaneously, causing the consumer's performance to sluggish. Summary of the Invention
[0003] In view of this, the purpose of this invention is to provide a file generation method, apparatus, device, and storage medium that can reduce maintenance costs, improve performance, and be easy to use. The specific solution is as follows:
[0004] In a first aspect, this application discloses a file generation method, applied to a first service in a server, comprising:
[0005] The system monitors the message queue in real time, and when a data change is detected in the message queue, it retrieves the first target data of the change.
[0006] The first target data is stored in the first database;
[0007] Read the tag information (data without generated files) from the first database to obtain the second target data;
[0008] The second target data is sent to an asynchronous thread so that the corresponding file generation operation can be completed based on the second target data to obtain the target file.
[0009] Optionally, before the real-time monitoring message queue, the system further includes:
[0010] The server receives requests from clients through a second service and extracts third target data for generating files from the requests.
[0011] The third target data is sent from the second service to the message queue.
[0012] Optionally, after receiving the request sent by the client through the second service in the server, the method further includes:
[0013] Determine whether the request meets the preset data entry rules;
[0014] If the request meets the preset storage rules, then the information to be stored is extracted from the request; the information to be stored includes parameters and login information.
[0015] The information to be stored is stored in the second database;
[0016] If the request does not meet the preset database entry rules, a corresponding prompt message is generated and returned to the client.
[0017] Optionally, after storing the first target data in the first database, the method further includes:
[0018] Delete the first target data from the message queue.
[0019] Optionally, the step of reading tag information from the first database as data for which no files have been generated, to obtain the second target data, includes:
[0020] Set a timer;
[0021] The timer periodically reads data from the first database that has not yet generated a file, thereby obtaining the second target data.
[0022] Optionally, after storing the first target data in the first database, the method further includes:
[0023] Determine the file generation status information corresponding to the first target data to obtain the first target status information; the file generation status information includes the status of no file generated, file generated, and file being generated.
[0024] Based on the first target status information, the corresponding tag information is generated, and the tag information is added to the area in the first database corresponding to the first target data.
[0025] Optionally, after sending the second target data to an asynchronous thread to complete the corresponding file generation operation based on the second target data and obtaining the target file, the method further includes:
[0026] The asynchronous thread generates and returns file generation information so that the tag information in the first database corresponding to the second target data can be changed to the generated file.
[0027] Secondly, this application discloses a file generation apparatus, applied to a first service in a server, comprising:
[0028] The first data acquisition module is used to monitor the message queue in real time. When a data change is detected in the message queue, the module acquires the changed first target data.
[0029] Data storage module, used to store the first target data in the first database;
[0030] The second data acquisition module is used to read data from the first database that has not generated a file due to tag information, in order to obtain the second target data;
[0031] The file generation module is used to send the second target data to an asynchronous thread so that the corresponding file generation operation can be completed based on the second target data to obtain the target file.
[0032] Thirdly, this application discloses an electronic device, including:
[0033] Memory, used to store computer programs;
[0034] A processor for executing the computer program to implement the steps of the file generation method disclosed above.
[0035] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the file generation method disclosed above.
[0036] As can be seen, this application provides a file generation method, including: real-time monitoring of a message queue; when a data change is detected in the message queue, obtaining the changed first target data; storing the first target data in a first database; reading data with tag information indicating no file generation from the first database to obtain second target data; and sending the second target data to an asynchronous thread to complete the corresponding file generation operation based on the second target data to obtain a target file. Thus, this application achieves system decoupling by monitoring for changed data in the message queue, extracting and storing the detected changed data, setting tag information for each data in the database, obtaining second target data with tag information indicating no file generation, and sending the second target data to an asynchronous thread to complete the file generation operation. This eliminates the need to write separate code for file generation, and each data corresponds to a separate asynchronous thread, ensuring no interference between them, reducing maintenance costs, improving performance, and facilitating use. Attached Figure Description
[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0038] Figure 1 This is a flowchart of a document generation method disclosed in this application;
[0039] Figure 2 This is a service call diagram disclosed in this application;
[0040] Figure 3 This is a flowchart of a specific document generation method disclosed in this application;
[0041] Figure 4 This is a flowchart of the workflow disclosed in this application;
[0042] Figure 5 A schematic diagram of the document generation device provided for this application;
[0043] Figure 6 This application provides a structural diagram of an electronic device. Detailed Implementation
[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0045] Currently, in practical Java web systems, actual file generation is quite resource-intensive for servers. This involves parsing the file generation method, performing byte conversion after generation, and finally writing the file to the server as a byte stream. Without decoupling file generation from API responses, the client's response will wait until the file is generated, which is inconvenient for users. Most existing solutions decouple file generation by sending messages via message queues (MQ) and having another MQ in the same or a separate service listen for the file in another service. However, if a large number of requests are received simultaneously, the consumer's performance may be insufficient, leading to message loss. Therefore, this application provides a file generation method that reduces maintenance costs, improves performance, and is easy to use.
[0046] This invention discloses a file generation method, see [link to relevant documentation]. Figure 1As shown, the first service applied in the server includes the following methods:
[0047] Step S11: Monitor the message queue in real time. When a data change is detected in the message queue, obtain the first target data of the change.
[0048] In this embodiment, the message queue is monitored in real time. When a data change is detected in the message queue, the changed first target data is obtained. It is understood that, as... Figure 2 As shown, in the server, the first service (i.e., service b) listens to the message queue in real time. It determines whether to read data from the message queue by checking if there is any changed data. Each time, the data read is the changed data in the message queue that differs from the previous read data; this is the first target data. A second service (i.e., service a) also exists in the server. The second service writes data to the message queue so that the first service can read the first target data.
[0049] Step S12: Store the first target data in the first database.
[0050] In this embodiment, when a data change is detected in the message queue, the changed first target data is obtained and then stored in the first database. It is understood that the first database stores all data, including the newly received first target data and historical first target data, and each data item is assigned a tag based on its corresponding status information, so as to determine whether the step of generating a file has not yet been performed based on the tag information.
[0051] Step S13: Read the data with tag information that has not been generated from the first database to obtain the second target data.
[0052] In this embodiment, after storing the first target data in the first database, data with tag information indicating no file generation is read from the first database to obtain the second target data. It can be understood that, based on the tag information of each data item in the first database, data with tag information indicating no file generation is read from the first database to obtain the second target data ready for the file generation step.
[0053] Step S14: Send the second target data to an asynchronous thread so that the corresponding file generation operation can be completed based on the second target data to obtain the target file.
[0054] In this embodiment, after reading data with tag information indicating no file has been generated from the first database and obtaining the second target data, the second target data is sent to an asynchronous thread to complete the corresponding file generation operation based on the second target data, thereby obtaining the target file. Furthermore, after obtaining the target file, the asynchronous thread generates and returns file generation information, so that the tag information in the first database corresponding to the second target data is changed to indicate that a file has been generated.
[0055] Understandably, message queues enable asynchronous message sending by splitting file generation into methods that receive messages in the message queue and publishing messages to the queue. However, in existing technologies, receiving a large number of requests simultaneously can lead to performance issues on the consumer side, resulting in message loss. Therefore, a decoupled file generation solution based on message queues (MQ), MySQL (a relational database management system), and thread pools is designed. Thread pools are a form of multi-threaded processing; tasks are added to the message queue during processing, and these tasks are automatically started after thread creation. Threads in the pool are generally not frequently created or deleted, effectively saving server performance. MySQL is a relational database management system, storing data in different tables instead of storing all data in one large repository, increasing processing speed and flexibility. Thus, in high-concurrency scenarios, only database writes are performed, while file generation and other operations are executed asynchronously through the thread pool. This achieves system decoupling; multiple systems only need to know the message queue address to initiate requests, eliminating the need to write separate file generation code.
[0056] As can be seen, this application provides a file generation method, including: real-time monitoring of a message queue; when a data change is detected in the message queue, obtaining the changed first target data; storing the first target data in a first database; reading data with tag information indicating no file generation from the first database to obtain second target data; and sending the second target data to an asynchronous thread to complete the corresponding file generation operation based on the second target data to obtain a target file. Thus, this application achieves system decoupling by monitoring for changed data in the message queue, extracting and storing the detected changed data, setting tag information for each data in the database, obtaining second target data with tag information indicating no file generation, and sending the second target data to an asynchronous thread to complete the file generation operation. This eliminates the need to write separate code for file generation, and each data corresponds to a separate asynchronous thread, ensuring no interference between them, reducing maintenance costs, improving performance, and facilitating use.
[0057] See Figure 3As shown, this embodiment of the invention discloses a file generation method. Compared with the previous embodiment, this embodiment further explains and optimizes the technical solution.
[0058] Step S21: Receive the request sent by the client through the second service in the server, and extract the third target data for generating the file from the request.
[0059] In this embodiment, the second service in the server receives requests sent by clients, performs business operations, extracts third target data for generating files from the requests, and sends the third target data from the second service to the message queue. Further, the second service in the server receives requests sent by clients, determines whether the requests meet preset storage rules, and if the requests meet the preset storage rules, extracts information to be stored from the requests and stores the information to be stored in the second database; if the requests do not meet the preset storage rules, generates corresponding prompt information and returns the prompt information to the client.
[0060] like Figure 4 As shown, the server receives a request from the client through the second service (i.e., service a). The request contains parameters and login information. The login operation is then completed using the parameters and login information. If the login is successful, the server proceeds to steps such as determining permissions and data format. If all the above steps are successful, the server determines that the request meets the preset database entry rules. The server extracts the information to be stored from the request and stores it in the second database. The server also extracts the third target data for generating the file and sends the third target data to the message queue. The business data is then successfully written.
[0061] Step S22: Monitor the message queue in real time. When a data change is detected in the message queue, obtain the first target data of the change.
[0062] Step S23: Store the first target data in the first database.
[0063] In this embodiment, the first service monitors the message queue in real time. When a new message is detected in the message queue, it directly writes the new message to the first database without performing any file generation operations. After storing the first target data in the first database, the first target data is deleted from the message queue.
[0064] Step S24: Determine the file generation status information corresponding to the first target data to obtain the first target status information.
[0065] In this embodiment, the file generation status information corresponding to the first target data is determined to obtain the first target status information. The file generation status information includes "file not generated," "file generated," and "file being generated." It is understood that the tag information can also be other types of tags; there are no restrictions, as long as it reflects whether the current data has been generated into a file.
[0066] Step S25: Generate the corresponding tag information based on the first target state information, and add the tag information to the area in the first database corresponding to the first target data.
[0067] In this embodiment, after determining the file generation status information corresponding to the first target data and obtaining the first target status information, the corresponding tag information is generated based on the first target status information, and the tag information is added to the area in the first database corresponding to the first target data. It can be understood that after each data is stored in the first database, corresponding tag information is automatically generated for marking, and all data in the database is set with corresponding tag information.
[0068] Step S26: Set a timer to periodically read data with ungenerated tag information from the first database to obtain the second target data.
[0069] In this embodiment, a timer is set to periodically read data from the first database whose tag information indicates that no file has been generated, in order to obtain the second target data. Specifically, the timer continuously queries the database for records of ungenerated files, that is, it periodically reads data from the first database whose tag information indicates that no file has been generated. Using thread pool technology and a multi-threaded approach, the CPU (Central Processing Unit) performance is fully utilized to generate the corresponding file.
[0070] Step S27: Send the second target data to an asynchronous thread so that the corresponding file generation operation can be completed based on the second target data to obtain the target file.
[0071] In this embodiment, the second target data is sent to an asynchronous thread to complete the corresponding file generation operation based on the second target data, thereby obtaining the target file. It is understood that each second target data corresponds to one asynchronous thread, and multiple asynchronous threads are independent of each other and do not interfere with each other. Furthermore, after the file is generated, file generation information is returned to the first database to change the tag information of the data in the first database, avoiding the need to perform the file generation step again.
[0072] For details regarding step S22, please refer to the corresponding content disclosed in the foregoing embodiments, which will not be repeated here.
[0073] As can be seen, this embodiment of the application receives requests sent by clients through a second service in the server and extracts third target data for generating files from the requests; it monitors the message queue in real time, and when data changes are detected in the message queue, it obtains the changed first target data; it stores the first target data in a first database; it determines the file generation status information corresponding to the first target data to obtain first target status information; it generates corresponding tag information based on the first target status information and adds the tag information to the area in the first database corresponding to the first target data; it sets a timer to periodically read data with tag information indicating no file generation from the first database to obtain second target data; and it sends the second target data to an asynchronous thread to complete the corresponding file generation operation based on the second target data to obtain the target file. This approach is simple in structure, reduces maintenance costs, improves performance, and is convenient to use.
[0074] See Figure 5 As shown in the embodiments, this application also discloses a file generation device, applied to a first service in a server, including:
[0075] The first data acquisition module 11 is used to monitor the message queue in real time, and when a data change is detected in the message queue, it acquires the changed first target data.
[0076] Data storage module 12 is used to store the first target data in the first database;
[0077] The second data acquisition module 13 is used to read data whose tag information is not generated from the first database in order to obtain the second target data;
[0078] The file generation module 14 is used to send the second target data to an asynchronous thread so that the corresponding file generation operation can be completed based on the second target data to obtain the target file.
[0079] As can be seen, this application includes: real-time monitoring of a message queue; when a data change is detected in the message queue, acquiring the changed first target data; storing the first target data in a first database; reading data with tag information indicating no file generation from the first database to obtain second target data; and sending the second target data to an asynchronous thread to complete the corresponding file generation operation based on the second target data to obtain the target file. Thus, this application achieves system decoupling by monitoring for changes in the message queue, extracting and storing the detected changes, setting tag information for each data in the database, acquiring second target data with tag information indicating no file generation, and sending the second target data to an asynchronous thread to complete the file generation operation. This eliminates the need to write separate file generation code, and each data corresponds to a separate asynchronous thread, ensuring no interference between them, reducing maintenance costs, improving performance, and facilitating use.
[0080] In some specific embodiments, the first data acquisition module 11 specifically includes:
[0081] A request receiving unit is configured to receive requests sent by clients through a second service in the server.
[0082] The third target data extraction unit is used to extract third target data for generating a file from the request;
[0083] A request determination unit is used to determine whether the request meets the preset entry rules;
[0084] The information to be stored extraction unit is used to extract information to be stored from the request if the request meets the preset storage rules; the information to be stored includes parameters and login information;
[0085] A storage unit for information to be stored is used to store the information to be stored in a second database;
[0086] The prompt message return unit is used to generate corresponding prompt messages and return the prompt messages to the client if the request does not meet the preset database entry rules;
[0087] The third target data sending unit is used to send the third target data from the second service to the message queue;
[0088] The message queue listening unit is used to monitor the message queue in real time.
[0089] The first target data acquisition unit is used to acquire the changed first target data when a data change is detected in the message queue.
[0090] In some specific embodiments, the data storage module 12 specifically includes:
[0091] The first target data storage unit is used to store the first target data into the first database;
[0092] A status information generation unit is used to determine the file generation status information corresponding to the first target data to obtain the first target status information; the file generation status information includes the status of no file generated, file generated, and file being generated.
[0093] A tag information generation unit is used to generate corresponding tag information based on the first target state information;
[0094] A tag information adding unit is used to add the tag information to the area in the first database corresponding to the first target data;
[0095] The first target data deletion unit is used to delete the first target data in the message queue.
[0096] In some specific embodiments, the second data acquisition module 13 specifically includes:
[0097] The timer setting unit is used to set the timer;
[0098] The second target data acquisition unit is used to periodically read data with tag information that has not been generated into a file from the first database through the timer, so as to obtain the second target data.
[0099] In some specific embodiments, the file generation module 14 specifically includes:
[0100] The file generation unit is used to send the second target data to an asynchronous thread so that the corresponding file generation operation can be completed based on the second target data to obtain the target file;
[0101] The file generation information return unit is used to generate and return file generation information through the asynchronous thread, so as to change the tag information in the first database corresponding to the second target data to the generated file.
[0102] Furthermore, embodiments of this application also provide an electronic device. Figure 6 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.
[0103] Figure 6This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the file generation method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.
[0104] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.
[0105] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.
[0106] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the file generation method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.
[0107] Furthermore, this application also discloses a storage medium storing a computer program, which, when loaded and executed by a processor, implements the file generation method steps disclosed in any of the foregoing embodiments.
[0108] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.
[0109] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0110] The above provides a detailed description of a file generation method, apparatus, device, and storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.
Claims
1. A method for generating a file, characterized in that, The first service applied to the server includes: The server receives requests sent by clients through a second service and extracts third target data for generating files from the requests. Send the third target data from the second service to the message queue; The message queue is monitored in real time, and when a data change is detected in the message queue, the first target data of the change is obtained; The first target data is stored in a first database; each piece of data stored in the first database is assigned tag information based on the status information corresponding to the current data. Determine the file generation status information corresponding to the first target data to obtain the first target status information; the file generation status information includes no file generated, file generated, and file being generated. Generate corresponding tag information based on the first target status information, and add the tag information to the area in the first database corresponding to the first target data; Based on the tag information of each data in the first database, data whose tag information has not been generated into a file is read from the first database to obtain the second target data; The second target data is sent to an asynchronous thread so that the corresponding file generation operation can be completed based on the second target data to obtain the target file; The asynchronous thread generates and returns file generation information so that the tag information in the first database corresponding to the second target data can be changed to the generated file.
2. The file generation method according to claim 1, characterized in that, After receiving the request sent by the client through the second service in the server, the process further includes: Determine whether the request meets the preset data entry rules; If the request meets the preset storage rules, then the information to be stored is extracted from the request; the information to be stored includes parameters and login information. The information to be stored is stored in the second database; If the request does not meet the preset database entry rules, a corresponding prompt message is generated and returned to the client.
3. The file generation method according to claim 1, characterized in that, After storing the first target data in the first database, the method further includes: Delete the first target data from the message queue.
4. The file generation method according to claim 1, characterized in that, The step of reading tag information from the first database (data for which no files have been generated) to obtain the second target data includes: Set a timer; The timer periodically reads data from the first database that has not yet generated a file, thereby obtaining the second target data.
5. A document generation device, characterized in that, The first service applied to the server includes: The first data acquisition module is used to receive requests sent by clients through a second service in the server, and extract third target data for generating files from the requests; send the third target data from the second service to a message queue; monitor the message queue in real time, and when a data change is detected in the message queue, acquire the changed first target data; The data storage module is used to store the first target data in a first database; each piece of data stored in the first database is assigned tag information based on the status information corresponding to the current data; the file generation status information corresponding to the first target data is determined to obtain the first target status information; the file generation status information includes no file generated, file generated, and file being generated; the corresponding tag information is generated based on the first target status information, and the tag information is added to the area in the first database corresponding to the first target data; The second data acquisition module is used to read data whose tag information is not generated into a file from the first database according to the tag information of each data in the first database, so as to obtain the second target data; The file generation module is used to send the second target data to an asynchronous thread so that the corresponding file generation operation can be completed based on the second target data to obtain the target file; the asynchronous thread generates and returns file generation information so that the tag information corresponding to the second target data in the first database can be changed to the generated file.
6. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the file generation method as described in any one of claims 1 to 4.
7. A computer-readable storage medium, characterized in that, Used for storing computer programs; wherein, when the computer program is executed by a processor, it implements the file generation method as described in any one of claims 1 to 4.