Report exporting method, report exporting system, exporting server and storage medium

Abstract

The invention discloses a report exporting method, a report exporting system, an exporting server and a computer readable storage medium, and the report exporting method is applied to the exporting server and comprises the following steps: selecting a target exporting task from a message queue according to a first-in first-out rule; according to the target export task and all the export tasks in execution, determining a total occupied memory amount; and when the total occupied memory capacity exceeds a threshold value, putting the target export task back to the first place of the message queue to wait for re-selection, or when the total occupied memory capacity does not exceed the threshold value, taking the target export task as a new export task in execution, and performing report export control on a service system end corresponding to the new export task. Therefore, according to the technical scheme, memory overflow can be avoided when the export service executes the export task, so that the stability of the export performance of the export service and the high efficiency of the export efficiency are ensured, and the user experience is improved.

Description

Technical Field

[0001] The present application relates to the technical field of file management, and in particular to a report exporting method, a report exporting system, an exporting server, and a computer-readable storage medium. Background Art

[0002] File export is a very common scenario in information management software systems. In information management software systems, usually every business module needs to export query data results in the form of reports for analysis.

[0003] The current report export methods include synchronous export and asynchronous export: the traditional synchronous export method can meet 80% of the file export requirements, and the commonly used Java export methods include poi and streaming export; the traditional asynchronous export method is based on the microservice architecture and runs independently of the business module, which allows the report file generation thread to run in the background without waiting time. After the processing is completed, the front desk is notified to display and download. This method not only greatly reduces the pressure on the business module, but also improves the user experience. Among them, the microservice architecture is a new concept proposed in recent years. The microservice system includes multiple small services with complete business functions. The services are connected in series through APIs or messages. At the same time, the development, testing and deployment of each service are carried out independently, which greatly reduces the degree of coupling between systems and improves the scalability of the system.

[0004] However, the asynchronous export method based on the microservice architecture has the risk of memory overflow when exporting large amounts of report files in the case of asynchronous processing with high concurrency, which will lead to a decrease in the export performance and export efficiency of the export service of the microservice architecture. Therefore, how to avoid the problem of memory overflow in the export service when executing the export task, which leads to a decrease in export performance, is a technical problem that technicians in this field need to solve urgently. Summary of the invention

[0005] The purpose of this application is to provide a report export method, a report export system, an export server and a computer-readable storage medium, which can avoid memory overflow when the export service performs an export task, thereby ensuring the stability of the export performance of the export service and the high efficiency of the export, and improving the user experience.

[0006] To achieve the above objectives:

[0007] An embodiment of the present application provides a report export method, which is applied to an export server, including: selecting a target export task from a message queue according to a first-in-first-out rule; determining the total occupied memory amount based on the target export task and all export tasks being executed; when the total occupied memory amount exceeds a threshold, placing the target export task back to the first position of the message queue to wait for re-selection, or when the total occupied memory amount does not exceed the threshold, treating the target export task as a new export task being executed, and performing report export control on the business system end corresponding to the new export task.

[0008] In one embodiment, according to the first-in-first-out rule, before the step of selecting the target export task from the message queue, it includes: receiving a report export request for the business system end, the report export request includes at least one of a request method, a request parameter, a query condition, a data query address and a user login information; according to the report export request, determining the export task and its corresponding first processing occupied memory; associating the determined export task with the first processing occupied memory, and storing the determined export task in the message queue.

[0009] In one embodiment, when the total occupied memory amount does not exceed the threshold, the target export task is used as a new export task in execution, and the step of report export control is performed on the business system end corresponding to the new export task, including: determining the number of threads allocated to the new export task according to the export restriction rules of the business system end corresponding to the new export task, wherein the export restriction rules include at least one of the number of pages for asynchronous export of the task, the maximum total number of exported rows, the maximum number of rows for each Excel file, the concurrency of the export request batch, and the concurrency within the export request batch; calling at least one thread corresponding to the number of threads allocated according to the new export task to pull report data from the business system end corresponding to the new export task through at least one thread; and obtaining at least one report file based on the report data fed back by at least one thread.

[0010] In one embodiment, when the total occupied memory amount does not exceed the threshold, the target export task is treated as a new export task in execution, and after the step of performing report export control on the business system end corresponding to the new export task, it includes: based on the report download method corresponding to the new export task, performing synchronous download control or asynchronous download control on at least one report file obtained by the report export control.

[0011] In one embodiment, a synchronous download control step is performed on at least one report file obtained by performing report export control, including: determining the number of synchronous download tasks; determining the download traffic limit according to the number of synchronous download tasks; and according to the download traffic limit, feeding back to the front end at least one report file obtained by performing report export control.

[0012] In one embodiment, the step of performing asynchronous download control on at least one report file obtained by performing report export control includes: uploading at least one report file obtained by performing report export control to a file server according to preset upload rules, and obtaining download index information corresponding to at least one report file fed back by the file server, the download index information including a download address; and feeding back the download index information to the front end.

[0013] In one embodiment, the preset upload rule includes at least one of the following:

[0014] Upload a report file obtained by report export control directly to the file server;

[0015] Upload multiple report files obtained through report export control to the file server respectively;

[0016] The multiple report files obtained by the report export control are compressed to obtain compressed files, and the compressed files are uploaded to the file server.

[0017] In one embodiment, the method of the present application further includes: storing the task record corresponding to each export task in the message queue in a database.

[0018] In one embodiment, the method of the present application further includes: after performing report export control to obtain at least one report file, associating task status information for each report file, and storing the task status information in a database, wherein the task status information includes a unique file identifier and a task completion status.

[0019] The embodiment of the present application also provides a report export system, including a business system end and a microservice architecture; the microservice architecture is used to access the business system end and serve as a report export interface for the business system end, and the microservice architecture includes a front end, an export server, and a file server; wherein the front end is used to respond to the export operation for the business system end, obtain and send a report export request to the export server, so that the export server determines the export task according to the received report export request, and adds the export task to the message queue; wherein the export server is used to select the target export task from the message queue according to the first-in-first-out rule, and determine the total occupied memory according to the target export task and all the export tasks in execution, and when the total occupied memory exceeds the threshold, put the target export task back to the first place in the message queue to wait for re-selection, or when the total occupied memory does not exceed the threshold, use the target export task as a new export task in execution, and perform report export control for the business system end; wherein the file server is used to receive the report file obtained and uploaded by the export server after performing report export control, and provide a report file download service.

[0020] An embodiment of the present application also provides an export server, comprising: a processor and a memory storing a computer program, and when the processor runs the computer program, the steps of the report export method described in any one of the above items are implemented.

[0021] An embodiment of the present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the report export method described in any one of the above items are implemented.

[0022] The report export method, report export system, computing device and computer-readable storage medium provided by the embodiment of the present application, the report export method is applied to the export server, including: selecting the target export task from the message queue according to the first-in-first-out rule; determining the total occupied memory according to the target export task and all the export tasks in execution; when the total occupied memory exceeds the threshold, the target export task is put back to the first place of the message queue to wait for selection again, or when the total occupied memory does not exceed the threshold, the target export task is used as a new export task in execution, and the report export control is performed for the business system end corresponding to the new export task. In this way, the technical scheme of the present application can make the processing occupied memory corresponding to all the export tasks in execution in the export server not exceed the threshold, so that the situation of memory overflow can be avoided when the export server asynchronously and concurrently processes multiple export tasks (that is, when the report file of large data volume is exported in the case of asynchronous processing with high concurrency, the situation of memory overflow is avoided). Therefore, the technical scheme of the present application can avoid memory overflow in the export service when executing the export task, thereby ensuring the stability of the export performance of the export service and the high efficiency of the export efficiency, and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A flowchart of a report export method provided in the first embodiment of the present application;

[0024] Figure 2 A schematic diagram of an export configuration interface provided by a front end in an embodiment of the present application;

[0025] Figure 3 A structural diagram of a report export system provided in the second embodiment of the present application;

[0026] Figure 4 A flowchart of generating and executing export tasks for the microservice architecture in the report export system of the example of this application;

[0027] Figure 5 A flowchart of viewing the results of export tasks and downloading files for the microservice architecture in the report export system of this application example;

[0028] Figure 6 A schematic diagram of the structure of the export server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0030] It should be noted that, in this article, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element. In addition, components, features, and elements with the same name in different embodiments of the present application may have the same meaning or different meanings, and their specific meanings need to be determined by their explanation in the specific embodiment or further combined with the context of the specific embodiment.

[0031] It should be understood that, although the terms first, second, third, etc. may be used to describe various information in this article, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this article, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "at the time of..." or "when..." or "in response to determination". Furthermore, as used in this article, the singular forms "one", "one" and "the" are intended to also include plural forms, unless there is an opposite indication in the context. It should be further understood that the terms "comprising", "including" indicate that there are described features, steps, operations, elements, components, projects, kinds, and / or groups, but do not exclude the existence, occurrence or addition of one or more other features, steps, operations, elements, components, projects, kinds, and / or groups. The terms "or" and "and / or" used herein are interpreted as inclusive, or mean any one or any combination. Thus, “A, B, or C” or “A, B and / or C” means “any of the following: A; B; C; A and B; A and C; B and C; A, B, and C.” An exception to this definition will occur only when a combination of elements, functions, steps, or operations are inherently mutually exclusive in some manner.

[0032] It should be understood that, although the various steps in the flowchart in the embodiment of the present application are displayed in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and it can be performed in other orders. Moreover, at least a portion of the steps in the figure may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and their execution order is not necessarily performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0033] It should be noted that in this article, step codes such as S1 and S2 are used for the purpose of expressing the corresponding content more clearly and concisely, and do not constitute a substantial limitation on the order. When implementing the step, those skilled in the art may execute S2 first and then S1, etc., but these should all be within the scope of protection of this application.

[0034] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0035] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present application and have no specific meaning. Therefore, "module", "component" or "unit" can be used in a mixed manner.

[0036] First embodiment

[0037] See also Figure 1 , Figure 1 This is a flow chart of a report export method provided in the first embodiment of the present application. The method can be executed by an export server provided in the embodiment of the present application. The export server can be implemented in software and / or hardware.

[0038] The report export method provided in this embodiment includes the following steps (for example, steps S1 to S3):

[0039] Step S1: According to the first-in-first-out rule, select the target export task from the message queue.

[0040] In one embodiment, step S1: before selecting the target export task from the message queue according to the first-in-first-out rule, it may include: receiving a report export request for the business system end; determining the export task and its corresponding first processing occupied memory based on the report export request; associating the determined export task with the first processing occupied memory, and storing the determined export task in the message queue.

[0041] In one implementation, the report export request may represent guidance to obtain various information satisfying the user's report data export requirements from the business system.

[0042] In one implementation, the report export request may include at least one of a request method, a request parameter, a query condition, a data query address, and user login information.

[0043] In one embodiment, the request method may represent the type of request sent by the export server to the business system, such as GET, POST, PUT, DELETE, etc. These request methods define the communication method between the export server and the business system. GET is usually used to query data, POST is used to submit data, PUT is used to update resources, and DELETE is used to delete resources. These request methods have clear definitions and uses in the HTTP protocol.

[0044] In one embodiment, the request parameter may represent a method of transmitting data, and a key-value pair may be added to the end of the URL, or placed in the request header or request body. The request parameters may be divided into query string parameters, path parameters, request body parameters, and header parameters. Query string parameters are usually used for GET requests, path parameters are used to identify resources, and request body parameters and header parameters are used to transmit data in requests such as POST and PUT.

[0045] In one embodiment, the query condition can represent the data to be displayed in the report to be filtered by the content of the field set as the condition. The query condition can be set to select the query field, query range (such as time range, location range, subject range, etc.). The query condition is used to filter data in the report so that the user can obtain the required data according to specific conditions.

[0046] In one embodiment, the data query address may represent an address for sending a request to a URL or client of a specific resource or service on the business system side to obtain data. This address may generally include a protocol, a server address to the business system side, a port number, and a resource path, etc. The data query address is the entry point for the export server to interact with the business system side, and determines the specific target of the request.

[0047] In one embodiment, the user login information may represent a user's login authentication information for the business system. The business system may determine the scope of the user's access rights to data based on the user login information, and determine that the pull operation of the report data of the export server is a trusted operation that has passed authentication. In this way, the technical solution of this embodiment can ensure the data security of data access and export.

[0048] In one implementation, the report export request may further include report export configuration and / or report download method.

[0049] In one implementation, the report export configuration may include at least one of an export type, an export file name, and the like.

[0050] In one embodiment, the report downloading mode includes a synchronous downloading mode and an asynchronous downloading mode. When the export server supports both synchronous downloading of report files and asynchronous downloading of report files, the report export request may include the report downloading mode.

[0051] In one embodiment, the export server may be included in a microservice framework, and the microservice framework also includes a front end.

[0052] In one embodiment, the front end may provide an interactive interface for users to access the business system end, for example, see Figure 2The front end provides an export configuration interface, and displays the export type options, export file name setting box, export field options, etc. in the export configuration interface, so that users can perform export operations on the business system side. In addition, the export configuration interface can also provide a "view export task" option for clicking and jumping to the progress viewing interface of the export task.

[0053] In one embodiment, the front end can serve as an interactive interface for one or more business system terminals. Optionally, the user can log in through the front end to log in and access one or more business system terminals, thereby triggering an export configuration interface for one or more business system terminals.

[0054] In one embodiment, the business system end can represent a system that processes and manages specific information or business in production scenarios and life scenarios (such as an enterprise's bidding management system, attendance management system, etc.), and the system can be built on a server and / or terminal product.

[0055] In one implementation, after the front end receives the export operation for the business system, it can trigger a report export request and send it to the export server.

[0056] In one implementation, the front end and the export server can be independent services, that is, development, testing and deployment are all performed independently, which greatly reduces the degree of coupling between systems and improves the scalability of the system.

[0057] In one embodiment, the export task may include all or part of the information in the report export request. When the export task includes user login information, it can avoid the user login operation on the front end from expiring, resulting in the failure of task execution in the export server when there are many export tasks and the task execution is too slow.

[0058] In one implementation, the export task may represent a collection of various information or parameters that enables the export server to obtain and export a report file that meets the user's report data export requirements from the business system.

[0059] In one embodiment, according to the report export request, the step of determining the memory occupied by the export task and its corresponding first processing may include: according to the report export request, obtaining the export data information from the business system end, wherein the export data information may include at least one of the total number of data, the number of fields, the field type, etc.; and calculating the memory occupied by the first processing corresponding to the report export request or the export task according to the export data information. In this way, the technical solution of this embodiment can determine the memory occupied by the processing required by the export server to execute the export task according to the report export request (such as the request method, request parameters, query conditions, data query address, user login information, etc.), so that the export server can realize the number of export tasks being executed or the control of the memory occupied by the processing, and avoid the memory overflow of the export service when executing the export task.

[0060] Step S2: Determine the total memory usage according to the target export task and all the export tasks being executed.

[0061] In one embodiment, step S2: determining the total occupied memory based on the target export task and all export tasks being executed may include: obtaining the first processing occupied memory corresponding to the target export task and the second processing occupied memory corresponding to all export tasks being executed; adding the first processing occupied memory to the second processing occupied memory to obtain the total occupied memory.

[0062] In one implementation, the memory occupied by the second processing corresponding to all the export tasks being executed may be obtained by adding the memory occupied by the third processing corresponding to each of the export tasks being executed.

[0063] In one embodiment, the third processing memory occupied by the executing export task can be the first processing memory occupied by the executing export task, or can be the current processing memory occupied by the executing export task. In the technical solution of this embodiment, the second processing memory occupied by all the executing export tasks is determined based on the first processing memory occupied by each executing export task before the execution. This can ensure that the export server has a certain memory redundancy when exporting report files with large amounts of data under high concurrency conditions, and can effectively avoid memory overflow when the export service executes the export task. In the technical solution of this embodiment, the second processing memory occupied by all the executing export tasks is determined based on the current processing memory occupied by the executing export task, which can fully utilize the memory of the export server within the tolerance threshold and improve the consumption efficiency or execution efficiency of the export tasks in the message queue.

[0064] Step S3: When the total occupied memory exceeds the threshold, the target export task is put back to the first position of the message queue to wait for selection again; or when the total occupied memory does not exceed the threshold, the target export task is treated as a new export task in execution, and the report export control is performed for the business system end corresponding to the new export task.

[0065] In one embodiment, when the total occupied memory amount does not exceed the threshold, the target export task is treated as a new export task in execution, and the step of performing report export control on the business system end corresponding to the new export task may include: determining the number of threads allocated to the new export task according to the export restriction rules of the business system end corresponding to the new export task; calling at least one thread corresponding to the number of threads allocated according to the new export task to pull report data from the business system end corresponding to the new export task through at least one thread; and obtaining at least one report file based on the report data fed back by at least one thread.

[0066] In one implementation, the export restriction rule may represent various configuration information that the business system end restricts other ends from accessing or querying in order to ensure its own business processing capability.

[0067] In one embodiment, the export restriction rules may include at least one of the number of pages for asynchronous export of tasks, the maximum total number of rows to be exported, the maximum number of rows per Excel file, the concurrency of export request batches, and the concurrency within export request batches.

[0068] In one embodiment, the step of determining the number of threads to be allocated for a new export task based on the export restriction rules of the business system end corresponding to the new export task may include: obtaining the number of allocated threads for other export tasks on the business system end corresponding to the new export task; determining the number of remaining allocatable threads based on the export restriction rules of the business system end corresponding to the new export task and the number of allocated threads; and determining the number of threads to be allocated for the new export task based on the number of remaining allocatable threads.

[0069] In this way, the technical solution of this embodiment can avoid too many threads requesting access to the same business system end, or too large a number of paging queries to the business system end, when multiple users export report data for the same business system end in the same time period, resulting in memory overflow of the business system end and reduced business processing efficiency or even a crash of the business system end.

[0070] In one implementation, when exporting large table data on the business system side, in order to optimize performance and avoid deep paging problems, the following strategies can be adopted:

[0071] (1) Use a slave database to export large table data to a read-only slave database to reduce the pressure on the master database. Read-write separation is an effective method to solve high-concurrency scenarios and large data volume query and export;

[0072] (2) Pay attention to deep paging: As the paging goes deeper (i.e. the offset increases), the query speed will slow down significantly. It is necessary to optimize the LIMIT paging operation, which can improve performance by reducing memory burden and network transmission overhead;

[0073] (3) Optimize query statements: Avoid using large transactions and multi-table joins, and minimize the use of multi-table joins and large table joins. Optimizing query statements can improve execution efficiency.

[0074] (3) Data splitting: For database usage requirements with extremely large data volumes, high concurrency, and low latency, data can be split according to the data subject and solved with the help of global secondary indexes and eventually consistent transactions.

[0075] (4) Avoid using large transactions: split large transactions into small transactions and submit transactions in a timely manner to avoid lock waits.

[0076] (5) Monitoring and optimizing slow query logs is an effective way to improve database performance;

[0077] (6) Request data on demand: Avoid using select*. Requesting data on demand can reduce I / O, memory burden, and network transmission overhead.

[0078] (7) Data sharding and read-write separation: For DQL operations, data can be read concurrently based on the read load balancing strategy to achieve data sharding and read-write separation.

[0079] Through the above strategies, we can effectively optimize the export of large table data and improve system performance and user experience.

[0080] In one embodiment, when the total occupied memory amount does not exceed the threshold, after the step of treating the target export task as a new export task in execution and performing report export control on the business system end corresponding to the new export task, it may include: based on the report download mode corresponding to the new export task, performing synchronous download control or asynchronous download control on at least one report file obtained by performing report export control. In this way, the export server in the technical solution of this embodiment can support asynchronous download and synchronous download at the same time.

[0081] In one embodiment, based on the report download mode corresponding to the new export task, the step of performing synchronous download control or asynchronous download control on at least one report file obtained by performing report export control may include: when the report download mode corresponding to the new export task is the synchronous download mode, obtaining the current number of synchronous download tasks; when the number of synchronous download tasks exceeds the number threshold, modifying the report download mode corresponding to the new export task to the asynchronous download mode, so as to perform asynchronous download control on at least one report file obtained by performing report export control. In this way, the technical solution of this embodiment enables the export server to limit the number of synchronous download tasks, and performs asynchronous download control when the number exceeds the limit range, so as to ensure the stability of the export performance of the export server.

[0082] In one embodiment, the step of performing synchronous download control on at least one report file obtained by performing report export control may include: determining the number of synchronous download tasks; determining the download flow limit according to the number of synchronous download tasks; and feeding back the at least one report file obtained by performing report export control to the front end according to the download flow limit. In this way, the technical solution of this embodiment can limit the flow and downgrade according to the increase in the number of synchronous download tasks, so as to avoid the export service overload of the export server due to the increase in the number of synchronous download tasks.

[0083] In one embodiment, the step of performing asynchronous download control on at least one report file obtained by performing report export control may include: uploading at least one report file obtained by performing report export control to a file server according to preset upload rules, and obtaining download index information corresponding to at least one report file fed back by the file server, the download index information including a download address; and feeding back the download index information to the front end.

[0084] In one implementation, the file server may be included in a microservice architecture and may independently provide a report file download service.

[0085] In one embodiment, the preset upload rules include but are not limited to at least one of the following:

[0086] Upload a report file obtained by report export control directly to the file server;

[0087] Upload multiple report files obtained through report export control to the file server respectively;

[0088] The multiple report files obtained by the report export control are compressed to obtain compressed files, and the compressed files are uploaded to the file server.

[0089] In one implementation, the report export method provided in this embodiment may further include: storing the task record corresponding to each export task in the message queue in a database. In this way, the technical solution of this embodiment can achieve the processing trace of the export task, so as to facilitate subsequent task record query.

[0090] In one implementation, the report export method provided in this embodiment may further include: after performing report export control to obtain at least one report file, associating task status information with each report file, and storing the task status information in a database.

[0091] In one implementation, the task status information may include a unique file identifier and a task completion status.

[0092] In one embodiment, the report export method provided in this embodiment may also include: receiving a query task triggered by a front end, and feeding back the execution status of the export task to the front end based on the query task, so that the front end associates and displays the export task in the export task list with its corresponding execution status.

[0093] The report export method provided by the first embodiment of the present application includes the following steps: step S1: select the target export task from the message queue according to the first-in-first-out rule; step S2: determine the total occupied memory according to the target export task and all the export tasks in execution; step S3: when the total occupied memory exceeds the threshold, put the target export task back to the first place in the message queue to wait for re-selection, or when the total occupied memory does not exceed the threshold, use the target export task as a new export task in execution, and perform report export control for the business system end corresponding to the new export task. In this way, the technical scheme of the first embodiment of the present application can make the processing occupied memory corresponding to all the export tasks in execution in the export service end not exceed the threshold, so that the memory overflow can be avoided when the export service end asynchronously and concurrently processes multiple export tasks (that is, when exporting a report file with a large amount of data in the case of asynchronous processing with high concurrency, the memory overflow can be avoided). Therefore, the technical scheme of the first embodiment of the present application can avoid memory overflow when the export service executes the export task, thereby ensuring the stability of the export performance of the export service and the high efficiency of the export efficiency, and improving the user experience.

[0094] The technical solution of this embodiment enables the export server to implement a general export service, which can focus on executing export tasks to obtain report files and can be deployed as an independent microservice.

[0095] Second embodiment

[0096] See also Figure 3 , Figure 3 A structural diagram of a report export system provided in the second embodiment of the present application.

[0097] The report export system provided in this embodiment includes: a business system end A1 and a microservice architecture B1.

[0098] Among them, the microservice architecture B1 is used to access the business system end A1 and serves as a report export interface for the business system end A1.

[0099] Among them, the microservice architecture B1 includes a front-end B11, an export server B12, and a file server B13.

[0100] Among them, the front end B11 is used to respond to the export operation on the business system end A1, obtain and send the report export request to the export server B12, so that the export server B12 determines the export task according to the received report export request and adds the export task to the message queue.

[0101] Among them, the export server B12 is used to select the target export task from the message queue according to the first-in-first-out rule, and determine the total occupied memory based on the target export task and all export tasks in execution. When the total occupied memory exceeds the threshold, the target export task is put back to the first place in the message queue to wait for re-selection, or when the total occupied memory does not exceed the threshold, the target export task is used as a new export task in execution, and report export control is performed for the business system end A1.

[0102] In one implementation, the specific implementation and beneficial effects of the export server B12 in this embodiment can refer to the report export method provided in the first embodiment, and will not be described in detail here.

[0103] In one embodiment, the export service end B12 may include an export service module and a business microservice module. The export service module may be used to independently implement the control logic of the export task; the business microservice module is used to implement the interface function of calling various data (such as report data) in the business system end A1, so as to provide an interface according to the control logic of the export task of the export service module, so that the export service module can call or pull various data in the business system end A1.

[0104] In one implementation, the export service module and the business microservice module may also be two independent services, that is, development, testing, and deployment are all performed independently.

[0105] The file server B13 is used to receive the report file obtained and uploaded by the export server B12 after performing report export control, and to provide a download service for the report file.

[0106] Based on the same inventive concept as the above embodiments, the microservice architecture in the report export system is exemplarily described below through a specific application scenario:

[0107] The supported devices for the specific application scenarios of this example include but are not limited to:

[0108] (1) MySQL database: used for persistent storage of export tasks, export results, and various configuration information about export tasks on the business system side (such as configuration information in export restriction rules);

[0109] (2) Redis database: used to cache the queue of export tasks to be executed and the export tasks in execution;

[0110] (3) Message queue (middleware): decoupling the export service module from the front end to implement an asynchronous mechanism;

[0111] (4) ES: used to store export operation audit logs.

[0112] The process steps of the report export method corresponding to the specific application scenario of this example are divided into two stages:

[0113] Phase 1: Generation and execution of export tasks (see Figure 4 )

[0114] 1) Front-end: The user operates the business document report list, customizes the export fields and export order, and clicks the export operation; in this way, the front-end obtains and sends the report export request to the export service module;

[0115] 2) Export service module: after receiving the report export request, the request method, request parameters, query conditions, data query address and user login information included in the report export request are written into the message queue;

[0116] Optionally, after completing the writing of information in the message queue based on the report export request, information indicating successful request can be returned to the front end;

[0117] 3) Export service module: consumes messages in the message queue, and calls the business system end through the business microservice module to obtain export data information according to the request method, data query address, query conditions, and user login information in the message. Then, according to the total number of data and the number of fields and field types in the exported data information, calculates the memory size required, stores the relevant information in the priority message queue in the Redis database to obtain the export task, and stores the task record of the export task in the MySQL database;

[0118] Optionally, the relevant information may include parameters in the report export request, the size of the occupied memory and / or the time when the report export request was received, etc.

[0119] 4) Export service module: Based on the local scheduled task, obtain the export task A from the priority message queue in the Redis database in the first-in-first-out priority order;

[0120] When there is a risk of memory overflow when executing export task A, task A will not be executed temporarily, and export task A will be put back to the top of the priority message queue according to its priority;

[0121] When there is no risk of memory overflow when executing export task A, execute export task A and write the executing export task A into the local cache task pool in the Redis database;

[0122] Optionally, if the maximum memory occupied by executing export task A plus the sum of the maximum memory occupied by all running export tasks in the export service module exceeds a preset threshold, it is determined that there is a memory overflow risk in executing export task A; otherwise, it is determined that there is no memory overflow risk in executing export task A.

[0123] 5) Export service module: modify the status of export task A to being executed, and start executing export task A; dynamically enable multiple threads according to the export restriction rules preset by the business system end, so that multiple threads obtain business document export tasks respectively according to the parameters in the report export request corresponding to export task A (such as request method, request parameters, query conditions, and data query address), so as to pull report data from the business system end in parallel through business microservices;

[0124] 6) The export service module calculates whether it is necessary to write multiple Excel files based on the total number of business document export tasks and the maximum number of rows in the Excel file corresponding to each business document export task. When there are more than one Excel file, they can be written in parallel to multiple Excel files of the local service. After all are written, the Excel files are zipped to obtain compressed files and uploaded to the file server. When only one Excel file is needed, after the report data is written, it is directly uploaded to the file server, and the unique identifier of the Excel file and the task completion status are written to the MySQL database for persistence, and the export task A is removed from the local cache task pool in the Redis database.

[0125] Optionally, the export server can support both asynchronous download and synchronous download. When downloading synchronously, the number of synchronous download tasks will be limited, and it is not allowed to exceed the number threshold, otherwise only asynchronous download can be used.

[0126] Optionally, the export server may perform flow limiting and downgrading according to the increase in the number of synchronous download tasks, so as to avoid overloading the export service of the export server due to the increase in the number of synchronous download tasks.

[0127] Stage 2: Viewing the results of the export task and downloading the file (see Figure 5 )

[0128] 1) The front end triggers a query task in response to the user's viewing operation, so that the export service module returns a task list, thereby allowing the user to view the export task list and view the task execution status;

[0129] 2) The export service module feeds back to the front end the download address of the compressed file or Excel file on the file server corresponding to the export task that represents the completion status of the completed task;

[0130] 3) The front end triggers a download request to the export service module in response to the user's download operation, so that the export service module writes the download log record to ES, and accesses the file server to download the compressed file or Excel file corresponding to the file unique identifier through the download address and the file unique identifier in the download request.

[0131] 4) The user views the export task list and the task execution status. When a task fails, the user can click the retry button to trigger the export server to retry the export task.

[0132] In summary, the report export method of the above example has the following beneficial effects:

[0133] (1) In the technical solution of this example, the request parameters corresponding to the export task include user login information. When the time interval between the execution time of the export task and the user login operation of the front-end is too long, resulting in the expiration of the access rights to the business system, the business system can be enabled to open the access rights through the request parameters of the export task itself, thereby ensuring the smooth execution of the export task;

[0134] (2) In the technical solution of this example, the export service module can ensure that the memory occupied by the processing corresponding to all export tasks in execution does not exceed the threshold, thereby avoiding memory overflow when asynchronously and concurrently processing multiple export tasks. Therefore, the export service module can avoid memory overflow when executing export tasks, thereby ensuring the stability of the export performance of the export service and the high efficiency of the export efficiency, thereby improving the user experience; wherein, the export service module performs memory self-check on the service status during the execution of the export task through export task management, thereby ensuring the high efficiency of the export task execution and the stability of the export service. At the same time, the export service module can also limit and downgrade the synchronous download task, thereby further ensuring the high availability of the export service; in addition, the export service module can receive the report export request triggered by the front end through the message queue, and access the business system end through the business microservice module, so that the export service module and the business system end can achieve non-business coupling, thereby reducing the access cost. Furthermore, the control logic of the export task of the export service module can be settled as a core capability, providing a unified standard export interface, which can be connected with the pre-standardized front-end and / or business microservice module to realize the export function of the report file, so that the development of the business system side can focus on the business management itself, without the need to repeatedly develop the export function of the report file, thus reducing the development cost of the business system side;

[0135] (3) In the technical solution of this example, the microservice architecture or the export service module in the microservice architecture can be deployed as an independent microservice, connected to multiple different business system terminals, and realize the export of report files of multiple different business system terminals. Therefore, the microservice architecture or the export service module in the microservice architecture in the technical solution of this example can meet the purpose of one-time development and sharing by multiple different business system terminals; among them, the export service module can be deployed as an independent microservice, support multiple nodes, elastic scaling, and improve resource utilization;

[0136] (4) In the technical solution of this example, the export service module can reasonably enable multiple threads to pull the report data in the business system based on the export restriction rules preset by the business system, which not only improves the query efficiency and controllability of the report data on the business system, but also ensures the stability of the business management performance of the business system itself;

[0137] (5) In the technical solution of this example, export tasks are established through the export service module, and the number of export tasks is controlled, thereby ensuring the high availability of the export performance of the export service. At the same time, by configuring the export restriction rules, export restriction configuration for different business types on the business system side is implemented, which can control the efficiency of querying data on the business system side, so as to ensure a balance between the efficiency and stability of the export tasks on each business system side.

[0138] Based on the same inventive concept as the above embodiments, the present application embodiment provides an export server, such as Figure 6 As shown, the export server includes: a processor 310 and a memory 311 storing a computer program; wherein, Figure 6 The processor 310 shown in the figure is not used to indicate that the number of the processor 310 is one, but is only used to indicate the positional relationship of the processor 310 relative to other devices. In actual applications, the number of the processor 310 may be one or more; similarly, Figure 6 The memory 311 shown in the figure has the same meaning, that is, it is only used to refer to the position relationship of the memory 311 relative to other devices. In practical applications, the number of memories 311 can be one or more. When the processor 310 runs the computer program, the report export method applied to the above-mentioned export server is implemented.

[0139] The export server may also include: at least one network interface 312. The various components in the device are coupled together via a bus system 313. It is understood that the bus system 313 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 313 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 6 Various buses are labeled as bus system 313.

[0140] The memory 311 may be a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memories. The nonvolatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM); the magnetic surface memory may be a disk memory or a tape memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAM bus random access memory (DRRAM, Direct Rambus Random Access Memory).The memory 311 described in the embodiments of the present application is intended to include but is not limited to these and any other suitable types of memories.

[0141] The memory 311 in the embodiment of the present application is used to store various types of data to support the operation of the export server. Examples of these data include: any computer program for operating on the export server, such as an operating system and an application program, etc. Among them, the operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application program can include various application programs, such as a media player (Media Player), a browser (Browser), etc., for implementing various application services. Here, the program for implementing the report export method of the embodiment of the present application can be included in the application program.

[0142] Based on the same inventive concept as the above-mentioned embodiment, this embodiment further provides a computer-readable storage medium, in which a computer program is stored. The computer-readable storage medium may be a ferromagnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); it may also be various devices including one or any combination of the above memories, such as a mobile phone, a computer, a tablet device, a personal digital assistant, etc. When the computer program stored in the computer-readable storage medium is executed by the processor, the above method is implemented. For the specific steps implemented when the computer program is executed by the processor, please refer to Figure 1 The description of the illustrated embodiment will not be repeated here.

[0143] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0144] In this document, the terms "comprises," "comprising," or any other variations thereof, are intended to cover a non-exclusive inclusion of elements other than those listed and may also include additional elements not expressly listed.

[0145] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A report export method, characterized in that: Applied to the export server, including: According to the first-in-first-out rule, select the target export task from the message queue; Determine the total amount of memory occupied according to the target export task and all export tasks being executed; When the total occupied memory exceeds the threshold, the target export task is put back to the first position of the message queue to wait for selection again; or when the total occupied memory does not exceed the threshold, the target export task is treated as a new export task in execution, and report export control is performed on the business system end corresponding to the new export task.

2. The method according to claim 1, characterized in that Before the step of selecting the target export task from the message queue according to the first-in-first-out rule, the method includes: Receiving a report export request for a business system end, wherein the report export request includes at least one of a request method, a request parameter, a query condition, a data query address, a user login information, and a report download method; Determine, according to the report export request, the memory occupied by the export task and its corresponding first processing; The determined export task is associated with the first processing occupied memory, and the determined export task is stored in the message queue.

3. The method according to claim 1, characterized in that When the total occupied memory amount does not exceed the threshold, the target export task is used as a new export task in execution, and the report export control is performed on the business system end corresponding to the new export task, including: Determine the number of threads allocated to the new export task according to the export restriction rule of the business system end corresponding to the new export task, wherein the export restriction rule includes at least one of the number of pages for asynchronous export of the task, the maximum total number of rows exported, the maximum number of rows per Excel file, the concurrency of the export request batch, and the concurrency within the export request batch; Invoke at least one thread corresponding to the number of threads allocated according to the new export task, so as to pull report data from the business system end corresponding to the new export task through the at least one thread; At least one report file is acquired based on the report data fed back by the at least one thread.

4. The method according to any one of claims 1 to 3, characterized in that: When the total occupied memory amount does not exceed the threshold, the target export task is used as a new export task in execution, and after the step of performing report export control on the business system end corresponding to the new export task, the method includes: Based on the report download mode corresponding to the new export task, synchronous download control or asynchronous download control is performed on at least one report file obtained by performing report export control.

5. The method according to claim 4, characterized in that The step of performing synchronous download control on at least one report file obtained by performing report export control includes: Determine the number of simultaneous download tasks; Determine the download traffic limit according to the number of simultaneous download tasks; According to the download traffic limit, at least one report file obtained by report export control is fed back to the front end.

6. The method according to claim 4, characterized in that The step of performing asynchronous download control on at least one report file obtained by performing report export control includes: Uploading at least one report file obtained by report export control to a file server according to a preset upload rule, and obtaining download index information corresponding to the at least one report file fed back by the file server, wherein the download index information includes a download address; Feeding back the download index information to the front end; The preset upload rule includes at least one of the following: Uploading a report file obtained by report export control directly to the file server; Uploading multiple report files obtained by report export control to the file server respectively; The multiple report files obtained by the report export control are compressed to obtain compressed files, and the compressed files are uploaded to the file server.

7. The method according to any one of claims 1 to 3, characterized in that: Also includes: The task record corresponding to each export task in the message queue is stored in the database; and / or, After performing report export control to obtain at least one report file, task status information is associated with each report file, and the task status information is stored in the database, wherein the task status information includes a file unique identifier and a task completion status.

8. A report export system, characterized in that: Including business system side and microservice architecture; The microservice architecture is used to access the business system end and serve as a report export interface of the business system end, and the microservice architecture includes a front end, an export service end and a file service end; The front end is used to respond to the export operation on the business system end, obtain and send a report export request to the export server end, so that the export server end determines the export task according to the received report export request, and adds the export task to the message queue; The export server is used to select a target export task from the message queue according to a first-in-first-out rule, and determine the total occupied memory amount according to the target export task and all export tasks in execution, and when the total occupied memory amount exceeds a threshold, put the target export task back to the first position of the message queue to wait for reselection, or when the total occupied memory amount does not exceed the threshold, use the target export task as a new export task in execution, and perform report export control for the business system end; The file server is used to receive the report file obtained and uploaded by the export server after the report export control is performed, and to provide a download service for the report file.

9. An export server, characterized in that: include: A processor and a memory storing a computer program, when the processor runs the computer program, implements the steps of the report exporting method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the steps of the report export method according to any one of claims 1 to 7 are implemented.

Images