Mass data export solution

Abstract

The invention discloses a solution for exporting mass data, which belongs to the technical field of electric digital data processing, and comprises the following steps: back-end implementation: receiving a front-end request, and receiving the request from the front end by setting an application programming interface at the back end; according to the method, the data is synchronously queried, and after the rear end receives the request of the front end, the rear end synchronously queries the required data from the database or other data sources, export is carried out in an asynchronous task mode, export operation only triggers and generates an export task, and the data export efficiency is greatly improved. According to the method, all the data to be exported is really fished for data assembly to generate the file, the generated task is asynchronously executed through the rear end for processing, the file is uploaded to the oss server after the file is assembled and generated through the rear end, a user can see the current processing progress in real time on a downloading task management interface, the user can directly download the current processing progress from the oss server after processing is completed, and the user experience is improved. A lot of waiting time of the user is saved, and the same data can be repeatedly downloaded by the user without being regenerated.

Description

Technical Field

[0001] The invention belongs to the technical field of electronic digital data processing, and in particular relates to a solution for exporting large quantities of data. Background Art

[0002] The front-end directly requests the back-end to obtain files. The back-end synchronously queries the data and assembles the data to generate files and returns the file stream to the front-end for download to the user's local computer. Whether large amounts of data can be exported normally, how long the export takes, and whether the progress is visible are important indicators of the interaction performance between the front-end and the back-end.

[0003] The prior art discloses some invention patents in the field of electronic digital data processing technology, among which the invention patent with publication number CN107656968B discloses a method and system for exporting large quantities of business data, wherein the method comprises: step 1, obtaining a business data export request, wherein the business data export request includes a target business ID of the export request and a start time of target data in the target business, and storing the export file request in the same form of a MYSQL database; step 2, regularly obtaining the file export request in the MYSQL database, judging whether the number of data items of the target business in the start time period of the target data exceeds a preset threshold m, and if so, obtaining m data items each time until all data of the target business in the start time period of the target data are obtained. The technical solution has the beneficial effect of reducing delay and failure risk when exporting large quantities of business data. However, the technical solution still has some shortcomings in the process of application, and is prone to being stuck in a data collection and processing stage, and a connection timeout error occurs when the time is too long, which leads to a poor user experience and inability to know the current processing progress, and the user needs to wait for the export result on the current page before performing other operations, and when the exported file is accidentally deleted by the user, the entire export process needs to be repeated.

[0004] Based on this, the present invention designs a large-volume data export solution to solve the above-mentioned problem. Summary of the invention

[0005] The purpose of the present invention is to solve the problems that the prior art still has some shortcomings in its application process, which easily causes the system to get stuck in a data collection and processing stage, resulting in a connection timeout error due to too long a time, which in turn leads to a poor user experience and inability to know the current processing progress. The user needs to wait for the export result on the current page before performing other operations. When the user accidentally deletes the exported file, the entire export flow needs to be re-exported. A solution to exporting large amounts of data is proposed.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] Bulk data export solutions, including:

[0008] Backend implementation:

[0009] Receive front-end requests by setting up an application programming interface on the back-end to receive requests from the front-end;

[0010] Synchronous query data: after the backend receives the request from the frontend, it synchronously queries the required data from the database or other data sources;

[0011] Data assembly: assemble or process the data queried from the backend as necessary to meet the needs of file generation;

[0012] Generate files using the file operation library provided by the backend language;

[0013] Set the file stream response, set the file generated by the backend as the HTTP response content, and set the correct Content-Type and Content-Disposition headers;

[0014] Send file stream, send the file stream to the front end;

[0015] Front-end implementation:

[0016] Send a request. The front end uses the application programming interface to send a request to the back end application programming interface;

[0017] Process the response. When receiving the response from the backend, the frontend needs to process the response in time. Since the backend has set the correct Content-Disposition header, the browser usually automatically triggers the download dialog box.

[0018] When the user downloads, he or she can choose to save the file locally after seeing the download dialog box option.

[0019] As a further description of the above technical solution:

[0020] The application programming interface is one or more of GET, POST, PUT and DELETE, which is set according to the requirements and data volume.

[0021] As a further description of the above technical solution:

[0022] When the amount of data required for the backend synchronous query is large, an asynchronous processing strategy is used.

[0023] As a further description of the above technical solution:

[0024] When using the asynchronous processing strategy, export is performed through asynchronous tasks. The export operation only triggers the generation of an export task. The actual acquisition of all the data to be exported and the data assembly to generate the file is processed by the task generated by the asynchronous execution of the back-end. After the back-end assembles the generated file, it uploads the file to the OSS server. The user can see the current processing progress in real time in the download task management interface. After the processing is completed, the user can download it directly from the OSS server, saving the user a lot of waiting time. For the same data, the user can download it repeatedly without regenerating it.

[0025] As a further description of the above technical solution:

[0026] The interface design between the front end and the database is based on SpringBoot, which mainly includes the basic data layer, data persistence layer, business logic layer, control layer and user view layer, and each layer needs to design a corresponding interface for interaction between layers.

[0027] As a further description of the above technical solution:

[0028] The basic data layer mainly designs database tables and stores data by using data management tools. The backend part is mainly responsible for data acquisition operations and various business logic processing. The data persistence layer mainly accesses the database through the persistence layer framework.

[0029] As a further description of the above technical solution:

[0030] The business logic layer mainly processes the data information of each module through Spring. The control layer is mainly responsible for obtaining user request access and receiving data from the front end, and injecting it into the Service layer. The front-end view layer is mainly responsible for interface interaction and data communication between components. The Vue project is built through the progressive JavaScript framework, and the routing management module is used for routing management. Axios is used to send get and post requests to the back end to obtain back-end data, and it is displayed in a normal way and in a data visualization way.

[0031] As a further description of the above technical solution:

[0032] The data queried by the backend is assembled to meet the needs of file generation, which usually involves data collection, data cleaning, data conversion and data formatting;

[0033] Data collection obtains data from the backend through an application programming interface;

[0034] After acquiring the data, clean the data to remove unnecessary fields or process outliers;

[0035] Convert data into a specific format based on the requirements of file generation;

[0036] Perform more complex formatting on the data.

[0037] As a further description of the above technical solution:

[0038] The file formats include CSV, Excel, PDF and JSON, and files in corresponding formats are generated according to requirements.

[0039] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0040] 1. In the present invention, export is performed by asynchronous tasks. The export operation only triggers the generation of an export task. The actual acquisition of all the data to be exported for data assembly and file generation is processed by the task generated by asynchronous execution of the back-end. After the back-end assembles and generates the file, the file is uploaded to the OSS server. The user can see the current processing progress in real time in the download task management interface. After the processing is completed, the user can download it directly from the OSS server, saving the user a lot of waiting time. For the same data, the user can download it repeatedly without regenerating it.

[0041] 2. In the present invention, the interface design between the front end and the database is based on SpringBoot, mainly including the basic data layer, the data persistence layer, the business logic layer, the control layer and the user view layer, and each layer needs to design the corresponding interface for interaction between the layers. The basic data layer mainly uses the data management tool to design the database table and the data storage operation. The back end part is mainly responsible for data acquisition operations and various business logic processing. The data persistence layer mainly accesses the database through the persistence layer framework. The business logic layer mainly processes the data information of each module through Spring. The control layer is mainly responsible for obtaining the user's request access and receiving the data from the front end, and injecting it into the Service layer. The front view layer is mainly responsible for the interaction of the interface and the data communication between components. The Vue project is constructed through the progressive JavaScript framework, and the routing management module is used for routing management. Axios is used to send get and post requests to the back end to obtain the back end data, and it is displayed in a normal way and in a data visualization way. Finally, the function of data interaction between the database and the front end interface is realized, and the data visualization technology is used to intuitively display the data to the front end. The design has strong reusability and scalability, can be applied in the remote monitoring system, and meets the design requirements.

[0042] 3. In the present invention, the data queried by the back end is assembled to meet the needs of file generation, which involves data collection, data cleaning, data conversion and data formatting. Data collection obtains data from the back end through an application programming interface. After obtaining the data, the data is cleaned to remove unnecessary fields or process abnormal values. According to the needs of file generation, the data is converted into a specific format, the data is formatted in a more complex way, and error handling logic is added to deal with abnormal situations such as data query failure and file generation failure. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A flow chart of the bulk data export solution proposed by the present invention;

[0044] Figure 2 This is a diagram of the interface design framework between the front end and the database in the large-volume data export solution proposed by the present invention;

[0045] Figure 3 This is a schematic diagram of the export process using an asynchronous task in the bulk data export solution proposed by the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] Please see attached Figure 1 -Attached Figure 3 The present invention provides a technical solution: a method for exporting large quantities of data, comprising:

[0048] Backend implementation:

[0049] Receive front-end requests by setting up an application programming interface on the back-end to receive requests from the front-end;

[0050] Synchronous query data: after the backend receives the request from the frontend, it synchronously queries the required data from the database or other data sources;

[0051] Data assembly: assemble or process the data queried from the backend as necessary to meet the needs of file generation;

[0052] Generate files using the file operation library provided by the backend language;

[0053] Set the file stream response, set the file generated by the backend as the HTTP response content, and set the correct Content-Type and Content-Disposition headers;

[0054] Send file stream, send the file stream to the front end;

[0055] Front-end implementation:

[0056] Send a request. The front end uses the application programming interface to send a request to the back end application programming interface;

[0057] Process the response. When receiving the response from the backend, the frontend needs to process the response in time. Since the backend has set the correct Content-Disposition header, the browser usually automatically triggers the download dialog box.

[0058] When the user downloads, he or she can choose to save the file locally after seeing the download dialog box option.

[0059] Specifically, the application programming interface is one or more of GET, POST, PUT and DELETE, which is set according to the requirements and data volume.

[0060] Specifically, when the amount of data required for the backend synchronous query is large, an asynchronous processing strategy is used.

[0061] Specifically, using the asynchronous processing strategy, export is performed by means of asynchronous tasks. The export operation only triggers the generation of an export task. The actual acquisition of all the data to be exported and the data assembly to generate the file is processed by the task generated by the asynchronous execution of the back-end. After the back-end assembles and generates the file, the file is uploaded to the OSS server. The user can see the current processing progress in real time in the download task management interface. After the processing is completed, the user can download it directly from the OSS server, saving the user a lot of waiting time. For the same data, the user can download it repeatedly without regenerating it.

[0062] Specifically, the interface design between the front end and the database is based on SpringBoot, which mainly includes the basic data layer, data persistence layer, business logic layer, control layer and user view layer, and each layer needs to design a corresponding interface for interaction between layers.

[0063] Specifically, the basic data layer mainly uses data management tools to design database tables and store data, the backend part is mainly responsible for data acquisition operations and various business logic processing, and the data persistence layer mainly accesses the database through the persistence layer framework.

[0064] Specifically, the business logic layer mainly processes the data information of each module through Spring. The control layer is mainly responsible for obtaining user request access and receiving data from the front end, and injecting it into the Service layer. The front-end view layer is mainly responsible for interface interaction and data communication between components. The Vue project is built through the progressive JavaScript framework, and the routing management module is used for routing management. Axios is used to send get and post requests to the back end to obtain back-end data, and it is displayed in a normal way and in a data visualization way.

[0065] Specifically, the data queried by the backend is assembled to meet the needs of file generation, which usually involves data collection, data cleaning, data conversion and data formatting;

[0066] Data collection obtains data from the backend through an application programming interface;

[0067] After acquiring the data, clean the data to remove unnecessary fields or process outliers;

[0068] Convert data into a specific format based on the requirements of file generation;

[0069] Perform more complex formatting on the data.

[0070] Specifically, the file formats include CSV, Excel, PDF and JSON, and files of corresponding formats are generated according to requirements.

[0071] Working principle, when using:

[0072] Backend implementation:

[0073] Receive front-end requests by setting up an application programming interface on the back-end to receive requests from the front-end. The application programming interface is one or more of GET, POST, PUT and DELETE, which is set according to the needs and data volume;

[0074] Synchronous query data. After the backend receives the request from the frontend, the backend synchronously queries the required data from the database or other data sources. When the amount of data required for the backend synchronous query is large, an asynchronous processing strategy is used. In the asynchronous processing strategy, the export is performed through asynchronous tasks. The export operation only triggers the generation of an export task. The actual retrieval of all the data to be exported for data assembly and file generation is processed by the tasks generated by asynchronous execution of the backend. After the backend assembles and generates the file, it uploads the file to the OSS server. The user can see the current processing progress in real time in the download task management interface. After the processing is completed, the user can download it directly from the OSS server, saving the user a lot of waiting time. For the same data, users can download it repeatedly without regenerating it. The interface design between the frontend and the database is based on SpringBoot, mainly including the basic data layer, data persistence layer, business logic layer, control layer and user view layer , and each layer needs to design corresponding interfaces for interaction between layers. The basic data layer mainly uses data management tools to design database tables and store data. The back-end part is mainly responsible for data acquisition operations and various business logic processing. The data persistence layer mainly accesses the database through the persistence layer framework. The business logic layer mainly processes the data information of each module through Spring. The control layer is mainly responsible for obtaining user request access and receiving data from the front end, and injecting it into the Service layer. The front-end view layer is mainly responsible for interface interaction and data communication between components. The Vue project is built through the progressive JavaScript framework, and the routing management module is used for routing management. Axios is used to send get and post requests to the back end to obtain back-end data, and it is displayed in a normal way and in a data visualization way.

[0075] Data assembly, which involves necessary assembly or processing of the data queried by the backend to meet the needs of file generation. The data queried by the backend is assembled to meet the needs of file generation, involving data collection, data cleaning, data conversion and data formatting. Data collection obtains data from the backend through the application programming interface. After obtaining the data, the data is cleaned to remove unnecessary fields or process abnormal values. According to the needs of file generation, the data is converted into a specific format and the data is formatted in a more complex way;

[0076] Generate files using the file operation library provided by the backend language. The file formats include CSV, Excel, PDF and JSON. Generate files in corresponding formats according to requirements;

[0077] Set the file stream response, set the file generated by the backend as the HTTP response content, and set the correct Content-Type and Content-Disposition headers;

[0078] Send file stream, send the file stream to the front end;

[0079] Front-end implementation

[0080] Send a request. The front end uses the application programming interface to send a request to the back end application programming interface;

[0081] Process the response. When receiving the response from the backend, the frontend needs to process the response in time. Since the backend has set the correct Content-Disposition header, the browser usually automatically triggers the download dialog box.

[0082] When the user downloads, he or she can choose to save the file locally after seeing the download dialog box option.

[0083] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A solution to export large amounts of data, characterized in that: include: Backend implementation: Receive front-end requests by setting up an application programming interface on the back-end to receive requests from the front-end; Synchronous query data: after the backend receives the request from the frontend, it synchronously queries the required data from the database or other data sources; Data assembly: assemble or process the data queried from the backend as necessary to meet the needs of file generation; Generate files using the file operation library provided by the backend language; Set the file stream response, set the file generated by the backend as the HTTP response content, and set the correct Content-Type and Content-Disposition headers; Send file stream, send the file stream to the front end; Front-end implementation: Send a request. The front end uses the application programming interface to send a request to the back end application programming interface; Process the response. When receiving the response from the backend, the frontend needs to process the response in time. Since the backend has set the correct Content-Disposition header, the browser usually automatically triggers the download dialog box. When the user downloads, he or she can choose to save the file locally after seeing the download dialog box option.

2. The method for exporting large amounts of data according to claim 1, characterized in that: The application programming interface is one or more of GET, POST, PUT and DELETE, which is set according to the requirements and data volume.

3. The method for exporting large amounts of data according to claim 1, characterized in that: When the amount of data required for the backend synchronous query is large, an asynchronous processing strategy is used.

4. The method for exporting large amounts of data according to claim 3, characterized in that: When using the asynchronous processing strategy, the export is performed by means of an asynchronous task. The export operation only triggers the generation of an export task. The actual acquisition of all the data to be exported and the data assembly and file generation are processed by the tasks generated by the asynchronous execution of the backend.

5. The method for exporting large amounts of data according to claim 4, characterized in that: After the backend assembles and generates the file, it uploads the file to the OSS server. The user can see the current processing progress in real time on the download task management interface. After the processing is completed, the user can download it directly from the OSS server, saving the user a lot of waiting time. For the same data, users can download it repeatedly without regenerating it.

6. The method for exporting large amounts of data according to claim 5, characterized in that: The interface design between the front end and the database is based on SpringBoot, which mainly includes the basic data layer, data persistence layer, business logic layer, control layer and user view layer, and each layer needs to design a corresponding interface for interaction between layers.

7. The method for exporting large amounts of data according to claim 6, characterized in that: The basic data layer mainly designs database tables and stores data by using data management tools. The backend part is mainly responsible for data acquisition operations and various business logic processing. The data persistence layer mainly accesses the database through the persistence layer framework.

8. The method for exporting large amounts of data according to claim 7, characterized in that The business logic layer mainly processes the data information of each module through Spring. The control layer is mainly responsible for obtaining user request access and receiving data from the front end, and injecting it into the Service layer. The front-end view layer is mainly responsible for interface interaction and data communication between components. The Vue project is built through the progressive JavaScript framework, and the routing management module is used for routing management. Axios is used to send get and post requests to the back end to obtain back-end data, and it is displayed in a normal way and in a data visualization way.

9. The method for exporting large amounts of data according to claim 1, characterized in that: Assembling the data queried by the backend to meet the needs of file generation, usually involving data collection, data cleaning, data conversion and data formatting; Data collection obtains data from the backend through an application programming interface; After acquiring the data, clean the data to remove unnecessary fields or process outliers; Convert data into a specific format based on the requirements of file generation; Perform more complex formatting on the data.

10. The method for exporting large amounts of data according to claim 9, characterized in that: The file formats include CSV, Excel, PDF and JSON, and files in corresponding formats are generated according to requirements.

Images