Unlock instant, AI-driven research and patent intelligence for your innovation.

Asynchronous file exporting method and system based on message queue and fast DFS micro-service architecture

A technology of message queue and file system, which is applied in the field of asynchronous file export method and system, can solve the problems such as not being able to meet the file export requirements, and achieve the effects of reducing storage dependence, enhancing generation stability, and improving download stability

Pending Publication Date: 2020-07-10
山东汇贸电子口岸有限公司
View PDF9 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, this technical solution cannot meet the needs of asynchronously processing files with large amounts of data in the case of high concurrency.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Asynchronous file exporting method and system based on message queue and fast DFS micro-service architecture
  • Asynchronous file exporting method and system based on message queue and fast DFS micro-service architecture

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0043] as attached figure 1 As shown, the asynchronous file export method based on the message queue and fastDFS micro-service architecture of the present invention is completed through the authentication authorization module, message assembly and push module, message consumption processing module, file asynchronous generation and upload module and asynchronous download module For the asynchronous generation and download of files, the distributed deployment is completed through the message queue, and then the loose coupling between modules is realized; the details are as follows:

[0044] S1, the user logs in in the browser (web client), and requests authentication and authorization from the authorization authentication service (Authorition Server) of the authentication and authorization module;

[0045] S2. After the authentication and authorization are passed, the user selects the data to be exported to perform an export operation. Since the export content is different for d...

Embodiment 2

[0058] as attached figure 2 As shown, the asynchronous file export system based on message queue and fastDFS micro-service architecture of the present invention, the system includes,

[0059] The authentication and authorization module is used to authorize the login user. The authentication is performed first, and the file can be exported only after the authentication is passed;

[0060] The message assembly and push module is used to generate a record whose status is being generated after the authentication is passed, and then assemble the message body and send it to the message consumption processing module (MQ message consumer end); the format of the assembled message is determined according to the export type, The export type includes exporting by directly executing sql query data and exporting after obtaining data by calling the interface.

[0061] The message consumption processing module is used for parsing and processing according to the obtained message after receiv...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses an asynchronous file exporting method and system based on a message queue and a fast DFS micro-service architecture. The invention belongs to the technical field of file management, and aims to solve the technical problem of how to export a large data volume of files under the condition of asynchronous processing and high concurrency without occupying too much IO on the service side and improving system stability. According to the technical scheme, asynchronous generation and downloading of files are completed through an authentication and authorization module, a message assembling and pushing module, a message consumption processing module, a file asynchronous generation and uploading module and an asynchronous downloading module, distributed deployment is completed through the message queue, and loose coupling among all the modules is achieved; the method specifically comprises the following steps: S1, logging in a browser by a user, and requesting authentication authorization from an authorization and authentication service of the authentication and authorization module; and S2, after the authentication authorization is passed, the user selects data needing to be exported for exporting operation, and the message assembling and pushing module automatically distinguishes and processes the messages according to services and assembles the messages.

Description

technical field [0001] The invention relates to the technical field of file management, in particular to an asynchronous file export method and system based on a message queue and fastDFS micro-service architecture. Background technique [0002] Although traditional synchronous export and download can meet 80% of file export requirements, for systems with large data volumes and complex business splicing, long waiting times or even timeouts, poor user experience, and high real-time pressure on system services are all problems. It is inevitable. Commonly used java export methods include poi and streaming. Exporting through poi will occupy a large amount of memory in real time. Once other multi-threads are concurrent, memory overflow will occur, causing the system to crash. The streaming export method Although it can meet the current needs to a certain extent, it also occupies the IO of the system in real time, which can easily lead to problems such as waiting for subsequent o...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): H04L29/08H04L29/06G06F16/182G06F21/62
CPCH04L67/06H04L67/1097H04L63/0815H04L63/10G06F16/183G06F21/6218
Inventor 徐士强魏金雷杨继伟
Owner 山东汇贸电子口岸有限公司