RabbitMQ and MongoDB based download system
A task, consumer technology applied in the computer field to save request response time, improve performance, and shorten development time
Active Publication Date: 2017-12-08
GUANGDONG POLYTECHNIC NORMAL UNIV
View PDF4 Cites 4 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0005] In order to solve the problem of the inefficiency of Mongo database as a message queue and the need for complex data conversion operations when RabbitMQ and general databases execute task queues during the downloading process of the distributed crawler system, the present invention provides a system based on RabbitMQ and MongoDB The download system combines the advantages of the two, reduces the resource consumption of distributed crawlers during the download process and improves the performance of system downloads
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0039] In order to better understand the technical solution proposed by the present invention, the present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.
[0040] Such as figure 1 As shown, a download system based on RabbitMQ and MongoDB, including MongoDB 1, producer module 2, RabbitMQ 3, consumer module 4, said MongoDB 1 is configured to store tasks and receive data returned by producer module 2;
[0041] The producer module 2 is used to interact with MongoDB 1, query tasks, and transfer tasks to RabbitMQ 3; the producer module 2 is used to receive and feed back RabbitMQ 3 tasks, and return task updates to MongoDB 1;
[0042] The RabbitMQ 3 is configured to transfer task data between the producer module 2 and the consumer module 4;
[0043] The consumer module 4 receives and executes the task imported by RabbitMQ 3, and feeds back the update of the task to the producer module 2 through RabbitMQ 3.
[0044] ...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention provides a RabbitMQ and MongoDB based download system. The RabbitMQ and MongoDB based download system includes a MongoDB, a producer module, a RabbitMQ, and a consumer module; the MongoDB is configured to store tasks and receive data returned by the producer module; the producer module is used for interacting with the MongoDB, querying the tasks, and transmitting the tasks to the RabbitMQ; the producer module is used for receiving a RabbitMQ task, and returning an update of the task to the MongoDB; the RabbitMQ is configured to transmit task data between the producer module and the consumer module; the consumer module receives and executes a task transmitted by the RabbitMQ, and feeds the update of the task to the producer module via the RabbitMQ. The download system uses the MongoDB as a task source and uses the RabbitMQ as a message queue, can simplify complex serialization operation during a task execution process by using a document-oriented database and JSON-like characteristics of the MongoDB, can reduce resource consumption, and can improve the system performance.
Description
technical field [0001] The invention belongs to the field of computers and relates to a download system, in particular to a download system based on RabbitMQ and MongoDB. Background technique [0002] MongoDB (Mongo database) is a highly scalable NoSQL database with automatic sharding, large-scale operations on big data, and high concurrency. MongoDB is a document-oriented database that stores data in the json-like binary data format BSON. [0003] RabbitMQ is a high-concurrency and high-reliability AMQP (Advanced Message Queuing Protocol) message queue middleware implemented in Erlang language. It receives messages from producers and delivers them to consumers. During this process, it implements caching and persistence according to routing rules. Therefore, using RabbitMQ for task distribution, remote call and other services can extract some time-consuming operations that do not require immediate return and perform asynchronous processing, and this asynchronous processing...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30G06F9/54H04L29/08
CPCG06F9/546G06F16/25G06F16/27G06F2209/547H04L67/565H04L67/56H04L67/60
Inventor 马震远林智勇刘少鹏方浩生林安成
Owner GUANGDONG POLYTECHNIC NORMAL UNIV