A download system based on rabbitmq and mongodb
A task, consumer technology, applied in the computer field, to achieve the effect of improving scalability, increasing throughput, and saving request response time
Active Publication Date: 2019-11-19
GUANGDONG POLYTECHNIC NORMAL UNIV
View PDF4 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0005] In order to solve the problem of the inefficiency of Mongo database as a message queue and the need for complex data conversion operations when RabbitMQ and general databases execute task queues during the downloading process of the distributed crawler system, the present invention provides a system based on RabbitMQ and MongoDB The download system combines the advantages of the two, reduces the resource consumption of distributed crawlers during the download process and improves the performance of system downloads
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0039] In order to better understand the technical solution proposed by the present invention, the present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.
[0040] like figure 1 As shown, a download system based on RabbitMQ and MongoDB, including MongoDB 1, producer module 2, RabbitMQ 3, consumer module 4, said MongoDB 1 is configured to store tasks and receive data returned by producer module 2;
[0041] The producer module 2 is used to interact with MongoDB 1, query tasks, and transfer tasks to RabbitMQ 3; the producer module 2 is used to receive and feed back RabbitMQ 3 tasks, and return task updates to MongoDB 1;
[0042] The RabbitMQ 3 is configured to transfer task data between the producer module 2 and the consumer module 4;
[0043] The consumer module 4 receives and executes the task imported by RabbitMQ 3, and feeds back the update of the task to the producer module 2 through RabbitMQ 3.
[0044] lik...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The present invention provides a download system based on RabbitMQ and MongoDB, including MongoDB, a producer module, RabbitMQ, a consumer module, and the MongoDB is configured to store tasks and receive the data returned by the producer module; the producer module, Used to interact with MongoDB, query tasks, and transfer tasks to RabbitMQ; the producer module is used to receive RabbitMQ tasks, and return task updates to MongoDB; the RabbitMQ is configured to pass the producer module and Task data between consumer modules; the consumer module receives and executes tasks imported by RabbitMQ, and feeds back task updates to the producer module through RabbitMQ. The present invention provides a download system using MongoDB as the task source and RabbitMQ as the message queue, utilizing MongoDB's document-oriented database and JSON-like features to simplify the cumbersome serialization operations in the process of executing tasks, reducing resource consumption and improving efficiency. system performance.
Description
technical field [0001] The invention belongs to the field of computers and relates to a download system, in particular to a download system based on RabbitMQ and MongoDB. Background technique [0002] MongoDB (Mongo database) is a highly scalable NoSQL database with automatic sharding, large-scale operations on big data, and high concurrency. MongoDB is a document-oriented database that stores data in the json-like binary data format BSON. [0003] RabbitMQ is a high-concurrency and high-reliability AMQP (Advanced Message Queuing Protocol) message queue middleware implemented in Erlang language. It receives messages from producers and delivers them to consumers. During this process, it implements caching and persistence according to routing rules. Therefore, using RabbitMQ for task distribution, remote call and other services can extract some time-consuming operations that do not require immediate return and perform asynchronous processing, and this asynchronous processing...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/25G06F16/27G06F9/54G06F11/14H04L29/08
CPCG06F9/546G06F16/25G06F16/27G06F2209/547H04L67/565H04L67/56H04L67/60
Inventor 马震远林智勇刘少鹏方浩生林安成
Owner GUANGDONG POLYTECHNIC NORMAL UNIV