Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

MongoDB data real-time synchronization method and system based on message queue

A technology of message queue and real-time synchronization, applied in the field of computer information, can solve the problems of slow synchronization rate and synchronization data loss, and achieve the effect of increasing rate, data synchronization rate and small delay.

Pending Publication Date: 2020-07-03
INST OF INFORMATION ENG CHINESE ACAD OF SCI
View PDF5 Cites 5 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0012] The purpose of the present invention is to propose a method and system for real-time synchronization of MongoDB data based on message queues, aiming to realize real-time data synchronization from MongoDB to the target system ElasticSearch, and to solve the problem of slow synchronization rate and due to The problem of synchronous data loss caused by the rapid writing of MongoDB ensures fast, accurate and reliable data synchronization from MongoDB to ElasticSearch

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • MongoDB data real-time synchronization method and system based on message queue
  • MongoDB data real-time synchronization method and system based on message queue
  • MongoDB data real-time synchronization method and system based on message queue

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0044] In order to make the above-mentioned features and advantages of the present invention more comprehensible, the following specific embodiments are described in detail in conjunction with the accompanying drawings.

[0045] This embodiment provides a real-time MongoDB data synchronization system based on a message queue, which belongs to a data synchronization tool with a cache mechanism, and is used to solve the problem of data loss caused by a small oplog capacity when there is a large amount of data, and then realize Reliable data synchronization from MongoDB to ElasticSearch. In the present invention, the data stream is directly synchronized from MongoDB to ElasticSearch instead of being cached from MongoDB to Kafka first, and then synchronized from Kafka to ElasticSearch, such as figure 1 shown.

[0046] This system includes five modules: connection creation module, data reading module, data caching module, data synchronization module and naming resolution module, s...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a MongoDB data real-time synchronization method and system based on a message queue. The objective of the invention is to realize real-time data synchronization from MongoDB toElasticSearch of a target system. The problems that an original synchronization tool is low in synchronization rate in the data synchronization process and synchronization data is lost due to the rapid writing condition of MongoDB are solved, and rapid, accurate and reliable data synchronization from MongoDB to ElasticSearch is guaranteed.

Description

technical field [0001] The invention relates to the field of computer information technology, in particular to a method and system for synchronizing MongoDB data in real time based on message queues. Background technique [0002] With the advent of the information age, the amount of data has increased dramatically, including various structured data and unstructured data. In the face of large-scale data volume and complex data types, traditional data storage and data retrieval methods have been unable to To meet the needs, the emergence of MongoDB and ElasticSearch (ES for short) effectively solves the problems of data storage and data retrieval under the big data model. [0003] As a mainstream NoSQL database, MongoDB stores documents based on documents. Unlike traditional relational databases, it does not require predefined data schemas. The slave node of the MongoDB replica set asynchronously replicates the data of the master node through the operation log. This operation...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/27G06F16/28G06F16/25
Inventor 戴琼石瑞聪彭怀梁王晓岩
Owner INST OF INFORMATION ENG CHINESE ACAD OF SCI
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com