Data real-time association method supporting total update

Abstract

The invention provides a real-time data association method supporting total update. The method includes a data acquisition module, a data management module, a Redis memory database, and a data association module, the data acquisition module acquires total data required to be synchronized from an external file or an external database and processes the total data; then, the data in the correspondingformat is transmitted to the data management module; the data management module processes the received data; inserting or updating the data into a Redis memory database; the Redis memory database stores various types of associated data, data synchronization is carried out in a master-slave mode, the data association module receives a data association query request, then an association result is queried from the Redis memory database, and after type conversion, the association result is returned to the data association query request. Under the condition of total data updating, the number of data updating in the Redis memory database is reduced, occupation of network and system resources during master-slave synchronization of the Redis memory database is greatly reduced, and the efficiencyof using the Redis memory database for data association can be greatly improved.

Description

Technical field [0001] The present invention relates to the field of big data processing, in particular to a real-time data association method supporting full update. Background technique [0002] With the development of Internet technology, the application of big data processing and cloud computing technology has become more and more extensive. Many business applications need to associate and integrate data from different sources in order to better tap the value of data. In the DT era, more and more application scenarios have put forward higher requirements for the timeliness of data, and real-time computing has become more and more important. When a piece of data is generated, if it cannot be processed in time and used in the business system, the value of the data cannot be maximized. [0003] Redis is an open source log-based, Key-Value database written in ANSI C, supporting the network, memory-based or persistent, and provides APIs in multiple languages. Redis is an in-memory...

AI Technical Summary

Problems solved by technology

When using Redis as a cache database to cache basic data, if the application scenario needs to update the basic data in full every time, it will cause an instantaneous decline in Redis performance, especially when the master-slave mode is used. The problem is more prominent

Therefore, it is necessary to find a solution to solve the problem of instantaneous decline in Redis performance during full update

Images