Data synchronization system and data synchronization method

Abstract

The invention relates to the technical field of data storage, and provides a data synchronization system and a data synchronization method, and the data synchronization system comprises a real-time transmission device and an associated proxy service plug-in correspondingly constructed for each data center. An associated proxy service plug-in is constructed, and data transmission and decoupling of an application cluster and Redis are achieved; a plurality of connection channels are established, synchronous data, management strategies and heartbeat information are transmitted through different connection channels, mutual influence among different types of data is isolated, and the data transmission efficiency is improved; the synchronous data is fragmented and compressed, so that the transmission efficiency of data synchronization is improved, and the timeliness of data synchronization is guaranteed; memory calculation is constructed, operations such as reduction are completed in a memory calculation area, and the real-time capability of data synchronization is further guaranteed; by monitoring memory calculation, a transmission state, heartbeat information and the like, the data synchronization efficiency under various scenes such as poor network quality and data transmission accumulation is ensured.

Description

Technical Field

[0001] The present application relates to the technical field of data storage, and in particular to a data synchronization system and a data synchronization method. Background Art

[0002] Redis (Remote Dictionary Server) is a remote dictionary service that supports periodic writing of updated data to disk or writing of modification operations to additional record files, and on this basis provides data synchronization between master and slave nodes. As the digital transformation of enterprises accelerates, business continuity capability building places higher requirements on existing Redis capabilities. In business continuity work, it is usually necessary to build dual-active or multi-active service capabilities for application systems in cross-data center scenarios. As a technical component widely used in application system construction, Redis cross-data center real-time synchronization has become a very critical consideration. In order to support the service requirements of application systems in dual-active or multi-active in different cities, whether as a cache database or a key-value database, building Redis multi-active service capabilities requires building a real-time, efficient, and accurate Redis multi-data center synchronization mechanism.

[0003] The existing solutions have the following technical problems: First, the existing solutions have high data synchronization delays and insufficient real-time data synchronization capabilities. Existing solutions are usually based on the following two implementation modes: one is synchronization based on RDB / AOF data files, and the other is synchronization based on key-value pair scanning mode. There is an obvious sequence dependency between these two data synchronization operations, that is, the data must be stored in the Redis of the main data center first, and then the data stream is formed through RDB / AOF files or scanning key-value pairs. After the relevant data is transferred to other data centers, the Redis in the same city or remote data center can be written. When faced with large amounts of data synchronization, the network is congested, storage is performed before writing, and the delay can reach several minutes to hours, which cannot meet the dual-active or multi-active requirements of the application system.

[0004] Second, the existing solutions lack flexibility in the service process. The existing solutions usually develop corresponding plugins on the Redis component to build the multi-data center synchronization capability, binding the solution tightly with the Redis component. If there are different deployment modes of the Redis architecture in the data center, such as synchronizing from the Cluster mode to the Standalone mode, the existing components need to be modified, and flexible online adjustment cannot be achieved. At the same time, the existing plugin service capabilities are single, such as single reading of RDB files and single writing of key-value pair data streams, etc. It is impossible to achieve flexible switching between modes such as reading and writing, file data streams, etc. by a single component, which results in the existing solution can only complete one-way synchronization and lacks two-way synchronization capabilities. When switching from a data center to the master data center, it cannot support reverse data synchronization, and the flexibility ability is relatively lacking. In addition, when network failures, network congestion, etc. occur, the existing solution cannot flexibly respond to the above abnormal transmission states.

[0005] Third, the existing data synchronization mode lacks technical capabilities such as data classification, sharding, and compression to improve data transmission efficiency. For example, in the RDB data file mode, after the relevant RDB files are transmitted to the destination data center, the RDB files are parsed and written. In the key-value pair scanning mode, the scanned key-value pairs are directly transmitted to the destination data center by relevant components for data writing. Metadata information such as data structure, data volume, and data distribution is transmitted together with the business data, resulting in data transmission delay rate and data transmission accuracy. When large datasets, network congestion, node anomalies, etc. occur, it is impossible to achieve data channel separation, data sharding and compression, data order restoration, data metadata verification, and real-time calculation of management data, resulting in obvious shortcomings in the stability of data synchronization. Summary of the Invention

[0006] This application provides a data synchronization system and a data synchronization method, which constructs an accompanying proxy service plugin and a real-time transmission device, and establishes a connection between the accompanying proxy service plugin and the real-time transmission device through multiple connection channels to solve the above problems.

[0007] This application adopts the following technical solutions: In the first aspect, this application provides a data synchronization system, which includes: a real-time transmission device and an accompanying proxy service plugin constructed for each data center; Each accompanying proxy service plugin is started when an update action occurs in the application cluster of the master data center, and respectively establishes a data transfer connection channel, a policy transmission connection channel, and a heartbeat information connection channel with the real-time transmission device; The associated proxy service plugin corresponding to the main data center reads and synchronizes data from the application cluster of the main data center according to the reading rules, slices and compresses the synchronized data into multiple sub-data blocks according to the slicing rules, and transmits the multiple sub-data blocks to the real-time transmission device through the data transmission connection channel according to the transmission rules; The real-time transmission device performs in-memory calculation on the multiple sub-data blocks to restore them to synchronized data, and transmits the synchronized data to the associated proxy service plugins corresponding to each slave data center through each data transmission connection channel respectively, so that the associated proxy service plugins corresponding to each slave data center write the synchronized data into the Redis of the corresponding slave data center respectively; The real-time transmission device monitors the in-memory calculation and the transmission status of the data transmission connection channel of the associated proxy service plugin corresponding to the main data center, and transmits the monitoring results to the associated proxy service plugin corresponding to the main data center through the policy transmission connection channel; The associated proxy service plugin corresponding to the main data center updates the reading rules and transmission rules according to the monitoring results; Each associated proxy service plugin transmits heartbeat information to the real-time transmission device through the corresponding heartbeat information connection channel; When the real-time transmission device determines that the associated proxy service plugin corresponding to any heartbeat information is abnormal, it generates a node handling strategy, and transmits the node handling strategy to the corresponding associated proxy service plugin through the policy transmission connection channel.

[0008] Optionally, establish the data transmission connection channel, policy transmission connection channel and heartbeat information connection channel based on the WebSocket mechanism, and establish long connections between the real-time transmission device and each connection channel based on the long connection bus.

[0009] Optionally, each associated proxy service plugin includes a basic layer and an application layer; The basic layer includes: a reading service sub-plugin and a writing service sub-plugin; When an update action occurs in the application cluster of the main data center, the basic layer of the associated proxy service plugin corresponding to the main data center activates the reading service sub-plugin, and the basic layer of the associated proxy service plugin corresponding to each slave data center activates the writing service sub-plugin; The application layer includes: a reading rule sub-plugin, a traffic control sub-plugin, a mode management sub-plugin and a node handling sub-plugin; The reading rule sub-plugin adjusts the reading rules according to the monitoring results of the in-memory calculation, the traffic control sub-plugin adjusts the transmission traffic according to the monitoring results of the transmission status, the mode management sub-plugin switches the deployment mode, and the node handling sub-plugin executes the self-healing mechanism on the associated proxy service plugin according to the node handling strategy.

[0010] Optionally, by rewriting the premain method in the Java Agent, the sub-plugins at the basic layer can be made effective when an update action occurs in the application cluster of the primary data center; by rewriting the agentmain method in the Java Agent, the sub-plugins at the application layer can be made effective during the synchronous data transmission process.

[0011] Optionally, the real-time transmission device includes: an incoming phase message queue; the incoming phase message queue includes: a metadata queue and a service data queue; The associated proxy service plugin corresponding to the primary data center slices the synchronous data into multiple service data according to the slicing rule, determines the metadata corresponding to the multiple service data, and respectively identifies the ID information for the metadata corresponding to the multiple service data; Compress the multiple service data into service data blocks respectively, and add the corresponding metadata ID information to the multiple service data blocks respectively; Transmit the multiple service data blocks and the corresponding metadata to the real-time transmission device through the data transmission connection channel according to the transmission rule of the associated proxy service plugin corresponding to the primary data center. Among them, the multiple service data blocks are transmitted to the service data queue, and the corresponding metadata is transmitted to the metadata queue.

[0012] Optionally, the real-time transmission device includes: a memory computing unit, and the memory computing unit is connected to the incoming phase message queue; the memory computing unit includes: a sequential data restoration module, a metadata verification module, and a synchronous data output module; The sequential data restoration module receives the multiple service data blocks and the corresponding metadata, and restores the corresponding multiple service data blocks into sequential data according to the metadata; The metadata verification module sequentially executes the format verification of the metadata corresponding to the sequential data, and transmits the format verification result to the associated proxy service plugin corresponding to the primary data center through the policy transmission connection channel; The synchronous data output module processes the sequential data into synchronous data.

[0013] Optionally, the sequential data restoration module includes: four memory data storage areas and a unified processing area; Divide the metadata and the corresponding service data blocks into different memory data storage areas according to the hash result of the metadata ID information; Restore the multiple service data blocks into sequential data according to the method group loop mode; The method group includes: planning the restoration calculation task, broadcasting the restoration calculation task to each memory data storage area, executing the restoration calculation task in each memory data storage area respectively, sending the results of the restoration calculation tasks executed in each memory data storage area to the unified processing area, and forming sequential data in the unified processing area.

[0014] Optionally, the real-time transmission device includes: an outgoing phase message queue, which is connected to the in-memory computing unit; the outgoing phase message queue includes: outgoing data queues corresponding to each slave data center; The synchronous data transmission module transmits the synchronous data to the outgoing data queues corresponding to each slave data center respectively; Each outgoing data queue transmits the synchronous data to the associated proxy service plug-in corresponding to each slave data center through each data transmission connection channel.

[0015] Optionally, the real-time transmission device includes: a management service unit; the management service unit includes: a data verification module, a capacity evaluation module, a bandwidth monitoring module, a mode management module, a node monitoring module, and a configuration center; During the synchronous data transmission process, the metadata verification module sends the format verification result to the data verification module, and the data verification module transmits the format verification result to the associated proxy service plug-in corresponding to the master data center through the policy transmission connection channel; The capacity evaluation module obtains the data transmission volume of the data transmission connection channel of the associated proxy service plug-in corresponding to the master data center, calculates the transmission capacity threshold according to the data transmission volume, and transmits the transmission capacity threshold to the associated proxy service plug-in corresponding to the master data center through the policy transmission connection channel; The bandwidth monitoring module obtains the data transmission bandwidth of the data transmission connection channel of the associated proxy service plug-in corresponding to the master data center, calculates the bandwidth occupancy ratio according to the data transmission bandwidth, and transmits the bandwidth occupancy ratio to the associated proxy service plug-in corresponding to the master data center through the policy transmission connection channel; The mode management module determines the cluster deployment mode or the single-machine deployment mode according to the mode of each data center; The node monitoring module determines the node health status of the associated proxy service plug-in corresponding to each heartbeat signal, generates a corresponding node handling strategy according to the abnormal node health status, and transmits the node handling strategy to the associated proxy service plug-in corresponding to the node through the policy transmission connection channel; The configuration center provides plug-in configuration information to each associated proxy service plug-in through each policy transmission connection channel.

[0016] In a second aspect, the present application provides a data synchronization method. The data synchronization method is applied to a data synchronization system. The data synchronization system includes: a real-time transmission device and an associated proxy service plug-in constructed for each data center; the data synchronization method includes: Each associated proxy service plug-in is started when an update action occurs in the application cluster of the master data center, and respectively establishes a data transmission connection channel, a policy transmission connection channel, and a heartbeat information connection channel with the real-time transmission device; The associated proxy service plugin corresponding to the main data center reads and synchronizes data from the application cluster of the main data center according to the reading rules, slices and compresses the synchronized data into multiple sub-data blocks according to the slicing rules, and transmits the multiple sub-data blocks to the real-time transmission device through the data transmission connection channel according to the transmission rules; The real-time transmission device performs in-memory calculation on the multiple sub-data blocks to restore them to synchronized data, and transmits the synchronized data to the associated proxy service plugins corresponding to the respective slave data centers through the respective data transmission connection channels, so that the associated proxy service plugins corresponding to the respective slave data centers respectively write the synchronized data into the Redis of the corresponding slave data centers; The real-time transmission device monitors the in-memory calculation and the transmission status of the data transmission connection channel of the associated proxy service plugin corresponding to the main data center, and transmits the monitoring result to the associated proxy service plugin corresponding to the main data center through the policy transmission connection channel; The associated proxy service plugin corresponding to the main data center updates the reading rules and transmission rules according to the monitoring result; Each associated proxy service plugin transmits heartbeat information to the real-time transmission device through the corresponding heartbeat information connection channel; When the real-time transmission device determines that the corresponding associated proxy service plugin is abnormal according to any heartbeat information, it generates a corresponding node handling policy, and transmits the node handling policy to the corresponding associated proxy service plugin through the policy transmission connection channel.

[0017] The above at least one technical solution adopted by this application can achieve the following beneficial effects: Build an associated proxy service plugin to achieve decoupling of data transmission from the application cluster and Redis.

[0018] The associated proxy service plugin supports hot-pluggable service switching, and can realize the synchronous switching of the data synchronization transmission direction to take effect after the role switching of the main and slave data centers.

[0019] Based on the capabilities of the associated proxy service plugin, build the real-time perception ability of the business application cluster and Redis data update, and achieve quasi-real-time synchronous writing of the main and slave data centers.

[0020] Establish multiple connection channels, and transmit synchronized data, management policies, and heartbeat information through different connection channels respectively, isolate the mutual influence between different types of data, and improve the data transmission efficiency.

[0021] During the transmission of synchronized data, first perform slicing and compression processing, and then complete the transmission of the sliced and compressed multiple sub-data blocks according to the transmission rules, improve the transmission efficiency of data synchronization, and ensure the timeliness of data synchronization.

[0022] Build in-memory computing to complete operations such as restoration of synchronized data that has been fragmented and compressed during transmission in the in-memory computing area, improve the data processing efficiency in the in-memory computing mode, and further ensure the real-time ability of data synchronization.

[0023] By monitoring in-memory computing, transmission status, heartbeat information, etc., ensure the data synchronization efficiency in various scenarios such as poor network quality and data transmission congestion. Brief Description of the Drawings

[0024] The drawings described herein are used to provide a further understanding of the present application, form a part of the present application, and the schematic embodiments and descriptions thereof are used to explain the present application and do not constitute an improper limitation of the present application. In the drawings: Figure 1 Show a schematic implementation structure diagram of a data synchronization system according to an embodiment of the present application; Figure 2 Show a schematic implementation logic diagram of a data synchronization system according to an embodiment of the present application; Figure 3 Show a schematic implementation flow diagram of a data synchronization method according to an embodiment of the present application; Figure 4 Show a schematic implementation structure diagram of a data synchronization system according to another embodiment of the present application; Figure 5 Show a schematic implementation logic diagram of a data synchronization system according to another embodiment of the present application. Detailed Embodiments

[0025] To make the objectives, technical solutions, and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below in conjunction with specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts belong to the scope of protection of the present application.

[0026] The concept of this application lies in that Redis, as a key technical component, undertakes important responsibilities in the construction process of enterprise application systems. With the advancement of enterprise digital transformation and business continuity work, it is necessary to actively explore Redis cross-data center synchronization solutions to effectively handle data synchronization work for different data volumes and different application requirements. In the existing business scenario requirements, how to build real-time data synchronization capabilities, improve transmission quality and efficiency, strengthen process monitoring, and support flexible online mode adjustment has become a technical problem to be solved urgently. In the practice of existing Redis cross-data center synchronization solutions, there are obvious shortcomings in real-time capabilities, and there is insufficient flexibility in the service process, unable to flexibly support deployment modes, synchronization directions, traffic regulation, etc. In addition, the existing data transmission modes have problems such as data mixing, difficult verification, and insufficient control and scheduling capabilities, resulting in the inability to effectively guarantee the timeliness and quality of transmission.

[0027] In view of the above drawbacks, this application proposes a data synchronization system and a data synchronization method, constructs an associated proxy service plugin and a real-time transmission device, and establishes a connection between the associated proxy service plugin and the real-time transmission device through multiple connection channels to solve the above problems.

[0028] Figure 1 FIG. shows the implementation structure diagram of the data synchronization system proposed in an embodiment of this application; Figure 2 FIG. shows the implementation logic diagram of the data synchronization system proposed in an embodiment of this application; Figure 3 FIG. shows the implementation flow diagram of the data synchronization method proposed in an embodiment of this application. According to Figure 1 and Figure 2 The data synchronization system shown in Figure 3 describes the data synchronization method shown in

[0029] Referring to Figure 1 and Figure 2 shown, the data synchronization system includes: a real-time transmission device and an associated proxy service plugin constructed for each data center.

[0030] In Figure 1 and Figure 2 three data centers (Data Center A, Data Center B, Data Center C) are taken as examples. Any one of the three data centers can be used as the master data center, and the other two data centers are used as slave data centers. The master data center can synchronize the synchronization data to each slave data center through the data synchronization method proposed in this embodiment. For a more clear description of the data synchronization method in the following, in this embodiment, Data Center A is used as the master data center, and Data Center B and Data Center C are used as slave data centers.

[0031] Step S310: When an update action occurs in the application cluster of the primary data center, each associated proxy service plugin is started, and respectively establishes a data transfer connection channel, a policy transfer connection channel, and a heartbeat information connection channel with the real-time transmission device.

[0032] In the prior art, usually after the synchronous data is written to Redis in the primary data center, the synchronous data is then synchronized to other slave data centers. However, for the technical solution proposed in this embodiment, when an update action occurs in the application cluster of the primary data center, the data synchronization of the synchronous data from the primary data center to other slave data centers is immediately started. When the data synchronization from the primary data center to other slave data centers is started, the Redis in the primary data center normally writes the synchronous data.

[0033] When an update action occurs in the application cluster of the primary data center, each associated proxy service plugin (RD TransferAgent) establishes three connection channels. The three connection channels include: a data transfer connection channel (DataUpload Channel) for synchronous data transfer, a policy transfer connection channel (StrategyTransChannel) for policy transfer management, and a heartbeat information connection channel (HeartBeat Channel) for heartbeat information transmission. The channel identification information (Channel ID) can be composed of "associated proxy service plugin ID + channel category".

[0034] For example, in Figure 1 the shown situation, when the data of application cluster A is updated, the associated proxy service plugin A establishes a data transfer connection channel A, a policy transfer connection channel A, and a heartbeat information connection channel A with the real-time transmission device; the associated proxy service plugin B establishes a data transfer connection channel B, a policy transfer connection channel B, and a heartbeat information connection channel B with the real-time transmission device; the associated proxy service plugin C establishes a data transfer connection channel C, a policy transfer connection channel C, and a heartbeat information connection channel C with the real-time transmission device.

[0035] Step S320: The associated proxy service plugin corresponding to the primary data center reads the synchronous data from the application cluster of the primary data center according to the reading rule, slices and compresses the synchronous data into multiple sub-data blocks according to the slicing rule, and transmits the multiple sub-data blocks to the real-time transmission device through the data transfer connection channel according to the transmission rule.

[0036] The associated proxy service plug-in corresponding to the main data center takes effect for the read service role, and the associated proxy service plug-ins corresponding to each slave data center take effect for the write service role. The associated proxy service plug-in corresponding to the main data center takes effect based on the read service role and reads and synchronizes data from the application cluster of the main data center according to the read rules. The read rules are updated by the monitoring results of the memory calculation monitored by the real-time transmission device. The associated proxy service plug-in corresponding to the main data center slices and compresses the synchronized data into multiple sub-data blocks according to the slicing rules. The slicing rules are preset fixed rules. The associated proxy service plug-in corresponding to the main data center transmits the multiple sub-data blocks to the real-time transmission device through the data transmission connection channel according to the transmission rules. The transmission rules are updated by the monitoring results of the transmission status monitored by the real-time transmission device.

[0037] For example, in Figure 2 the shown case, the associated proxy service plug-in A takes effect for the read service role, and the associated proxy service plug-ins B and C take effect for the write service role. The associated proxy service plug-in A reads and synchronizes data from the application cluster of data center A according to the read rules, slices and compresses the synchronized data into multiple sub-data blocks according to the slicing rules, and transmits the multiple sub-data blocks to the real-time transmission device through the data transmission connection channel A according to the transmission rules.

[0038] Step S330, the real-time transmission device performs in-memory calculation on the multiple sub-data blocks to restore them to synchronized data, and transmits the synchronized data to the associated proxy service plug-ins corresponding to each slave data center through each data transmission connection channel, so that the associated proxy service plug-ins corresponding to each slave data center write the synchronized data into the Redis of the corresponding slave data center respectively.

[0039] The real-time transmission device performs in-memory calculation on the multiple sub-data blocks in its in-memory calculation area to restore the multiple sub-data blocks to synchronized data. The real-time transmission device transmits the synchronized data to the associated proxy service plug-ins corresponding to each slave data center through each data transmission connection channel respectively. The associated proxy service plug-ins corresponding to each slave data center take effect based on the write service role and write the synchronized data into the Redis of the slave data center.

[0040] For example, in Figure 2 the shown case, the real-time transmission device performs in-memory calculation on the multiple sub-data blocks in its in-memory calculation area to restore the multiple sub-data blocks to synchronized data. The real-time transmission device transmits the synchronized data to the associated proxy service plug-in B through the data transmission connection channel B, and the associated proxy service plug-in B writes the synchronized data into Redis-B of data center B. The real-time transmission device transmits the synchronized data to the associated proxy service plug-in C through the data transmission connection channel C, and the associated proxy service plug-in C writes the synchronized data into Redis-C of data center C.

[0041] Step S340: The real-time transmission device monitors the transmission status of the data transmission connection channel of the memory calculation and the associated proxy service plug-in corresponding to the main data center, and transmits the monitoring result to the associated proxy service plug-in corresponding to the main data center through the policy transmission connection channel.

[0042] The real-time transmission device monitors the memory calculation situation while performing memory calculation, and also monitors the transmission status of the data transmission connection channel of the associated proxy service plug-in corresponding to the main data center, so as to transmit the monitoring results of the memory calculation and the transmission status to the associated proxy service plug-in corresponding to the main data center through the policy transmission connection channel.

[0043] For example, in Figure 2 the shown situation, the real-time transmission device monitors the memory calculation and the transmission status of the data transmission connection channel A, and transmits the monitoring result to the associated proxy service plug-in A through the policy transmission connection channel A.

[0044] Step S350: The associated proxy service plug-in corresponding to the main data center updates the reading rule and the transmission rule according to the monitoring result.

[0045] The associated proxy service plug-in corresponding to the main data center updates the reading rule and the transmission rule according to the monitoring result.

[0046] For example, in Figure 2 the shown situation, the associated proxy service plug-in A updates the reading rule and the transmission rule according to the monitoring result.

[0047] Step S360: Each associated proxy service plug-in transmits heartbeat information to the real-time transmission device through the corresponding heartbeat information connection channel.

[0048] Each associated proxy service plug-in sends heartbeat information to the real-time transmission device through the corresponding heartbeat information connection channel.

[0049] For example, in Figure 2 the shown situation, the associated proxy service plug-in A transmits a heartbeat packet to the real-time transmission device through the heartbeat information connection channel A, the associated proxy service plug-in B transmits a heartbeat packet to the real-time transmission device through the heartbeat information connection channel B, and the associated proxy service plug-in C transmits a heartbeat packet to the real-time transmission device through the heartbeat information connection channel C.

[0050] Step S370: When the real-time transmission device determines that the corresponding associated proxy service plug-in is abnormal according to any heartbeat information, it generates a node handling policy and transmits the node handling policy to the corresponding associated proxy service plug-in through the policy transmission connection channel.

[0051] The real-time transmission device senses and detects the node health status of each associated proxy service plugin according to each heartbeat message, and timely generates and issues node handling strategies such as restarting for abnormal nodes. The node handling strategies are respectively transmitted to the corresponding associated proxy service plugins through each strategy transmission connection channel.

[0052] For example, in Figure 2 the shown case, the real-time transmission device separately senses and detects the heartbeat packets of heartbeat information connection channel A, heartbeat information connection channel B, and heartbeat information connection channel C. If it is determined that the associated proxy service plugin B is abnormal according to the heartbeat packet of heartbeat information connection channel B, a node handling strategy is timely generated, and the node handling strategy is transmitted to the associated proxy service plugin B through strategy transmission connection channel B, so that the associated proxy service plugin B executes the self-healing mechanism according to the node handling strategy.

[0053] From Figures 1 to 2 the shown system and Figure 3 the shown method, it can be seen that the data synchronization system and data synchronization method provided by this application have the following advantages: I. Regarding transmission efficiency and real-time performance: 1. Based on the capabilities of the associated proxy service plugins, build the real-time sensing ability of the business application cluster and Redis data updates, and achieve quasi-real-time synchronous writing of the master and slave data centers. When the application cluster in the master data center starts a data update action, each associated proxy service plugin senses in a timely manner and starts the data synchronization process in a timely manner, and synchronously writes the synchronized data to the Redis of each slave data center.

[0054] 2. Establish multiple connection channels, and respectively transmit synchronized data, management policies, and heartbeat information through different connection channels to isolate the mutual influence between different types of data and improve the data transmission efficiency.

[0055] 3. During the transmission of synchronized data, first perform fragmentation and compression processing, and then complete the transmission of the fragmented and compressed multiple sub-data blocks according to the transmission rules, improve the transmission efficiency of data synchronization, and ensure the timeliness of data synchronization.

[0056] 4. Build in-memory computing, and perform operations such as restoration on the synchronized data that has been fragmented and compressed during transmission in the in-memory computing area, and improve the data processing timeliness in the mode of in-memory computing to further ensure the real-time ability of data synchronization.

[0057] 5. By monitoring in-memory computing, transmission status, heartbeat information, etc., ensure the data synchronization efficiency in various scenarios such as poor network quality and data transmission congestion.

[0058] II. Regarding flexibility: 1. Build an associated proxy service plugin to decouple data transmission from the application cluster and Redis.

[0059] 2. The associated proxy service plugin supports hot-pluggable service switching, enabling the synchronous switching of the data synchronization transmission direction to take effect after the primary and secondary data center roles are switched.

[0060] Figure 4 The implementation structure schematic diagram of the data synchronization system proposed in another embodiment of the present application is shown. Figure 5 The implementation logic schematic diagram of the data synchronization system proposed in another embodiment of the present application is shown. According to Figure 4 and Figure 5 shown, the data synchronization method is further described.

[0061] Referring to Figure 4 and Figure 5 shown, the data synchronization system includes: each associated proxy service plugin and a real-time transmission device.

[0062] The associated proxy service plugin includes: a basic layer and an application layer.

[0063] The basic layer includes: a read service sub-plugin and a write service sub-plugin.

[0064] The application layer includes: a read rule sub-plugin, a traffic control sub-plugin, a mode management sub-plugin, and a node handling sub-plugin.

[0065] The real-time transmission device includes: an incoming phase message queue, a memory calculation unit, an outgoing phase message queue, and a management service unit.

[0066] The incoming phase message queue includes: a metadata queue and a business data queue.

[0067] The memory calculation unit includes: a sequential data restoration module, a metadata verification module, and a synchronized data output module; the sequential data restoration module includes: four memory data storage areas and a unified handling area.

[0068] The outgoing phase message queue includes: an outgoing data queue corresponding to each slave data center.

[0069] The management service unit includes: a data verification module, a capacity evaluation module, a bandwidth monitoring module, a mode management module, a node monitoring module, and a configuration center.

[0070] In Figure 4 and Figure 5In this example, three data centers (Data Center A, Data Center B, and Data Center C) are still used as examples. Any one of the three data centers can be used as the primary data center, and the other two data centers are used as secondary data centers. The primary data center can synchronize the synchronization data to each secondary data center through the data synchronization method proposed in this embodiment. For a more clear description of the data synchronization method in the following, in this embodiment, Data Center A is used as the primary data center, and Data Center B and Data Center C are used as secondary data centers.

[0071] Based on the Java Agent technology, various companion agent service plugins are constructed, including a basic capability framework with bytecode enhancement and various data transmission capability components.

[0072] In the basic capability framework with bytecode enhancement, by overriding the addTransformer method and retransformClasses method in the Instrumentation API, interception is performed during class loading and reloading to modify the bytecode of the input class or dynamically inject jar files into the classpath of the ClassLoader.

[0073] In various data transmission capability components, a two-layer design and a lazy loading mode are provided.

[0074] The basic layer includes a read service sub-plugin (Data Reader service) and a write service sub-plugin (Data Writer service) for controlling the direction of synchronization data transmission.

[0075] The application layer includes a read rule sub-plugin, a traffic control sub-plugin, a mode management sub-plugin, a node handling sub-plugin, etc.

[0076] Each companion agent service plugin takes effect on the sub-plugins of the basic layer when an update action occurs in the application cluster of the primary data center by overriding the premain method in Java Agent. Each companion agent service plugin takes effect on the sub-plugins of the application layer during the process of synchronizing data transmission by overriding the agentmain method in JavaAgent.

[0077] Each companion agent service plugin is started through premain, establishes a data transmission connection channel, a policy transmission connection channel, and a heartbeat information connection channel based on the WebSocket mechanism, and establishes a long connection between the real-time transmission device and each connection channel based on the long connection bus.

[0078] In the WebSocket header, an extension field is enabled to store the connection channel identification field (ChannelID). During the establishment of the connection channel, each associated proxy service plugin calls the connection channel creation method to create data transfer connection channels, policy transmission connection channels, and heartbeat information connection channels based on the extended and customized WebSocket protocol. The multi-channel isolation management mechanism is used to ensure that data does not affect each other and enhance the real-time performance of data transmission.

[0079] The real-time transmission device establishes long connections with each connection channel. According to different functional purposes, each unit / module of the real-time transmission device is connected to the corresponding connection channel.

[0080] For example: The node monitoring module senses each heartbeat signal from the heartbeat information connection channel, detects the node health status of each associated proxy service plugin, generates corresponding node handling strategies based on the abnormal node health status, and transmits the node handling strategies to the corresponding associated proxy service plugin through the policy transmission connection channel. For example, for abnormal nodes, node handling strategies such as restart are timely sent through the policy transmission connection channel.

[0081] For example: The capacity evaluation module obtains the data transfer volume of the data transfer connection channel of the associated proxy service plugin corresponding to the main data center, calculates the transfer capacity threshold based on the data transfer volume, and transmits the transfer capacity threshold to the associated proxy service plugin corresponding to the main data center through the policy transmission connection channel; the bandwidth monitoring module obtains the data transfer bandwidth of the data transfer connection channel of the associated proxy service plugin corresponding to the main data center, calculates the bandwidth occupancy ratio based on the data transfer bandwidth, and transmits the bandwidth occupancy ratio to the associated proxy service plugin corresponding to the main data center through the policy transmission connection channel; the associated proxy service plugin corresponding to the main data center adjusts the transfer flow in a timely manner (such as flow control, pause, etc.) based on monitoring results such as the transfer capacity threshold and the bandwidth occupancy ratio.

[0082] For example: The mode management module determines the cluster deployment mode (Cluster) or the single-machine deployment mode (Standalone) according to the modes of each data center, and transmits the cluster deployment mode or the single-machine deployment mode to the associated proxy service plugin corresponding to each data center through each policy transmission connection channel; the mode management sub-plugin of the application layer of the associated proxy service plugin corresponding to each data center switches the deployment mode accordingly.

[0083] According to the master / slave roles of each data center and in accordance with the transmission directions defined in premain, each companion agent service plug-in activates the sub-plug-ins at the basic layer. When it is the companion agent service plug-in corresponding to the master data center, the read service sub-plug-in at the basic layer is activated, that is, the data Reader service role becomes effective. When it is the companion agent service plug-in corresponding to the slave data center, the write service sub-plug-in at the basic layer is activated, that is, the data Writer service role becomes effective.

[0084] After the sub-plug-ins at the basic layer become effective, the synchronized data of the master data center is transmitted through its data transfer connection channel, and the synchronized data of each slave data center is transmitted through the corresponding data transfer connection channel.

[0085] If the roles of the master data center and the slave data center are swapped, the configuration center of the management service unit can issue the configuration information of the effective sub-plug-ins at the basic layer of each companion agent service plug-in through each policy transmission connection channel. Each companion agent service plug-in redefines the transmission direction in premain according to the configuration information of the effective sub-plug-ins at the basic layer, and after restarting, the corresponding sub-plug-ins at the basic layer become effective.

[0086] Start and activate each sub-plug-in in the application layer according to agentmain. Each companion agent service plug-in depends on the monitoring results and node handling policies issued by the management service unit through each policy transmission connection channel, and pulls the relevant plug-in configuration information from the configuration center of the management service unit.

[0087] For the companion agent service plug-in corresponding to the master data center, it activates the hot-pluggable read rule sub-plug-in, traffic control sub-plug-in, mode management sub-plug-in, and node handling sub-plug-in.

[0088] The read rule sub-plug-in adjusts the read rule according to the monitoring results of memory calculation issued by the real-time transmission device through policy transmission connection channel A. Specifically, the read rule sub-plug-in adjusts the read rule according to the format verification results issued by the data verification module of the management service unit, and the format verification results of the data verification module come from the format verification results of the metadata of the metadata verification module of the memory calculation unit.

[0089] The traffic control sub-plug-in adjusts the transmission traffic according to the monitoring results of the transmission status of data transfer connection channel A issued by the real-time transmission device through policy transmission connection channel A. Specifically, the traffic control sub-plug-in adjusts the transmission traffic according to the transmission capacity threshold issued by the capacity evaluation module of the management service unit and the bandwidth occupancy ratio issued by the bandwidth monitoring module.

[0090] The mode management sub-plugin switches the deployment mode. Through the mode management sub-plugin, differential management operations for the Redis deployment mode are provided. For the companion proxy service plugin corresponding to the effective read sub-plugin in the primary data center and the companion proxy service plugin corresponding to the effective write sub-plugin in the secondary data center, the ability to switch between the cluster deployment mode and the single-machine deployment mode is provided.

[0091] The node handling sub-plugin executes the self-healing mechanism for the companion proxy service plugin according to the node handling policy sent by the real-time transmission device through the corresponding policy transmission connection channel. Specifically, the node handling sub-plugin schedules the self-healing restart mechanism according to the node monitoring module of the management service unit's node handling policy for heartbeat loss, and resumes data transmission after restarting the companion proxy service plugin.

[0092] The companion proxy service plugin corresponding to the primary data center slices the synchronized data into multiple business data according to the slicing rule, determines the metadata corresponding to the multiple business data, and respectively identifies the ID information for the metadata corresponding to the multiple business data.

[0093] The metadata stores some attribute information about the business data, including the business data category ID, the business data slicing order, the business data structure, the size of the business data volume, etc.

[0094] Compress the multiple business data into business data blocks respectively, and add the corresponding metadata ID information to the multiple business data blocks respectively.

[0095] Compress each business data respectively to form multiple business data blocks. Add metadata ID information to each business data block.

[0096] Transmit the multiple business data blocks and the corresponding metadata to the real-time transmission device through the data transmission connection channel according to the transmission rule of the companion proxy service plugin corresponding to the primary data center. Among them, the multiple business data blocks are transmitted to the business data queue, and the corresponding metadata is transmitted to the metadata queue.

[0097] The business data blocks and the corresponding metadata enter the data transmission connection channel of the companion proxy service plugin corresponding to the primary data center and are transmitted to the incoming phase message queue of the real-time transmission device according to the transmission rule.

[0098] The incoming phase message queue supports multiple message queues such as Rocket MQ, Kafka, and Rabbit MQ. Extracting the common characteristics of queue transmission, the incoming phase message queue constructs the metadata queue and the business data queue on the basis of the underlying message queue. The metadata queue and the business data queue are transmitted in parallel. Separating the metadata queue and the business data queue can increase the data transmission volume and can also verify the data transmission quality through the metadata verification mode.

[0099] During the incoming phase, the message queue receives the processed synchronization data transmitted through the data transfer connection channel of the associated proxy service plugin in the primary data center. Classified by business data blocks and metadata, if it is metadata, it enters the metadata queue for transmission; if it is a business data block, it enters the business data queue for transmission.

[0100] In addition, the business data queue supports horizontal capacity expansion, can timely sense the congestion of the business data queue, and achieve flexible scaling.

[0101] Since the sharding order of business data blocks is stored in the corresponding metadata, the transmission of business data blocks in the business data queue can be unrestricted by order. The parallel transmission of each business data block can enhance the data transmission efficiency.

[0102] After the incoming phase message queue receives each business data block and the corresponding metadata, it transmits them to the in-memory computing unit connected to the incoming phase message queue.

[0103] The sequential data restoration module receives multiple business data blocks and the corresponding metadata, and restores the corresponding multiple business data blocks into sequential data according to the metadata.

[0104] The sequential data restoration module plans four memory data storage areas and a unified processing area. According to the hash result of the metadata ID information, the metadata and the corresponding business data blocks are respectively divided into different memory data storage areas to achieve the co-location storage of the metadata and the corresponding business data blocks.

[0105] As the data increases and the data processing capacity is calculated, the sequential data restoration module can also timely receive the expansion strategy of the management service unit, expand the memory data storage area in multiples of four in a timely manner, and synchronously activate the new hash function. The newly received metadata and the corresponding business data blocks are respectively co-located and divided into the new memory data storage areas. When the storage capacity of the original memory data storage area drops below 50%, the hash function is automatically switched so that the newly received metadata and the corresponding business data blocks are respectively co-located and divided into the original memory data storage areas.

[0106] Calculation is directly started in each memory data storage area. During the calculation, multiple business data blocks are restored into sequential data according to the method group loop mode.

[0107] The method group includes: planning the restoration calculation task, broadcasting the restoration calculation task to each memory data storage area, respectively executing the restoration calculation task in each memory data storage area, sending the results of executing the restoration calculation task in each memory data storage area to the unified processing area, and forming sequential data in the unified processing area.

[0108] Restore the computing tasks through compute(), broadcast the tasks through the broadcast method, and distribute all the computing logics of the restored computing tasks to the corresponding memory data storage areas. Execute the specific computing logic in the memory data storage area through the call() method, and send the results to the unified processing area through the inform() method to form sequential data in the unified processing area. The data used for computing is directly obtained from the memory data storage area, and a memory closed-loop is achieved through task planning - broadcast transmission - task execution - result upload.

[0109] The metadata verification module sequentially executes the format verification of the metadata corresponding to the sequential data, and transmits the format verification results to the associated proxy service plugin corresponding to the main data center through the policy transmission connection channel.

[0110] The metadata verification module calls the metadata verification logic to sequentially execute the format verification of the metadata corresponding to the sequential data. The metadata verification module can transmit the format verification results to the data verification module of the management service unit, and the data verification module then transmits the format verification results to the reading rule sub-plugin of the application layer of the associated proxy service plugin corresponding to the main data center through the policy transmission connection channel of the associated proxy service plugin corresponding to the main data center. The reading rule sub-plugin of the application layer of the associated proxy service plugin corresponding to the main data center updates the reading rules according to the format verification results.

[0111] The synchronous data output module processes the sequential data into synchronous data.

[0112] The synchronous data output module reorganizes the sequential data into synchronous data and transmits it to the output phase message queue connected to the memory computing unit. Implementing a series of computations such as metadata verification and synchronous data reorganization in the form of memory computing can improve the data processing efficiency and further ensure the real-time nature of data synchronization.

[0113] The output phase message queue supports multiple message queues such as Rocket MQ, Kafka, and Rabbit MQ. By abstracting the common characteristics of queue transmission, the output phase message queue constructs multiple output data queues corresponding to each slave data center on the basis of the underlying message queue.

[0114] The synchronous data output module transmits the synchronous data to the output data queues corresponding to each slave data center respectively, and each output data queue transmits the synchronous data to the associated proxy service plugin corresponding to each slave data center through each data transmission connection channel.

[0115] The parallelism of each output data queue corresponding to each slave data center can achieve parallel processing of synchronous data writing, reduce the latency of cross-data center transmission, and further improve the data transmission efficiency.

[0116] The associated proxy service plug-ins corresponding to each slave data center write the synchronized data into the Redis of each slave data center.

[0117] The data synchronization system and data synchronization method proposed in this embodiment: 1. The real-time transmission device continuously improves the data transmission efficiency and quality based on long connections and message queues. The long connection is used to transmit management policies to ensure the quality and efficiency of the transmission of management policies. The incoming-phase message queues include a metadata queue and a service data queue. The metadata queue is used to transmit the attribute information of the data, and the data queue is used to transmit the sharded and compressed service data blocks; the outgoing-phase message queues include outgoing data queues corresponding to each slave data center; the transmission efficiency is ensured through parallel transmission.

[0118] 2. In the face of the classified transmission forms of metadata and service data blocks, the sequential data restoration, metadata verification, and synchronized data recombination transmission are realized through the in-memory computing mechanism.

[0119] 3. Each module of the management service unit monitors the transmission status, network status, transmission accuracy, deployment mode, etc. in real time, and synchronizes management policies such as capacity assessment, bandwidth monitoring, deployment mode, node monitoring, data quality, and transmission direction to the policy transmission connection channel through long connections, so as to be transmitted to the master and slave data centers in real time, ensuring that the management policies have real-time and online adjustment and effectiveness capabilities, and improving the flexibility level.

[0120] It should be noted that the above data synchronization system can implement the data synchronization method one by one, and details are not described herein again.

[0121] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A data synchronization system, characterized in that, The data synchronization system includes: a real-time transmission device and an associated proxy service plugin constructed for each data center; Each associated proxy service plugin is started when an update action occurs in the application cluster of the primary data center, and respectively establishes a data transfer connection channel, a policy transmission connection channel, and a heartbeat information connection channel with the real-time transmission device; The associated proxy service plugin corresponding to the primary data center reads synchronization data from the application cluster of the primary data center according to the reading rule, slices and compresses the synchronization data into multiple sub-data blocks according to the slicing rule, and transmits the multiple sub-data blocks to the real-time transmission device through the data transfer connection channel according to the transmission rule; The real-time transmission device performs in-memory calculation on the multiple sub-data blocks to restore them to synchronization data, and transmits the synchronization data to the associated proxy service plugins corresponding to each secondary data center through each data transfer connection channel, so that the associated proxy service plugins corresponding to each secondary data center respectively write the synchronization data into the Redis of the corresponding secondary data center; The real-time transmission device monitors the in-memory calculation and the transmission status of the data transfer connection channel of the associated proxy service plugin corresponding to the primary data center, and transmits the monitoring result to the associated proxy service plugin corresponding to the primary data center through the policy transmission connection channel; The associated proxy service plugin corresponding to the primary data center updates the reading rule and the transmission rule according to the monitoring result; Each associated proxy service plugin transmits heartbeat information to the real-time transmission device through the corresponding heartbeat information connection channel; When the real-time transmission device determines that the associated proxy service plugin corresponding to any heartbeat information is abnormal, it generates a node handling strategy, and transmits the node handling strategy to the corresponding associated proxy service plugin through the policy transmission connection channel.

2. The data synchronization system according to claim 1, wherein The data transfer connection channel, the policy transmission connection channel, and the heartbeat information connection channel are established based on the WebSocket mechanism, and the long connection between the real-time transmission device and each connection channel is established based on the long connection bus.

3. The data synchronization system according to claim 1, wherein Each associated proxy service plugin includes a basic layer and an application layer; The basic layer includes: a reading service sub-plugin and a writing service sub-plugin; When an update action occurs in the application cluster of the primary data center, the basic layer of the associated proxy service plugin corresponding to the primary data center activates the reading service sub-plugin, and the basic layer of the associated proxy service plugins corresponding to each secondary data center activates the writing service sub-plugin; The application layer includes: a reading rule sub-plugin, a traffic control sub-plugin, a mode management sub-plugin, and a node handling sub-plugin; The reading rule sub-plugin adjusts the reading rule according to the monitoring result of the in-memory calculation, the traffic control sub-plugin adjusts the transmission traffic according to the monitoring result of the transmission status, the mode management sub-plugin switches the deployment mode, and the node handling sub-plugin executes the self-healing mechanism on the associated proxy service plugin according to the node handling strategy.

4. The data synchronization system according to claim 3, wherein By rewriting the premain method in Java Agent, the sub-plugins of the basic layer are activated when an update action occurs in the application cluster of the primary data center; by rewriting the agentmain method in Java Agent, the sub-plugins of the application layer are activated during the synchronization data transmission process.

5. The data synchronization system according to claim 1, wherein The real-time transmission device includes: an incoming stage message queue; the incoming stage message queue includes: a metadata queue and a service data queue; The associated proxy service plugin corresponding to the main data center slices the synchronization data into multiple service data according to the slicing rule, determines the metadata corresponding to the multiple service data, and respectively identifies the ID information for the metadata corresponding to the multiple service data; Compresses the multiple service data into service data blocks respectively, and adds the corresponding metadata ID information to the multiple service data blocks respectively; Transmits the multiple service data blocks and the corresponding metadata to the real-time transmission device through the data transmission connection channel according to the transmission rule of the associated proxy service plugin corresponding to the main data center. Among them, the multiple service data blocks are transmitted to the service data queue, and the corresponding metadata is transmitted to the metadata queue.

6. The data synchronization system according to claim 5, wherein The real-time transmission device includes: a memory computing unit, and the memory computing unit is connected to the incoming stage message queue; the memory computing unit includes: a sequential data restoration module, a metadata verification module, and a synchronization data output module; The sequential data restoration module receives the multiple service data blocks and the corresponding metadata, and restores the corresponding multiple service data blocks into sequential data according to the metadata; The metadata verification module sequentially performs format verification on the metadata corresponding to the sequential data, and transmits the format verification result to the associated proxy service plugin corresponding to the main data center through the policy transmission connection channel; The synchronization data output module processes the sequential data into synchronization data.

7. The data synchronization system according to claim 6, wherein The sequential data restoration module includes: four memory data storage areas and a unified processing area; Divides the metadata and the corresponding service data blocks into different memory data storage areas according to the hash result of the metadata ID information; Restores the multiple service data blocks into sequential data according to the method group loop mode; The method group includes: planning a restoration calculation task, broadcasting the restoration calculation task to each memory data storage area, respectively executing the restoration calculation task in each memory data storage area, sending the results of the restoration calculation tasks executed in each memory data storage area to the unified processing area, and forming sequential data in the unified processing area.

8. The data synchronization system according to claim 6, wherein The real-time transmission device includes: an outgoing stage message queue, and the outgoing stage message queue is connected to the memory computing unit; the outgoing stage message queue includes: outgoing data queues corresponding to each slave data center; The synchronization data output module transmits the synchronization data to the outgoing data queues corresponding to each slave data center respectively; Each outgoing data queue transmits the synchronization data to the associated proxy service plugin corresponding to each slave data center through each data transmission connection channel.

9. The data synchronization system according to claim 6, wherein The real-time transmission device includes: a management service unit; the management service unit includes: a data verification module, a capacity evaluation module, a bandwidth monitoring module, a mode management module, a node monitoring module, and a configuration center; During the process of transmitting the synchronization data, the metadata verification module sends the format verification result to the data verification module, and the data verification module transmits the format verification result to the associated proxy service plugin corresponding to the main data center through the policy transmission connection channel; The capacity evaluation module obtains the data transfer volume of the data transfer connection channel of the associated proxy service plug-in corresponding to the primary data center, calculates the transfer capacity threshold according to the data transfer volume, and transmits the transfer capacity threshold to the associated proxy service plug-in corresponding to the primary data center through the policy transfer connection channel; The bandwidth monitoring module obtains the data transfer bandwidth of the data transfer connection channel of the associated proxy service plug-in corresponding to the primary data center, calculates the bandwidth occupancy ratio according to the data transfer bandwidth, and transmits the bandwidth occupancy ratio to the associated proxy service plug-in corresponding to the primary data center through the policy transfer connection channel; The mode management module determines the cluster deployment mode or the single-machine deployment mode according to the modes of each data center; The node monitoring module determines the node health status of the corresponding associated proxy service plug-in according to each heartbeat signal, generates the corresponding node handling strategy according to the abnormal node health status, and transmits the node handling strategy to the corresponding associated proxy service plug-in through the policy transfer connection channel; The configuration center provides plug-in configuration information to each associated proxy service plug-in through each policy transfer connection channel.

10. A data synchronization method, characterized in that, The data synchronization method is applied to a data synchronization system, and the data synchronization system includes: a real-time transmission device and an associated proxy service plug-in constructed for each data center; the data synchronization method includes: Each associated proxy service plug-in is started when an update action occurs in the application cluster of the primary data center, and respectively establishes a data transfer connection channel, a policy transfer connection channel, and a heartbeat information connection channel with the real-time transmission device; The associated proxy service plug-in corresponding to the primary data center reads the synchronization data from the application cluster of the primary data center according to the reading rule, slices and compresses the synchronization data into multiple sub-data blocks according to the slicing rule, and transmits the multiple sub-data blocks to the real-time transmission device through the data transfer connection channel according to the transmission rule; The real-time transmission device performs in-memory calculation on the multiple sub-data blocks to restore them to synchronization data, and transmits the synchronization data to the associated proxy service plug-ins corresponding to each secondary data center through each data transfer connection channel, so that the associated proxy service plug-ins corresponding to each secondary data center respectively write the synchronization data into the Redis of the corresponding secondary data center; The real-time transmission device monitors the in-memory calculation and the transmission status of the data transfer connection channel of the associated proxy service plug-in corresponding to the primary data center, and transmits the monitoring result to the associated proxy service plug-in corresponding to the primary data center through the policy transfer connection channel; The associated proxy service plug-in corresponding to the primary data center updates the reading rule and the transmission rule according to the monitoring result; Each associated proxy service plug-in transmits heartbeat information to the real-time transmission device through the corresponding heartbeat information connection channel; When the real-time transmission device determines that the corresponding associated proxy service plug-in is abnormal according to any heartbeat information, it generates a node handling strategy and transmits the node handling strategy to the corresponding associated proxy service plug-in through the policy transfer connection channel.

Images