Distributed real-time data storage, access and synchronization method and system

Abstract

The invention discloses a distributed real-time data storage, access and synchronization method and system, and relates to the technical field of distributed systems.The distributed real-time data storage, access and synchronization method comprises the steps that a Thrift micro-service architecture is adopted, a service interface and a communication protocol are defined, and a distributed system framework is constructed; based on a distributed system framework, customizing a data type and a data model, and constructing a real-time data storage structure; based on a real-time data storage structure, a node partitioning strategy is formulated, and a network topology adaptive to the Thrift service is established; for a data model and a network topology, developing a high-speed read-write interface based on Thrift; a version comparison mechanism is integrated in a high-speed read-write interface, and corresponding incremental and full data synchronization strategies are designed. According to the invention, a Thrift micro-service architecture is adopted to define a service interface and a communication protocol, and an efficient cross-language distributed system framework is constructed.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed systems, and in particular to a distributed real-time data storage, access and synchronization method and system. Background Art

[0002] With the increase in the number of users, the traditional single-service application architecture cannot meet the system load requirements of a single service or central service, and when the system fails, all services will be paralyzed, causing serious losses. Therefore, the main ways to solve the decentralization problem include distributed storage, decentralized architecture, edge computing and blockchain technology. Among them, distributed storage is the core technology of decentralization. It ensures the high availability and flexibility of the system through cross-domain distribution and multi-node storage, while enhancing data security, improving server load capacity and data access speed.

[0003] Power monitoring software, such as photovoltaic power stations, hydropower stations or power grid monitoring systems, usually needs to monitor and access data from multiple nodes. These systems range from small to large, including monitoring of unit data to centralized control systems, and may need to manage tens of millions of data points. Monitoring software has high security requirements for data. On the one hand, system redundant backup is required to ensure that a single node failure does not affect the operation of the overall system; on the other hand, the user application needs to respond to changes in collected data in a timely manner and make corresponding operations within an acceptable time frame to ensure the security and reliability of the service. Summary of the invention

[0004] In view of the problems existing in the existing distributed storage and access systems in terms of performance, reliability, scalability and data consistency, the present invention is proposed.

[0005] Therefore, the problem to be solved by the present invention is how to implement a method and system for distributed real-time data storage, access and synchronization.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In the first aspect, an embodiment of the present invention provides a distributed real-time data storage, access, and synchronization method, which includes adopting a thrift microservice architecture, defining a service interface and a communication protocol, and building a distributed system framework; based on the distributed system framework, customizing data types and data models, and building a real-time data storage structure; based on the real-time data storage structure, formulating a node partitioning strategy, and establishing a network topology adapted to the thrift service; developing a thrift-based high-speed read and write interface for the data model and network topology; integrating a version comparison mechanism in the high-speed read and write interface, and designing corresponding incremental and full data synchronization strategies to complete distributed real-time data storage, access, and synchronization.

[0008] As a preferred solution of the distributed real-time data storage, access and synchronization method described in the present invention, the real-time data storage structure is constructed, including the following steps: creating a class_def table in the SQLite local configuration library of each data node; creating a field_def table in the same SQLit local configuration library; designing an SQLite configuration synchronization mechanism, synchronizing central configuration information to the SQLite local storage of each data node through ZooKeeper; designing a dynamic extension interface of data types and data models based on the class_def table and the field_def table, supporting the addition or modification of data types and fields at runtime, and pushing the changes to all nodes through ZooKeeper; formulating real-time data table naming rules; performing data configuration based on the real-time data table naming rules and node names, and using ZooKeeper to coordinate each node; developing an SQLite configuration reading function, extracting data structure definitions from local class_def tables and field_def tables; establishing a mapping relationship between data models and Thrift service interfaces; constructing a real-time data storage structure on each data node according to the data structure definition and the mapping relationship, and coordinating data distribution of each node through ZooKeeper.

[0009] As a preferred solution of the distributed real-time data storage, access and synchronization method described in the present invention, the node partitioning strategy is formulated, which includes the following steps: based on the naming rules of the real-time data table, a data node partitioning strategy is designed to determine the distribution mode of data on different data nodes; a partition information node is created in ZooKeeper, and the partition information node is used to store the partition strategy and node allocation information; a partition management service is developed to provide a Thrift interface for querying, updating and synchronizing partition information; a dynamic partition adjustment mechanism is designed to adjust the data distribution according to the load situation and notify each data node through ZooKeeper; based on the Thrift service interface, a data routing layer is designed to forward the client request to the corresponding data node; a node health check mechanism is developed to monitor the data node status through ZooKeeper and trigger corresponding operations when the data node status changes; based on the mapping relationship between the data model and the Thrift service interface, a client request allocation mechanism is designed to allocate the request to the corresponding data node; a cross-data node query and aggregation mechanism is designed to allow access to data on multiple data nodes through a single Thrift service interface; a network topology structure adapted to the Thrift service is constructed; a metadata management service is developed to manage the data model, data node partition information and network topology structure.

[0010] As a preferred solution of the distributed real-time data storage, access and synchronization method of the present invention, wherein: adjusting data distribution according to load conditions comprises the following steps: constructing a load monitoring module to collect a performance indicator set of each data node at a preset time interval T1; mapping the collected performance indicator set I to a normalized load score S of the data node, where 0≤S≤1; setting a load threshold interval [L min ,L max ], when the normalized load score S of the data node satisfies S>L max Or S<L min When the condition lasts longer than the preset time T2, the partition adjustment process is triggered; a data partition migration decision module is designed based on the normalized load score set S = {S1,...,S n} and the current partition distribution state P to generate a migration plan M = {M1,...,M k}; Create a migration task node set Z in ZooKeeper = {Z1,...,Z k}; Develop a data migration execution module E(Z) to monitor the migration task node set Z in ZooKeeper and perform data migration operations according to the preset concurrency C; design a partition information update mechanism to i After completion, update the partition mapping relationship in ZooKeeper; when the partition mapping relationship changes, push the updated partition information to all relevant nodes through ZooKeeper within the preset time window T3; develop a rollback mechanism B(M,R0,E), where M is the currently executed migration plan, R0 is the initial partition mapping relationship before adjustment, and E is a predefined set of abnormal situations. When an abnormal event in E is detected during the partition adjustment process, perform the following operations: terminate all ongoing data migration operations; perform reverse data transmission for the completed migration tasks and restore the data to the source node; reset the partition mapping relationship to R0 in ZooKeeper; broadcast the rollback completion notification to all relevant nodes to enable them to update the locally cached partition information.

[0011] As a preferred solution of the distributed real-time data storage, access and synchronization method of the present invention, the calculation formula of the normalized load score S is as follows:

[0012]

[0013] Among them, α, β, γ, and δ are weight coefficients, and α+β+γ+δ=1, CPU uage is the CPU usage, MEM uage is the memory usage, IO rate is the disk I / O rate, IO max is the maximum theoretical value of disk I / O, NETrate is the network traffic rate, NET max It is the maximum theoretical value of network traffic.

[0014] As a preferred solution of the distributed real-time data storage, access and synchronization method described in the present invention, the node health check mechanism includes: creating a corresponding temporary node for each data node in ZooKeeper; designing a health report data structure, the health report data structure includes basic information and health status indicators of the data node; designing a data node health report module, writing the latest health report to the corresponding ZooKeeper temporary node at a preset time interval T4; developing a central monitoring service, monitoring the temporary nodes of all data nodes through the Watch mechanism of ZooKeeper; setting a health status determination rule set; when a first situation is detected, marking the data node as an abnormal state, the first situation includes that the ZooKeeper temporary node is deleted, and the indicators in the health report H violate any rule in the health status determination rule set; designing a state change processing module, when the data node state changes from normal to abnormal, executing a first operation; when the abnormal node returns to normal state, executing a second operation.

[0015] As a preferred solution of the distributed real-time data storage, access and synchronization method described in the present invention, the first operation includes removing the abnormal node from the load balancer, triggering the data redistribution process, reallocating the data partitions on the abnormal node to the healthy node, and sending an alarm notification to the system administrator; the second operation includes re-adding the node to the load balancer, triggering the data synchronization process, updating the system status, and recording the node recovery event.

[0016] In the second aspect, an embodiment of the present invention provides a distributed real-time data storage, access, and synchronization system, which includes a framework construction module for adopting the thrift microservice architecture, defining service interfaces and communication protocols, and building a distributed system framework; a storage structure construction module for building a real-time data storage structure based on the distributed system framework, customizing data types and data models; a partitioning strategy module for formulating a node partitioning strategy based on the real-time data storage structure, and establishing a network topology adapted to the thrift service; a read-write interface module for developing a high-speed read-write interface based on thrift for data models and network topologies; a comparison and synchronization module for integrating a version comparison mechanism in a high-speed read-write interface, and designing corresponding incremental and full data synchronization strategies to complete distributed real-time data storage, access, and synchronization.

[0017] In a third aspect, an embodiment of the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program instructions are executed by the processor, the steps of the distributed real-time data storage, access, and synchronization method as described in the first aspect of the present invention are implemented.

[0018] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program instructions are executed by a processor, the steps of the distributed real-time data storage, access, and synchronization method as described in the first aspect of the present invention are implemented.

[0019] The beneficial effects of the present invention are as follows: the present invention adopts the Thrift microservice architecture to define the service interface and communication protocol, and constructs an efficient cross-language distributed system framework. Distributed configuration management is performed through ZooKeeper, and SQLite is introduced as the local configuration storage to achieve configuration consistency and dynamic management capabilities. The system designs an overall framework including data nodes, metadata management and load balancer, and supports dynamic expansion of custom data types and data models. Based on the Thrift service interface and data model mapping relationship, an efficient real-time data storage structure is constructed. The rationality of data distribution and load balancing are ensured through partitioning strategies and dynamic adjustment mechanisms. The system provides a high-speed read-write interface and data routing layer based on Thrift to ensure the efficiency and consistency of data reading and writing. Through the version comparison mechanism and incremental and full synchronization strategies, data consistency and real-time synchronization in a distributed environment are ensured. The present invention greatly improves the performance, reliability, scalability and management flexibility of the system, and meets the needs of distributed real-time data storage, access and synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 The framework flow chart of the distributed real-time data storage, access, and synchronization method.

[0022] Figure 2 It is a schematic diagram of incremental synchronization of distributed real-time data storage, access and synchronization methods.

[0023] Figure 3 It is a full synchronization illustration of distributed real-time data storage, access, and synchronization methods.

[0024] Figure 4A computer device diagram for distributed real-time data storage, access, and synchronization methods. DETAILED DESCRIPTION

[0025] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0026] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0027] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0028] Example 1

[0029] Reference Figure 1 to Figure 4 , which is the first embodiment of the present invention, and provides a distributed real-time data storage, access, and synchronization method, including:

[0030] S1: Use Thrift microservice architecture to define service interfaces and communication protocols and build a distributed system framework.

[0031] Specifically, in order to achieve efficient cross-language service calls and flexible system expansion, the Thrift microservice architecture is used to define the Thrift service interface and communication protocol; to ensure configuration consistency and dynamic management capabilities, ZooKeeper is selected as the distributed configuration management system, and SQLite is introduced as the local configuration storage; in order to build a scalable and highly available distributed storage system, based on the Thrift service interface, ZooKeeper configuration and SQLite local storage, an overall distributed system framework including data nodes, metadata management and load balancer is designed.

[0032] S2: Based on the distributed system framework, custom data types and data models, build a real-time data storage structure.

[0033] Specifically, the following steps are included:

[0034] S2.1: Create a class_def table in the SQLite local configuration library of each data node to store data model types.

[0035] It should be noted that the class_def table contains integer id, model name, and version number fields.

[0036] S2.2: Create a field_def table in the same SQLit local configuration library to define data model fields.

[0037] It should be noted that the field_def table contains the data model type, data type, and column number information in the real-time data table to which the field belongs.

[0038] S2.3: Design a SQLite configuration synchronization mechanism to synchronize the central configuration information to the SQLite local storage of each data node through ZooKeeper.

[0039] Specifically, a ZooKeeper-based configuration change listener and incremental synchronization strategy are designed: when the central configuration changes, the system can automatically detect and synchronize only the changed parts, thereby reducing network transmission overhead and improving synchronization efficiency; at the same time, a version control mechanism is introduced to ensure that the configuration versions of all nodes are consistent, and support rapid rollback to the previous stable version when problems occur.

[0040] S2.4: Based on the class_def table and field_def table, design a dynamic extension interface for data types and data models to support adding or modifying data types and fields at runtime, and push the changes to all nodes through ZooKeeper.

[0041] S2.5: Establish a naming rule for real-time data tables, using the format of "node name + model type name" to uniquely identify data tables of different nodes and models, and register these data table names in ZooKeeper.

[0042] S2.6: Configure data based on the real-time data table naming rules and node names, and use ZooKeeper to coordinate the nodes so that each node contains all or part of the data type.

[0043] S2.7: Develop SQLite configuration reading function to extract data structure definitions from local class_def table and field_def table to improve configuration access efficiency.

[0044] S2.8: Establish a mapping relationship between the data model and the Thrift service interface to ensure the consistency of the data model and the service interface.

[0045] S2.9: Based on the data structure definition and mapping relationship, a real-time data storage structure is built on each data node, and the data distribution of each node is coordinated through ZooKeeper.

[0046] S3: Based on the real-time data storage structure, formulate a node partitioning strategy and establish a network topology that adapts to the Thrift service.

[0047] Specifically, the following steps are included:

[0048] S3.1: Based on the real-time data table naming rules in S2.5, design the data node partitioning strategy and determine how the data is distributed on different data nodes.

[0049] S3.2: Create a partition information node in ZooKeeper to store partition strategies and node allocation information.

[0050] S3.3: Develop partition management services and provide Thrift interfaces for querying, updating, and synchronizing partition information.

[0051] S3.4: Design a dynamic partition adjustment mechanism to adjust data distribution according to load conditions and notify each data node through ZooKeeper.

[0052] Preferably, a load monitoring module is constructed to collect a performance indicator set I = {I1, ..., I4} of each data node at a preset time interval T1, where I1 is the CPU usage rate, I2 is the memory occupancy rate, I3 is the disk I / O, and I4 is the network traffic rate; the collected performance indicator set I is mapped to a normalized load score S of the data node, where 0≤S≤1; and a load threshold interval [L min ,L max ], when the normalized load score S of the data node satisfies S>L max Or S<L min When the condition lasts longer than the preset time T2, the partition adjustment process is triggered; a data partition migration decision module is designed based on the normalized load score set S = {S1,...,S n} and the current partition distribution state P to generate a migration plan M = {M1,...,M k}, where P includes the size, access frequency and node information of each partition, M i A migration task includes a source node, a target node, and a partition to be migrated. In ZooKeeper, create a migration task node set Z = {Z1,...,Z k}, where Z i Corresponding migration task M i , including M i Develop a data migration execution module E(Z) to monitor the migration task node set Z in ZooKeeper and perform data migration operations according to the preset concurrency C; design a partition information update mechanism to perform data migration operations in the migration task M iAfter completion, update the partition mapping relationship in ZooKeeper to reflect the latest data distribution status; when the partition mapping relationship changes, push the updated partition information to all relevant nodes through ZooKeeper within the preset time window T3; develop a rollback mechanism B(M,R0,E), where M is the currently executed migration plan, R0 is the initial partition mapping relationship before adjustment, and E is a predefined set of abnormal situations. When an abnormal event in E is detected during the partition adjustment process, perform the following operations: terminate all ongoing data migration operations; perform reverse data transmission for the completed migration tasks and restore the data to the source node; reset the partition mapping relationship to R0 in ZooKeeper; broadcast the rollback completion notification to all relevant nodes to enable them to update the locally cached partition information.

[0053] Preferably, the time interval T1 is set to 30 seconds to balance the monitoring accuracy and system overhead; the load threshold interval [L min ,L max ] is set to [0.2,0.8], the preset time T2 is set to 5 minutes to avoid unnecessary adjustments triggered by short-term fluctuations; the concurrency C is set to 3 to control the number of data migration tasks performed simultaneously; the time window T3 is set to 5 seconds to ensure timely synchronization of partition information.

[0054] Furthermore, the abnormal situation set E includes but is not limited to: sudden node offline, data inconsistency, migration timeout and other events that may cause system instability.

[0055] Preferably, the data partition migration decision module preferentially selects data partitions with lower access frequency for migration, and the amount of data migrated at a single time does not exceed 10% of the total data amount of the source node to reduce the impact of migration on system performance.

[0056] It should be noted that the calculation formula of the normalized load score S is as follows:

[0057]

[0058] Among them, α, β, γ, and δ are weight coefficients, and α+β+γ+δ=1, CPU uage is the CPU usage, MEM uage is the memory usage, IO rate is the disk I / O rate, IO max is the maximum theoretical value of disk I / O, NET rate is the network traffic rate, NET max It is the maximum theoretical value of network traffic.

[0059] S3.5: Based on the Thrift service interface defined in S1, design the data routing layer to forward client requests to the corresponding data nodes.

[0060] S3.6: Develop a node health check mechanism to monitor the data node status through ZooKeeper and trigger corresponding actions when the data node status changes.

[0061] Specifically, create a corresponding temporary node for each data node in ZooKeeper; design a health report data structure, including basic information of the data node (such as IP address, port number) and health status indicators (such as heartbeat timestamp, CPU load, memory usage, disk space); design a data node health report module to write the latest health report to the corresponding ZooKeeper temporary node at a preset time interval T4; develop a central monitoring service to monitor the temporary nodes of all data nodes through ZooKeeper's Watch mechanism; set a health status determination rule set, including but not limited to heartbeat timeout threshold, CPU load upper limit, memory usage Upper limit and lower limit of disk space; when the following situations are detected, the data node is marked as abnormal: the ZooKeeper temporary node is deleted, and the indicators in the health report H violate any rule in the health status determination rule set; design a state change processing module, when the data node state changes from normal to abnormal, perform the first operation: remove the abnormal node from the load balancer, trigger the data redistribution process, reallocate the data partitions on the abnormal node to the healthy node, and send an alarm notification to the system administrator; when the abnormal node returns to normal, perform the second operation: re-add the node to the load balancer, trigger the data synchronization process, update the system status, and record the node recovery event.

[0062] It should be noted that the time interval T4 is set to 10 seconds and the heartbeat timeout threshold Th is set to 30 seconds, so as to strike a balance between timely reflecting node status changes and avoiding misjudgment.

[0063] S3.7: Based on the mapping relationship between the data model in S2.8 and the Thrift service interface, design a client request allocation mechanism to allocate requests to corresponding data nodes.

[0064] S3.8: Design cross-data node query and aggregation mechanisms to allow access to data on multiple data nodes through a single Thrift service interface.

[0065] S3.9: Build a network topology that is compatible with Thrift services.

[0066] It should be noted that the network topology includes the physical deployment scheme and connection relationship of data nodes, routing nodes, management nodes and ZooKeeper cluster.

[0067] S3.10: Develop metadata management services to manage data models, data node partition information, and network topology, and provide Thrift service interfaces for other services to call.

[0068] S4: Develop a high-speed read and write interface based on Thrift for data models and network topologies.

[0069] Specifically, based on the class_def table and field_def table defined in S2, the Thrift service interface data structure is designed, including the request and response structure of read and write operations; a read service interface based on the Thrift protocol is constructed, and the read service interface includes a single data reading method, a batch data reading method and a conditional query method, and matches the data model and Thrift service interface mapping relationship established in S2.8; a write service interface based on the Thrift protocol is constructed, and the write service interface includes a single data writing method, a batch data writing method and a data update method, which also matches the data model and Thrift service interface mapping relationship established in S2.8; a data serialization and deserialization mechanism is designed for data conversion between the Thrift service interface and the real-time data storage structure; based on the data node partitioning strategy designed in S3.1 and the data routing layer in S3.5, a data routing mechanism for read and write requests is developed to distribute the requests to the corresponding data nodes; a concurrency control mechanism is designed, and ZooKeeper is used to implement distributed locks to handle the situation where multiple clients access the same data at the same time; a data access optimization mechanism is designed, including a data cache layer, a data consistency guarantee mechanism, an asynchronous processing mechanism and a performance monitoring interface.

[0070] It should be noted that the data cache layer is used to improve the reading efficiency of hot data; the data consistency guarantee mechanism ensures data consistency in a distributed environment; the asynchronous processing mechanism improves the efficiency of batch operations; and the performance monitoring interface is used to collect performance indicators of read and write operations.

[0071] S5: Integrate the version comparison mechanism in the high-speed read and write interface, and design corresponding incremental and full data synchronization strategies to complete distributed real-time data storage, access, and synchronization.

[0072] Specifically, add row version number and node version number fields in the real-time data table to track data update status; modify the write service interface in S4 to update the row version number and node version number when data is written; build a version comparison interface for exchanging and comparing version information between nodes; establish an incremental synchronization buffer to store the latest incremental synchronization data (arranged in chronological order); develop an incremental synchronization mechanism to transmit incremental data based on version comparison results; formulate full synchronization trigger conditions to start full synchronization when a request with a version number of 0 is received; establish a full synchronization mechanism to transmit the complete data set of the node; formulate a concurrency control strategy to coordinate the simultaneous incremental synchronization and full synchronization; establish a data merge mechanism to handle data conflicts and version updates during the synchronization process, and at the same time track and update the online, offline, and initialization status of the node.

[0073] Furthermore, the present embodiment also provides a distributed real-time data storage, access, and synchronization system, including a framework construction module, which is used to adopt the thrift microservice architecture, define service interfaces and communication protocols, and build a distributed system framework; a storage structure construction module, which is used to build a real-time data storage structure based on the distributed system framework, customize data types and data models; a partitioning strategy module, which is used to formulate a node partitioning strategy based on the real-time data storage structure, and establish a network topology adapted to the thrift service; a read-write interface module, which is used to develop a high-speed read-write interface based on thrift for data models and network topologies; a comparison and synchronization module, which is used to integrate a version comparison mechanism in a high-speed read-write interface, and design corresponding incremental and full data synchronization strategies to complete distributed real-time data storage, access, and synchronization.

[0074] This embodiment also provides a computer device, which is suitable for the distributed real-time data storage, access, and synchronization method, including a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute computer-executable instructions to implement the distributed real-time data storage, access, and synchronization method proposed in the above embodiment.

[0075] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0076] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the distributed real-time data storage, access, and synchronization method proposed in the above embodiment is implemented.

[0077] In summary, the present invention adopts Thrift microservice architecture to define service interface and communication protocol, and builds an efficient cross-language distributed system framework. Distributed configuration management is performed through ZooKeeper, and SQLite is introduced as local configuration storage to achieve configuration consistency and dynamic management capabilities. The system designs an overall framework including data nodes, metadata management and load balancer, and supports dynamic expansion of custom data types and data models. Based on the Thrift service interface and data model mapping relationship, an efficient real-time data storage structure is constructed. The rationality and load balancing of data distribution are ensured through partitioning strategy and dynamic adjustment mechanism. The system provides a high-speed read-write interface and data routing layer based on Thrift to ensure the efficiency and consistency of data reading and writing. Through version comparison mechanism and incremental and full synchronization strategy, data consistency and real-time synchronization in a distributed environment are ensured. The present invention greatly improves the performance, reliability, scalability and management flexibility of the system, and meets the needs of distributed real-time data storage, access and synchronization.

[0078] Example 2

[0079] Reference Figure 1 to Figure 4 , which is the second embodiment of the present invention, and this embodiment provides a distributed real-time data storage, access, and synchronization method. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0080] Specifically, a cluster of 5 servers was deployed in the experimental environment. Each server is equipped with an Intel Xeon E5-2630 v4@2.20GHz CPU (10 cores), 32GB memory, 1TB SSD hard disk, and Gigabit Ethernet. The software environment includes Ubuntu 20.04LTS operating system, Thrift0.14.2, ZooKeeper 3.6.2, and SQLite3.31.1. The experimental dataset simulates two data types: sensor data and user behavior data. The former contains timestamp, sensor ID, and sensor value fields, and the latter contains timestamp, user ID, behavior type, and behavior details fields.

[0081] First, class_def and field_def tables were created in the SQLite local configuration repository of each data node, and the initial data model definition was inserted. Through ZooKeeper, the central configuration information was synchronized to the SQLite local storage of each data node. The configuration synchronization time was about 5 seconds, and the configuration versions of all nodes were consistent, without any configuration inconsistency.

[0082] Next, the data structure of the Thrift service interface is defined, including data reading and writing methods, and a mapping relationship is established with the data model. Through this mapping relationship, data reading and writing operations proceed smoothly.

[0083] In the process of building the real-time data storage structure, each data node creates a corresponding real-time data table according to the data model definition and registers these data table names in ZooKeeper. The data table naming rule is "node name + model type name" to ensure that the data tables of different nodes and models are uniquely identified.

[0084] To ensure high availability and load balancing of the system, the partition strategy is dynamically adjusted and partition information nodes are created in ZooKeeper. The node health check mechanism effectively detects and handles node failures, and the system always maintains high availability.

[0085] The high-speed read and write interface based on Thrift performed well. The average response time for single data read was 10 milliseconds, and the average response time for write was 12 milliseconds. In high-concurrency scenarios, the interface can still maintain low latency. The average response time for batch data read and write operations was 50 milliseconds and 60 milliseconds, respectively, processing 100 data at a time.

[0086] The incremental synchronization mechanism effectively reduces the amount of data transmission and ensures data consistency and real-time performance. Full synchronization is performed on large data volumes, with an average synchronization time of 5 minutes. The partition adjustment strategy is triggered when the node load changes significantly, with an average adjustment time of 1 minute. During data migration, the average migration time is 30 seconds, ensuring smooth partition adjustment.

[0087] The node health check mechanism regularly checks the health status of nodes through ZooKeeper at an interval of 10 seconds to ensure the stability and reliability of the system. When a node failure is detected, the system can quickly remove the abnormal node and trigger the data redistribution process.

[0088] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A distributed real-time data storage, access, and synchronization method, characterized in that: include, Adopt thrift microservice architecture, define service interface and communication protocol, and build distributed system framework; Based on the distributed system framework, custom data types and data models are used to build a real-time data storage structure; Based on the real-time data storage structure, formulate a node partitioning strategy and establish a network topology adapted to the thrift service; Developing a high-speed read and write interface based on thrift for the data model and the network topology; A version comparison mechanism is integrated into the high-speed read-write interface, and corresponding incremental and full data synchronization strategies are designed to complete distributed real-time data storage, access, and synchronization.

2. The distributed real-time data storage, access, and synchronization method according to claim 1, characterized in that: The construction of the real-time data storage structure comprises the following steps: Create a class_def table in the SQLite local configuration library of each data node; Create a field_def table in the same SQLit local configuration repository; Design a SQLite configuration synchronization mechanism to synchronize central configuration information to the SQLite local storage of each data node through ZooKeeper; Based on the class_def table and the field_def table, a dynamic extension interface of data types and data models is designed to support adding or modifying data types and fields at runtime, and push the changes to all nodes through ZooKeeper; Establish naming rules for real-time data tables; Perform data configuration based on the real-time data table naming rules and node names, and use ZooKeeper to coordinate each node; Develop SQLite configuration reading function to extract data structure definitions from local class_def and field_def tables; Establish the mapping relationship between the data model and the Thrift service interface; According to the data structure definition and mapping relationship, a real-time data storage structure is built on each data node, and the data distribution of each node is coordinated through ZooKeeper.

3. The distributed real-time data storage, access, and synchronization method according to claim 1, characterized in that: The node partitioning strategy is formulated including the following steps: Based on the naming rules of real-time data tables, design data node partitioning strategies and determine how data is distributed on different data nodes; Creating a partition information node in ZooKeeper, where the partition information node is used to store partition strategy and node allocation information; Develop partition management services and provide Thrift interfaces for querying, updating, and synchronizing partition information; Design a dynamic partition adjustment mechanism to adjust data distribution according to load conditions and notify each data node through ZooKeeper; Based on the Thrift service interface, a data routing layer is designed, which is used to forward client requests to corresponding data nodes; Develop a node health check mechanism to monitor the data node status through ZooKeeper and trigger corresponding actions when the data node status changes; Based on the mapping relationship between the data model and the Thrift service interface, a client request allocation mechanism is designed to allocate requests to corresponding data nodes; Design cross-data node query and aggregation mechanisms to allow access to data on multiple data nodes through a single Thrift service interface; Build a network topology that adapts to Thrift services; Develop metadata management services to manage data models, data node partition information, and network topology.

4. The distributed real-time data storage, access, and synchronization method according to claim 3, characterized in that: The step of adjusting data distribution according to load conditions comprises the following steps: Construct a load monitoring module to collect the performance indicator set of each data node at a preset time interval T1; Map the collected performance indicator set I to the normalized load score S of the data node, where 0≤S≤1; Set the load threshold interval [L min ,L max ], when the normalized load score S of the data node satisfies S>L max Or S<L min When the condition lasts longer than the preset time T2, the partition adjustment process is triggered; Design a data partition migration decision module based on the normalized load score set S = {S1,...,S n } and the current partition distribution state P to generate a migration plan M = {M1,...,M k }; Create a migration task node set Z in ZooKeeper = {Z1,...,Z k }; Develop a data migration execution module E(Z) to monitor the migration task node set Z in ZooKeeper and execute data migration operations according to the preset concurrency C; Design partition information update mechanism, in the migration task M i After completion, update the partition mapping relationship in ZooKeeper; When the partition mapping relationship changes, the updated partition information is pushed to all relevant nodes through ZooKeeper within the preset time window T3; Develop a rollback mechanism B(M,R0,E), where M is the currently executed migration plan, R0 is the initial partition mapping relationship before adjustment, and E is a predefined set of exception situations. When an abnormal event in E is detected during the partition adjustment process, the following operations are performed: Abort all ongoing data migration operations; Perform reverse data transfer on the completed migration task and restore the data to the source node; Reset the partition mapping relationship in ZooKeeper to R0; Broadcast the rollback completion notification to all relevant nodes so that they can update the partition information in their local cache.

5. The distributed real-time data storage, access, and synchronization method according to claim 4, characterized in that: The calculation formula of the normalized load score S is as follows: Among them, α, β, γ, and δ are weight coefficients, and α+β+γ+δ=1, CPU uage is the CPU usage, MEM uage is the memory usage, IO rate is the disk I / O rate, IO max is the maximum theoretical value of disk I / O, NET rate is the network traffic rate, NET max It is the maximum theoretical value of network traffic.

6. The distributed real-time data storage, access, and synchronization method according to claim 3, characterized in that: The node health check mechanism includes: Create a corresponding temporary node for each data node in ZooKeeper; Design a health report data structure, wherein the health report data structure includes basic information and health status indicators of the data node; Design a data node health report module to write the latest health report to the corresponding ZooKeeper temporary node at a preset time interval T4; Develop a central monitoring service to monitor the temporary nodes of all data nodes through ZooKeeper's Watch mechanism; Set up health status determination rule set; When a first situation is detected, the data node is marked as abnormal, wherein the first situation includes that the ZooKeeper temporary node is deleted, and an indicator in the health report H violates any rule in the health status determination rule set; Design a state change processing module to execute the first operation when the state of a data node changes from normal to abnormal; When the abnormal node returns to a normal state, the second operation is performed.

7. The distributed real-time data storage, access, and synchronization method according to claim 6, characterized in that: The first operation includes: Remove abnormal nodes from the load balancer, trigger the data redistribution process, redistribute data partitions on abnormal nodes to healthy nodes, and send alarm notifications to system administrators; The second operation includes: Re-add the node to the load balancer, trigger the data synchronization process, update the system status, and record the node recovery event.

8. A distributed real-time data storage, access, and synchronization system, based on the distributed real-time data storage, access, and synchronization method according to any one of claims 1 to 7, characterized in that: Also includes, The framework building module is used to adopt the thrift microservice architecture, define service interfaces and communication protocols, and build a distributed system framework; A storage structure building module, used to build a real-time data storage structure based on the distributed system framework, custom data types and data models; A partition strategy module is used to formulate a node partition strategy based on the real-time data storage structure and establish a network topology adapted to the thrift service; A read / write interface module, used to develop a high-speed read / write interface based on thrift for the data model and the network topology; The comparison and synchronization module is used to integrate the version comparison mechanism in the high-speed read and write interface, and to design corresponding incremental and full data synchronization strategies to complete distributed real-time data storage, access, and synchronization.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the distributed real-time data storage, access, and synchronization method described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the distributed real-time data storage, access, and synchronization method described in any one of claims 1 to 7 are implemented.

Images