Air traffic control distributed storage middleware based on data consistency copy and consensus algorithm

Abstract

The application relates to the technical field of distributed data distribution and reliable storage, and provides an air traffic control distributed storage middleware based on data consistency copy and a consensus algorithm, which comprises a storage module using Redis as a storage basic platform, a client request processing module used for listening to a network port, asynchronously receiving a client request and pre-processing the client request, a Raft log management module used for distributing operation sequences to distributed nodes through a Raft algorithm and maintaining the operation sequences on the distributed nodes through the Raft algorithm, an asynchronous execution module used for executing the operation sequences on the distributed nodes through the Raft algorithm and storing data according to a consistency storage rule of the storage basic platform, and a data persistence module used for persisting storage data through a system snapshot. According to the characteristics of air traffic control field data, the application realizes a data storage method which takes into account data consistency and partition tolerance, guarantees response time and as far as possible guarantees availability.

Description

Technical Field

[0001] This invention relates to the field of distributed data distribution and reliable storage technology, and in particular to an empty-management distributed storage middleware based on data consistency copying and consensus algorithms. Background Technology

[0002] Air traffic control (ADR), also known as air traffic management, is a technology that monitors and manages aircraft in flight. ADR is a crucial component of the aviation industry; without it, aircraft status is difficult to track, and flight safety and controllability are compromised. Through automated air traffic control systems, airlines can further improve the quantity and quality of flights. The increasing demand for air traffic volume has led to a surge in flight frequency, while the demand for higher quality has resulted in a significant increase in the amount of data required to record per flight. This has caused a dramatic increase in the amount of data that traditional automated air traffic control systems need to process, leading to performance bottlenecks. Currently, data persistence, data consistency, response speed, and partition tolerance in automated air traffic control systems are difficult to guarantee. Summary of the Invention

[0003] This invention aims to solve at least one of the technical problems existing in related technologies. To this end, this invention provides an air-managed distributed storage middleware based on data consistency copying and consensus algorithms. It implements a data storage method that balances data consistency and partition tolerance, guarantees response time, and maximizes availability. It achieves multi-replica consistent data distribution, improves the synchronization problem of data consistency across multiple nodes while ensuring performance, and has significant value in industrial and academic applications.

[0004] This invention provides an empty-management distributed storage middleware based on data consistency copying and consensus algorithms, comprising:

[0005] The storage module uses Redis as the underlying storage platform for actual data storage.

[0006] A client request processing module is used to listen to the network port and asynchronously receive client requests, preprocess the client requests, and obtain the operation sequence of the storage infrastructure platform.

[0007] The Raft log management module distributes the operation sequence to distributed nodes using the Raft algorithm and maintains the operation sequence on the distributed nodes using the Raft algorithm.

[0008] An asynchronous execution module parses the operation sequence on the distributed node, executes the operation sequence on the distributed node using the Raft algorithm, and stores the empty data on the storage platform of the distributed node according to the consistent storage rules of the storage platform, obtains the stored data, and returns the stored data to the client request processing module, which then returns the stored data to the client.

[0009] A data persistence module persists the stored data through system snapshots.

[0010] Furthermore, the storage infrastructure platform includes multiple storage partitions, and the storage infrastructure platform includes any one of memory, disk, and relational database.

[0011] Furthermore, preprocessing the client request includes:

[0012] The request body is designed based on Protobuf. The request body includes data metadata and actual data information. The data metadata includes a universally unique identifier and a timestamp. The actual data information includes the type of operation on the server and the content of the data itself.

[0013] Based on the request body, the client request is designed as a message base class. The message base class includes a universal unique identifier, a timestamp, and an indicator field. The indicator field is used to identify the sub-message corresponding to the message base class. The sub-message includes a request message and a response message. The request message has multiple fields set according to the operation type. The fields include add, delete, modify, and query. The response message includes a request status code and response information.

[0014] Furthermore, the consistent storage rule of the storage infrastructure platform is to design a language-independent universal rule as a request body, including the actual way of writing data, the target of writing, and the content of writing.

[0015] Furthermore, the Raft log management module includes:

[0016] The operation sequence is packaged into a Raft log;

[0017] The Raft logs are then transferred to the Raft algorithm cluster.

[0018] The Raft algorithm cluster distributes empty management data to distributed nodes through log replication.

[0019] Furthermore, the asynchronous execution module includes:

[0020] The operation sequence on the distributed nodes is deserialized to obtain the actual operation sequence;

[0021] Obtain the key values ​​of the actual operation sequence, perform hash mapping on the key values, and assign the actual tasks to the specified threads according to the mapping results;

[0022] The hash mapping is asynchronously invoked according to the consistent storage rules of the storage infrastructure platform to obtain the stored data.

[0023] Furthermore, when the asynchronous execution module fails to execute a Redis request, it triggers an exception handling process, which includes:

[0024] The system assesses the execution status and returns error information to the client if the request format or syntax error was caused by the client.

[0025] If the request failure is caused by a Redis server-side exception, the exception will be handled through retries, command compensation, and sending the request to the status monitoring module.

[0026] Furthermore, the data persistence module includes:

[0027] Whenever the Raft logs reach a specified number of entries, a system snapshot is taken to save the system state and the data in the storage platform. The snapshot is saved in the file system.

[0028] Deleting snapshotted data from the middleware's memory does not affect data in the storage infrastructure platform.

[0029] When a distributed node recovers from a failure, historical data is supplemented through system snapshots, and new data is supplemented through log replication.

[0030] Furthermore, it also includes a Redis connection pool management module, which is used to maintain a connection pool with Redis, allocate a Redis connection for each write request, and provide connection resources with the storage infrastructure platform through the Redis connection pool. The Redis connection pool has the functions of automatic connection reconstruction, connection quantity limit, and timeout notification.

[0031] Furthermore, it also includes an instruction buffering and scheduling module, which is used to semantically merge multiple written Redis instructions, including:

[0032] Set up a cache request queue for requests of the same type, and push the requests into the cache request queue for buffering;

[0033] Set a time threshold. If the difference between the current request time and the earliest request time in the request queue is greater than the time threshold, merge the requests in the cached request queue for execution.

[0034] Set a quantity threshold. If the number of requests in the cache request queue is greater than or equal to the quantity threshold, the requests in the cache request queue will be merged and executed.

[0035] The above-described one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects:

[0036] As an important component of the air traffic control distributed middleware, this invention plays a crucial role in data storage, real-time data distribution, and data subscription notification. It interacts with the system monitoring module, inter-process communication module, and process management module to jointly provide a complete interactive process for air traffic control operations.

[0037] This invention addresses the practical needs and applications of automated systems in air traffic control, aiming to propose an efficient distributed reliable data storage method. Based on the server-side and client-side logic, consensus algorithm, basic storage platform, high-performance serialization library, and distributed deployment, this invention achieves a data storage method that integrates data consistency and partition tolerance while maximizing availability. Through a data consistency copying approach, it achieves multi-replica consistent data distribution, ensuring performance while improving the synchronization problem of data consistency across multiple nodes, and possesses significant value in industrial and academic applications.

[0038] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0040] Figure 1 This is a schematic diagram of the internal structure of a Raft node in an empty-loop distributed storage middleware based on data consistency copying and consensus algorithm provided by the present invention.

[0041] Figure 2 This is a schematic diagram of the overall cluster structure of an empty-management distributed storage middleware based on data consistency copying and consensus algorithms.

[0042] Figure 3 This is a schematic diagram illustrating the functional application of an embodiment of the present invention.

[0043] Figure label:

[0044] 100. Client request processing module; 200. Storage module; 300. Raft log management module; 400. Asynchronous execution module; 500. Redis connection pool management module; 600. Data persistence module; 700. Data subscription module; 800. Instruction buffer and scheduling module. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention. The following embodiments are used to illustrate this invention but cannot be used to limit the scope of this invention.

[0046] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0047] The following is combined Figures 1 to 3 This invention describes an empty-loop distributed storage middleware based on data consistency copying and consensus algorithms.

[0048] An empty-loop distributed storage middleware based on data consistency copying and consensus algorithms includes:

[0049] like Figure 1 As shown, storage module 200 uses Redis as the storage platform for actual data storage;

[0050] Redis (REmote DIctionary Server) is a key-value store system written by Salvatore Sanfilippo. It is a cross-platform, non-relational database.

[0051] The client request processing module 100 listens to the network port and asynchronously receives client requests, preprocesses the client requests, and obtains the operation sequence of the storage infrastructure platform.

[0052] In some specific embodiments of the present invention, the network port needs to be located on the local machine to receive messages in UDP (User Datagram Protocol) format. The content of the message and the result after execution are defined by Protobuf, which can achieve language independence and platform independence. It supports messages created using common system software programming languages ​​such as C, C++, Java, and Rust. Sub-messages under the message base class include request messages and response messages. The request message sets multiple fields according to the operation type. The fields include add, delete, modify, and query. Specifically, it includes five parameters: operation type, key to be operated, field to be operated, index to be operated, and data content. The operation type can be divided into three types: read, write, and system operation. Read includes four types: sequential read, index read, selected paragraph read, and get. Write includes five operations: create, append write, index write, delete by item, and set content. System operations include instructions such as add server, view status, view server list, and help.

[0053] The response message includes a request status code and response information;

[0054] Specifically, the server will respond appropriately according to the request status code table based on the actual execution result of the request.

[0055] Preprocessing client requests includes:

[0056] The request body is designed based on Protobuf (Protocol Buffers, a lightweight and efficient structured data storage format). The request body includes data metadata and actual data information. The data metadata includes a universally unique identifier (UUID) and a timestamp. The actual data information includes the type of operation to the server and the content of the data itself.

[0057] Based on the request body, the client request is designed as a message base class. The message base class includes a universal unique identifier (UUID), a timestamp, and an indicator field. The indicator field is used to identify the sub-message corresponding to the message base class. The sub-message includes a request message and a response message. The request message has multiple fields set according to the operation type. The fields include add, delete, modify, and query. The response message includes a request status code and response information.

[0058] Parse the client request to obtain the request type;

[0059] Furthermore, after the server receives a request, it will parse the different types of the request in the system and index the corresponding request type to perform the operation.

[0060] The request type is converted into an operation sequence for the storage infrastructure platform.

[0061] Specifically, if the received request is a read request, the system requests the result from the storage infrastructure platform and returns the result to the client; if the received request is a write request, the system converts the write request into an operation sequence of the infrastructure platform.

[0062] The storage infrastructure platform includes multiple storage partitions, which can be configured independently according to actual needs. By simply connecting the read / write operation APIs of the corresponding storage modules to the storage method framework described in this invention, the storage module 200 can be configured independently. Selectable storage platforms include memory, disk, relational databases, etc.

[0063] The storage module 200 of this invention can use Redis Insight (the official visual management and monitoring tool launched by Redis) to display data, and can realize some basic storage monitoring functions, laying the foundation for actual access to data analysis and supervision.

[0064] Specifically, such as Figure 1 As shown, external requests (such as write operations) enter the Raft server of the Raft log management module 300 and are converted into Raft log entries.

[0065] The Raft server sends the Raft logs to the Raft algorithm, and log consensus is reached through the Raft protocol (such as a majority of nodes successfully replicating the logs).

[0066] Once consensus is reached, the Raft algorithm submits the logs to the asynchronous execution module 400 for asynchronous execution, decoupling the consensus process from business execution and improving performance.

[0067] The asynchronous execution module 400 receives the "committed" log, parses it into a business instruction, and sends it to the instruction buffer and scheduling module 800.

[0068] The instruction buffering and scheduling module 800 buffers instructions, schedules them according to rules, and forwards them to the Redis connection pool management module 500. The Redis connection pool management module 500 allocates active connections to Redis, and operates on Redis through the storage module 200. Data is written to Redis by applying the instructions, and intermediate data can be temporarily stored with the help of the storage module 200.

[0069] The Redis connection pool management module 500 reuses Redis connections through a connection pool to perform data read / write / replication.

[0070] The data persistence module 600 persists critical data (such as system status and snapshots) to reliable storage media (such as disks) and supports node restart recovery.

[0071] The Raft protocol avoids blocking Raft threads and ensures consensus efficiency by executing business logic asynchronously. Redis connection creation has high overhead; connection pooling reduces latency and improves data read / write performance. Data persistence ensures no data loss in the event of node failure.

[0072] Raft's snapshot mechanism can compress "old Raft logs" into snapshots, reducing storage usage and accelerating state synchronization when new nodes join.

[0073] Furthermore, the Raft log management module 300 distributes the operation sequence to the distributed nodes using the Raft algorithm, and maintains the operation sequence on the distributed nodes using the Raft algorithm.

[0074] Specifically, such as Figure 3 As shown, the operation sequence is packaged into a Raft log; Figure 3 The 300-square box in the Raft log management module represents Figure 1 The Raft node is shown.

[0075] The Raft logs are then transferred to the Raft algorithm cluster.

[0076] The Raft algorithm cluster performs master node election, log replication, majority confirmation, commit, and consistency guarantees.

[0077] like Figure 2 As shown, the three Raft nodes (Raft node 1, Raft node 2, and Raft node 3) communicate with each other through the Raft algorithm to achieve distributed consensus and ensure data consistency across multiple nodes.

[0078] Client processes within a node (such as client process 1, client process 2, client process 3, and client process 4) initiate a "service request" to the local server, which is then processed by the Raft node on the server side.

[0079] External processes (seats, other system modules, etc.) send a "service request" to the server on any node.

[0080] Client processes can directly access the local Redis (e.g., client process 1 → node 1 Redis, client process 2 → node 2 Redis, client process 3 → node 3 Redis).

[0081] After the server processes the request, it "submits" the data to the local Redis (e.g., server 1 → node 1 Redis, server 2 → node 2 Redis, server 3 → node 3 Redis) to achieve the connection between computing and storage.

[0082] The Raft algorithm enables the three-node server to reach a consensus, tolerating the failure of some nodes.

[0083] Client processes prioritize accessing the local Redis and server to reduce cross-node latency; cross-node interactions are ensured to maintain data consistency through Raft.

[0084] External processes can be flexibly connected and interact with the distributed system through "request services," decoupling business logic from the consensus layer.

[0085] When a brief inconsistency occurs in the distributed node data, a cursor-based data replication method is used for log replication;

[0086] The Raft algorithm cluster distributes empty management data to distributed nodes through log replication.

[0087] This invention uses the Raft consensus algorithm for master node election and log replication, and supports automatic routing of client requests to the master node for processing.

[0088] In a normally functioning distributed node cluster, each node already contains the Raft logs corresponding to the aforementioned requests.

[0089] Furthermore, set consistent storage rules for the storage infrastructure platform;

[0090] The request body is designed based on Protobuf. The request body includes data metadata and actual data information. The data metadata includes a universally unique identifier and a timestamp. The actual data information includes the type of operation performed on the server and the content of the data itself.

[0091] The consistent storage rule of the storage infrastructure platform is to design a language-independent universal rule as a request body, which includes the actual way of writing data, the target of the write, and the content to be written.

[0092] Specifically, the language-independent general rule design is implemented using Protobuf. Language independence means it's independent of the client's implementation language; it only requires generating serialization utility classes and functions suitable for the currently used programming language using a proto file written in Protobuf. Essentially, the server receives content serialized using Protobuf, and client applications can achieve language independence through Protobuf.

[0093] The consistent storage rules of the storage infrastructure platform are designed as a type of request, which includes the method of actually writing data, the target of the write, and the content of the write.

[0094] Furthermore, the asynchronous execution module parses the operation sequence on the distributed node, executes the operation sequence on the distributed node using the Raft algorithm, and stores the empty management data on the storage platform of the distributed node according to the consistent storage rules of the storage platform, obtains the storage data, and returns the storage data to the client request processing module 100, which then returns the storage data to the client.

[0095] Specifically, the operation sequence on the distributed nodes is deserialized to obtain the actual operation sequence;

[0096] Obtain the key values ​​of the actual operation sequence, perform hash mapping on the key values, and assign the actual tasks to the specified threads according to the mapping results;

[0097] The hash mapping is asynchronously invoked according to the consistent storage rules of the storage infrastructure platform to obtain the stored data.

[0098] This invention guarantees that key-value pairs that can obtain the same hash value will always be processed by the same thread. Therefore, hash mapping by key-value pair will not disrupt the consistency of the order in which data is written to the same key, and will not compromise the correctness of the log storage results.

[0099] When the asynchronous execution module 400 fails to execute a Redis request, it triggers an exception handling process, which includes:

[0100] The system assesses the execution status and returns error information to the client if the request format or syntax error was caused by the client.

[0101] If the request failure is caused by an exception in the Redis server, the exception will be handled through strategies such as retrying, command compensation, and sending the request to the status monitoring module.

[0102] Furthermore, the data persistence module 600 persists the stored data through system snapshots.

[0103] Specifically, whenever the Raft logs reach a specified number of entries, a system snapshot is used to save the system state and data in the storage platform. The snapshot is saved in the file system.

[0104] A snapshot is taken whenever the Raft logs reach a specified number of entries. The snapshot consists of two parts: one part is the current system state, and the other part is the actual storage content of the storage infrastructure platform.

[0105] Delete the snapshotted data that is occupying memory in the middleware. Since the Raft logs that package this data have now been applied to the storage platform, the task of saving these operation sequences has been completed, and the operation process can be replayed by applying snapshots. Therefore, the Raft logs before the snapshot can be deleted to reduce memory usage.

[0106] When a distributed node recovers from a failure, historical data is supplemented through system snapshots, and new data is supplemented through log replication.

[0107] When copying snapshots, it is necessary to read two parts of the snapshots stored. In the actual snapshot copying process, it is necessary to copy the actual snapshot of the storage platform.

[0108] Furthermore, the Redis connection pool management module is used to maintain the connection pool between Redis and the storage infrastructure platform, allocate a Redis connection for each write request, and provide connection resources with the storage infrastructure platform through the Redis connection pool. The Redis connection pool has the functions of automatic connection reconstruction, connection quantity limit, and timeout notification.

[0109] In some specific embodiments of the present invention, when hash mapping is performed on the key value of the operation sequence corresponding to each log, it is mapped to each thread in the thread pool. After the thread requests an active connection from the Redis connection pool management module, it writes to Redis through the connection. The management and allocation of the connection is the responsibility of the connection pool of the Redis connection pool management module.

[0110] Furthermore, it also includes an instruction buffering and scheduling module 800 for semantic merging of multiple written Redis instructions, including:

[0111] Set up a cache request queue for requests of the same type, and push the requests into the cache request queue for buffering;

[0112] Set a time threshold. If the difference between the current request time and the earliest request time in the request queue is greater than the time threshold, merge the requests in the cached request queue for execution.

[0113] Set a quantity threshold. If the number of requests in the cache request queue is greater than or equal to the quantity threshold, the requests in the cache request queue will be merged and executed.

[0114] Furthermore, it also includes a data subscription module 700 for client nodes to subscribe to data they are interested in and receive message notifications when the data changes.

[0115] like Figure 3As shown, after the client request processing module 100 processes and transforms the request, it forwards the "general request" to the master node of the Raft log management module 300 (in the Raft protocol, the master node is responsible for handling write requests, while read requests can be flexibly routed).

[0116] After the master node processes the requests in the Raft log management module 300, the requests are synchronized to the slave nodes (slave node 1, slave node 2... slave node n) through Raft log replication to ensure data consistency across multiple nodes.

[0117] The processing result is reverse-converted by the client request processing module 100 and then written back to the upper-layer application.

[0118] When the system starts up or when the current master node goes offline, a master node is elected from among the other available nodes through the Raft protocol's election mechanism to handle client requests.

[0119] The master node encapsulates requests into Raft logs, which are stored in the node's memory and replicated to all slave nodes through a log replication mechanism to ensure log sequence consistency.

[0120] The data subscription module 700 registers the data it is interested in (such as specific log changes or master-slave status switching) with the Raft log management module 300.

[0121] When the internal data of the Raft log management module 300 changes (such as log writing or master node switching), it actively notifies the data subscription module 700, triggering the subsequent logic of the data subscription module 700 (such as cache update, monitoring alarm, etc.).

[0122] The client request processing module 100 focuses on request access, the Raft log management module 300 focuses on distributed consensus, and the data subscription module 700 focuses on data subscription.

[0123] By employing the Raft algorithm for master-slave election and log replication, it tolerates node failures and ensures strong data consistency.

[0124] The data subscription module 700 allows external modules to be aware of changes in the state of the distributed cluster and to expand flexibly (such as real-time monitoring and heterogeneous system synchronization).

[0125] In some specific embodiments of this invention, actual integrated flight track data from domestic air traffic control departments were used for evaluation and verification. The integrated flight track data conforms to the ASTERIX CAT062 standard, with a total data volume of 118,437,923 KB, containing 767,683 integrated flight track data entries, and an average data entry length of 154.28 bytes. The evaluation and verification were conducted using five virtual machine nodes on a Kunpeng server. Each virtual machine node has 8 virtual CPUs, 12GB of memory, and runs on the OpenEuleler OS E2203 SP4 version on an ARM architecture server. The actual network bandwidth between the virtual machines was approximately 21,909 Mbits / sec. Experimental results measured the latency of a single request and the latency of request synchronization, both within the millisecond range. Furthermore, data consistency was verified during network partitioning, showing that when half or more of the nodes were online, all online nodes achieved data consistency.

[0126] As an important component of the air traffic control distributed middleware, this invention plays a crucial role in data storage, real-time data distribution, and data subscription notification. It interacts with the system monitoring module, inter-process communication module, and process management module to jointly provide a complete interactive process for air traffic control operations.

[0127] This invention addresses the practical needs and applications of automated systems in air traffic control, aiming to propose an efficient distributed reliable data storage method. Based on the server-side and client-side logic, consensus algorithm, basic storage platform, high-performance serialization library, and distributed deployment, this invention achieves a data storage method that integrates data consistency and partition tolerance while maximizing availability. Through a data consistency copying approach, it achieves multi-replica consistent data distribution, ensuring performance while improving the synchronization problem of data consistency across multiple nodes, and possesses significant value in industrial and academic applications.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A distributed storage middleware based on data consistency copying and consensus algorithms, characterized in that: include: The storage module uses Redis as the underlying storage platform for actual data storage. A client request processing module is used to listen to the network port and asynchronously receive client requests, preprocess the client requests, and obtain the operation sequence of the storage infrastructure platform. The Raft log management module distributes the operation sequence to distributed nodes using the Raft algorithm and maintains the operation sequence on the distributed nodes using the Raft algorithm. The Raft algorithm cluster performs master node election, log replication, majority confirmation, commit, and consistency guarantees, tolerates node failures, and ensures strong data consistency. An asynchronous execution module parses the operation sequence on the distributed node, executes the operation sequence on the distributed node using the Raft algorithm, and stores the empty data on the storage platform of the distributed node according to the consistent storage rules of the storage platform, obtains the stored data, and returns the stored data to the client request processing module, which then returns the stored data to the client. The asynchronous execution module includes: The operation sequence on the distributed nodes is deserialized to obtain the actual operation sequence; Obtain the key values ​​of the actual operation sequence, perform hash mapping on the key values, and assign the actual tasks to the specified threads according to the mapping results; The hash mapping is asynchronously invoked according to the consistent storage rules of the storage infrastructure platform to obtain the stored data; A data persistence module persists the stored data through system snapshots; The Redis connection pool management module is used to maintain a connection pool with Redis, allocate a Redis connection for each write request, and provide connection resources with the storage infrastructure platform through the Redis connection pool. The Redis connection pool has the functions of automatic connection reconstruction, connection quantity limit, and timeout notification. An instruction buffering and scheduling module, used to semantically merge multiple written Redis instructions, including: Set up a cache request queue for requests of the same type, and push the requests into the cache request queue for buffering; Set a time threshold. If the difference between the current request time and the earliest request time in the request queue is greater than the time threshold, merge the requests in the cached request queue for execution. Set a quantity threshold. If the number of requests in the cache request queue is greater than or equal to the quantity threshold, the requests in the cache request queue will be merged and executed.

2. The distributed storage middleware based on data consistency copying and consensus algorithm according to claim 1, characterized in that, The storage infrastructure platform includes multiple storage partitions, and the storage infrastructure platform includes any one of memory, disk, and relational database.

3. The empty-loop distributed storage middleware based on data consistency copying and consensus algorithm according to claim 1, characterized in that, Preprocessing the client request includes: The request body is designed based on Protobuf. The request body includes data metadata and actual data information. The data metadata includes a universally unique identifier and a timestamp. The actual data information includes the type of operation on the server and the content of the data itself. Based on the request body, the client request is designed as a message base class. The message base class includes a universal unique identifier, a timestamp, and an indicator field. The indicator field is used to identify the sub-message corresponding to the message base class. The sub-message includes a request message and a response message. The request message has multiple fields set according to the operation type. The fields include add, delete, modify, and query. The response message includes a request status code and response information.

4. The empty-loop distributed storage middleware based on data consistency copying and consensus algorithm according to claim 3, characterized in that, The consistent storage rule of the storage infrastructure platform is designed as a request body, which is a language-independent general rule, including the method of actually writing data, the target of writing, and the content to be written.

5. The empty-loop distributed storage middleware based on data consistency copying and consensus algorithm according to claim 1, characterized in that, The Raft log management module includes: The operation sequence is packaged into a Raft log; The Raft logs are then transferred to the Raft algorithm cluster. The Raft algorithm cluster distributes empty management data to distributed nodes through log replication.

6. The empty-loop distributed storage middleware based on data consistency copying and consensus algorithm according to claim 1, characterized in that, When a Redis request fails to execute, the asynchronous execution module triggers an exception handling process, which includes: The system assesses the execution status and returns error information to the client if the request format or syntax error was caused by the client. If the request failure is caused by a Redis server-side exception, the exception will be handled through retries, command compensation, and sending the request to the status monitoring module.

7. The empty-loop distributed storage middleware based on data consistency copying and consensus algorithm according to claim 1, characterized in that, The data persistence module includes: Whenever the Raft logs reach a specified number of entries, a system snapshot is taken to save the system state and the data in the storage platform. The snapshot is saved in the file system. Deleting snapshotted data from the middleware's memory does not affect data in the storage infrastructure platform. When a distributed node recovers from a failure, historical data is supplemented through system snapshots, and new data is supplemented through log replication.

Images