Distributed multi-copy high-performance high-availability scheme based on data file

Abstract

The invention provides a distributed multi-copy high-performance high-availability scheme based on a data file, which comprises the following steps: acquiring a write request of a user, and executing the write request through a leader consensus layer in a distributed cluster; monitoring a data file generated in the storage engine; generating a loading item based on the data file, asynchronously sending the loading item to the follower to load the data file, and continuing to execute the next write-in request by the leader; and after the follower completes data loading through a direct loading technology, a loading result is updated to consensus data of the distributed cluster through an asynchronous callback mode. Compared with a current mainstream consensus protocol, the method adopts an asynchronous replication mechanism, returns a result that user write-in success does not depend on copy synchronization, does not need extra waiting, performs state synchronization by using a data file with a high compression ratio, effectively reduces the performance bottleneck of copy synchronization while ensuring the utilization efficiency of system resources, and improves the user experience. And the performance is linearly expanded along with the number of copies.

Description

Technical Field

[0001] The present disclosure relates to the technical field of computer data management, and in particular to a distributed multi-copy high-performance and high-availability solution based on data files. Background Art

[0002] In distributed systems, the amount of data generated by devices and nodes continues to grow, and the demand for computing and storage resources continues to rise. Due to the limited computing and storage capabilities in a single machine environment, performance bottlenecks are often encountered. Therefore, it is particularly important to develop an efficient distributed consensus algorithm that can manage the synchronization of multiple copies of data in large-scale data scenarios, support a replica synchronization mechanism for multiple copies, and ensure high availability of the system.

[0003] Distributed systems in current technology usually use data partitioning and data replication to organize and manage data. In this architecture, data is divided into multiple independent partitions, and multiple replicas are created for each partition. The data synchronization between replicas is completed through the consensus protocol to ensure high availability and fault tolerance of the data. Through the load balancing mechanism, the system can evenly distribute requests to each replica, thereby allocating traffic pressure and avoiding single point overload. Ideally, if the overhead of synchronization between replicas is small, the effect of data partitioning and traffic sharing can enable the performance of the system to scale horizontally with the number of replicas. This scalability allows the system to add nodes as the load grows in order to continue to provide efficient services. Therefore, the consensus protocol used to organize data synchronization between replicas has become the key to breaking through the performance bottleneck of a single replica in distributed systems. Summary of the invention

[0004] The present disclosure aims to solve one of the technical problems in the related art at least to some extent.

[0005] To this end, one purpose of the present disclosure is to propose a distributed multi-copy high-performance and high-availability solution based on data files.

[0006] The second objective of the present disclosure is to provide a distributed multi-copy high-performance and high-availability device based on data files.

[0007] A third objective of the present disclosure is to provide an electronic device.

[0008] A fourth object of the present disclosure is to provide a non-transitory computer-readable storage medium.

[0009] A fifth object of the present disclosure is to provide a computer program product.

[0010] To achieve the above-mentioned objectives, the first aspect of the implementation of the present disclosure proposes a distributed multi-copy high-performance and high-availability solution based on data files, including: obtaining the user's write request, and executing the write request through the leader consensus layer in the distributed cluster; after the leader consensus layer executes the write request, listening to the latest data file generated in the storage engine; generating a loading event based on the latest data file, and asynchronously sending the loading event to the follower to load the latest data file, while the leader continues to execute the next write request; after the follower completes loading the latest data file through direct loading technology, updating the loading result to the consensus data of the distributed cluster through asynchronous callback.

[0011] According to one embodiment of the present disclosure, the method further includes: in response to restarting a consensus task, obtaining a historical data file in the storage engine; determining the latest data file that has not been loaded by the follower based on the historical data file; and resending the loading event corresponding to the latest data file that has not been loaded to the follower to load the latest data file.

[0012] According to one embodiment of the present disclosure, executing the write request through the leader consensus layer in the distributed cluster includes: forwarding the write request to the state machine through the leader consensus layer; sorting and numbering the write requests to generate a target sorting, and executing the write request based on the target sorting through the state machine.

[0013] According to one embodiment of the present disclosure, after executing the write request based on the target sorting through the state machine, it includes: updating the execution result into the consensus data of the distributed cluster through the state machine.

[0014] According to an embodiment of the present disclosure, generating the loading items based on the latest data file includes: generating an operation identifier of the latest data file based on the target sorting; and generating the loading items based on the operation identifier and the latest data file.

[0015] According to an embodiment of the present disclosure, the asynchronously sending the loading event to the follower to load the latest data file includes: creating an asynchronous callback processor; and asynchronously sending the loading event to the follower to load the latest data file through the asynchronous callback processor.

[0016] According to one embodiment of the present disclosure, updating the loading result to the consensus data of the distributed cluster by means of asynchronous callback includes: in response to the loading result being a successful load, removing the event corresponding to the write request from the buffer, updating the consensus data, and releasing the resources corresponding to the write request; or, in response to the loading result being a loading failure, placing the loading event in a reloading queue for reloading.

[0017] To achieve the above-mentioned purpose, the second aspect of the present disclosure proposes a distributed multi-copy high-performance and high-availability device based on data files, including: an acquisition module, used to obtain the user's write request, and execute the write request through the leader consensus layer in the distributed cluster; a listening module, used to listen to the latest data file generated in the storage engine after the leader consensus layer executes the write request; a loading module, used to generate a loading event based on the latest data file, and asynchronously send the loading event to the follower to load the latest data file; an updating module, used to update the loading result to the consensus data of the distributed cluster through asynchronous callback after the follower completes loading the latest data file.

[0018] To achieve the above-mentioned purpose, the third aspect embodiment of the present disclosure proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to implement a distributed multi-copy high-performance and high-availability solution based on data files as described in the first aspect embodiment of the present disclosure.

[0019] To achieve the above-mentioned purpose, the fourth aspect embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to implement a distributed multi-copy high-performance and high-availability solution based on data files as described in the first aspect embodiment of the present disclosure.

[0020] To achieve the above-mentioned objectives, the fifth aspect embodiment of the present disclosure proposes a computer program product, including a computer program, which, when executed by a processor, is used to implement a distributed multi-copy high-performance and high-availability solution based on data files as described in the first aspect embodiment of the present disclosure.

[0021] Therefore, compared with the current mainstream consensus protocol, the present disclosure adopts an asynchronous replication mechanism to return the result of user write success without relying on replica synchronization, without the need for additional waiting; the present disclosure also uses high-compression data files for state synchronization, which effectively reduces the performance bottleneck of replica synchronization while ensuring the efficiency of system resource utilization, and has the advantage of linear performance expansion with the number of replicas. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a schematic diagram of a distributed multi-copy high-performance and high-availability solution based on data files according to an embodiment of the present disclosure;

[0023] Figure 2 It is a schematic diagram of another distributed multi-copy high-performance and high-availability solution based on data files according to an embodiment of the present disclosure;

[0024] Figure 3 It is a schematic diagram of another distributed multi-copy high-performance and high-availability solution based on data files according to an embodiment of the present disclosure;

[0025] Figure 4 It is a schematic diagram of another distributed multi-copy high-performance and high-availability solution based on data files according to an embodiment of the present disclosure;

[0026] Figure 5 This is an architecture diagram of a distributed multi-copy high-performance and high-availability solution based on data files disclosed in the present invention;

[0027] Figure 6 It is a schematic diagram of a distributed multi-copy high-performance and high-availability device based on data files according to an embodiment of the present disclosure;

[0028] Figure 7 is a schematic diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] Embodiments of the present disclosure are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.

[0030] The acquisition, storage, use, and processing of data in the technical solution disclosed in this disclosure are in compliance with the relevant provisions of relevant laws and regulations.

[0031] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be considered as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0032] Figure 1 Schematic diagram of a distributed multi-copy high-performance and high-availability solution based on data files according to an embodiment of the present disclosure. Figure 1 As shown, the distributed multi-copy high-performance and high-availability solution based on data files includes the following steps:

[0033] S101, obtain the user's write request and execute the write request through the leader consensus layer in the distributed cluster.

[0034] A distributed multi-copy high-performance high-availability solution based on data files in an embodiment of the present application can be applied to a distributed cluster data update scenario. The execution subject of a distributed multi-copy high-performance high-availability solution based on data files in an embodiment of the present application can be a distributed multi-copy high-performance high-availability device based on data files in an embodiment of the present application, and the distributed multi-copy high-performance high-availability device based on data files can be set on an electronic device. Followers are ordinary members in a cluster, and they follow the instructions of the leader and maintain the same state as the leader.

[0035] It should be noted that in a distributed system, leaders and followers are key roles in implementing consensus algorithms (such as Raft, Paxos, etc.). These roles help ensure that all nodes in the cluster can reach a consensus and correctly handle read and write requests even in the event of network partitions, node failures, etc.

[0036] The leader is a specific node that coordinates operations within the cluster and serves as the main entry point for client requests. Followers are ordinary members in the cluster that follow the instructions of the leader and maintain the same state as the leader. Since all writes are handled by a single leader, conflicts can be reduced and consistency issues can be simplified.

[0037] It should be noted that the leader can be designated or elected, and there is no limitation here.

[0038] S102, after the leader consensus layer executes the write request, monitor the latest data file generated in the storage engine.

[0039] In the embodiments of the present disclosure, the storage engine may be of various types, and no limitation is made herein. In one possible implementation, the 0-layer data file generated by flushing the disk in the Log-Structured Merge (LSM) storage engine may be written in real time by monitoring as the latest data file.

[0040] S103, generating a loading event based on the latest data file, and asynchronously sending the loading event to the follower to load the latest data file, while the leader continues to execute the next write request.

[0041] In the embodiment of the present disclosure, after the latest data file is acquired, the latest data file may be preprocessed, and then the preprocessed latest data file may be used to generate a loading item.

[0042] It should be noted that asynchronous sending allows the program to send data or requests without blocking the main thread, and receive the results through callback functions, events, Promises or other mechanisms after the sending is completed.

[0043] Asynchronous sending returns control to the caller immediately after initiating the send operation without waiting for the operation to complete. This allows the program to manage resources more efficiently and provide a better user experience. Different programming languages ​​and technology stacks have their own ways to implement asynchronous sending, and developers should choose the appropriate method based on specific needs.

[0044] S104, after the follower completes loading the latest data file through the direct loading technology, the loading result is updated to the consensus data of the distributed cluster through asynchronous callback.

[0045] In the disclosed embodiments, asynchronous callbacks allow a program to perform a time-consuming operation without blocking the main thread, but to notify the result by calling a specified function when the operation is completed. When an asynchronous operation is initiated, such as reading a file or sending a HyperText Transfer Protocol (HTTP) request, the main thread does not wait for the result of these operations, but continues to execute subsequent code.

[0046] In order to ensure the consistency of distributed cluster data, the current mainstream consensus protocols, such as the Raft protocol, must confirm each write request through a quorum write, which means that each write delay has at least one more round-trip time (RTT) for replica synchronization. As the number of replicas increases, the network communication overhead increases significantly, resulting in the continuous accumulation of write delays. Secondly, the Raft protocol relies on logs for operation synchronization. Each update requires additional operation logs to be written and sent. When lagging replicas catch up, they also need to replay the logs one by one, and go through multiple layers of writing steps of WAL and memtable, which significantly increases the network and storage resource consumption of each node in replica synchronization. Therefore, as the number of replicas increases, the write performance of Raft cannot be linearly expanded, and may even become lower and lower.

[0047] In the solution disclosed in the present invention, after the leader asynchronously sends the loading event to the follower to load the latest data file, there is no need to confirm the write and continue to execute the next write request. The return of the user's successful write result does not depend on the replica synchronization, and there is no need for additional waiting for quorum write. In addition, the present invention makes full use of high-compression data files for state synchronization, which not only greatly reduces network bandwidth consumption, but also omits the multi-layer writing steps of the follower WAL and memtable, thereby significantly reducing the catch-up overhead.

[0048] Therefore, compared with the current mainstream consensus protocols, the additional resource consumption of the scheme in this disclosure for replica synchronization is far less than that for processing writes themselves, and it hardly increases significantly with the increase in the number of replicas. Especially in a multi-replica group architecture, the catch-up overhead saved by the follower can enable the leaders of other replica groups to process write requests with higher throughput. Data sharding and load balancing also enable machines to release higher throughput. Therefore, the present disclosure has the advantage of linear expansion of performance with the number of replicas, while ensuring the efficiency of system resource utilization, it effectively reduces the performance bottleneck of replica synchronization. Through experimental verification, the method based on this scheme can achieve nearly twice the performance of a single node single replica in the case of 2 replicas commonly used in production environments.

[0049] In the disclosed embodiment, the user's write request is first obtained, and the write request is executed through the leader consensus layer in the distributed cluster. Then, after the leader consensus layer executes the write request, the latest data file generated in the storage engine is listened to, and then a loading event is generated based on the latest data file, and the loading event is asynchronously sent to the follower to load the latest data file. Finally, after the follower has finished loading the latest data file, the loading result is updated to the consensus data of the distributed cluster through an asynchronous callback. Therefore, compared with the current mainstream consensus protocol, the disclosed method adopts an asynchronous replication mechanism to return the result of the user's successful write without relying on replica synchronization, without additional waiting; the disclosed method also uses high-compression data files for state synchronization, which effectively reduces the performance bottleneck of replica synchronization while ensuring the efficiency of system resource utilization, and has the advantage of linear performance expansion with the number of replicas.

[0050] In one possible implementation, in response to the restart of the consensus task, the historical data file in the storage engine is obtained, and then the latest data file that has not been loaded by the follower is determined based on the historical data file, and finally the loading items corresponding to the latest data file that has not been loaded are resent to the follower to load the latest data file. Therefore, when the consensus task is restarted, not only can the consistency of data and the stability of the system be effectively guaranteed, but also significant advantages can be brought in terms of resource utilization, management and maintenance, user experience, etc. This is crucial for building high-performance, high-availability distributed systems, especially in fields such as finance and blockchain that have extremely high requirements for data consistency and security.

[0051] It should be noted that the restart of consensus tasks (such as leader election or log replication in consensus algorithms such as Raft and Paxos) is usually to deal with various abnormal situations and ensure high availability of the system and data consistency. When certain conditions are met, the restart of the consensus task may be triggered.

[0052] In the disclosed embodiment, there may be various restart conditions for the consensus task, without any limitation herein. For example, the restart condition may be leader failure, network partition, configuration change, etc.

[0053] In the above embodiment, the write request is executed through the leader consensus layer in the distributed cluster, and the Figure 2 Explaining further, the method includes:

[0054] S201, forward the write request to the state machine through the leader consensus layer.

[0055] It should be noted that the state machine is a mathematical model or a computing model set in the leader and the follower, which is used to perform certain specific operations or monitor certain specific data, etc.

[0056] The state machine in the disclosed embodiment updates its internal state according to the operation described in the log entry. For example, if it is a state machine of a key-value database, it may perform operations of inserting, deleting or updating a specific key value.

[0057] S202, sort and number the write requests to generate a target sort, and execute the write requests based on the target sort through a state machine.

[0058] In the disclosed embodiment, the leader usually sorts and numbers the write requests. This process ensures that each write request has a unique sequence identifier in this write operation, thereby helping all nodes reach a consensus and correctly apply these requests to their respective state machines.

[0059] In the disclosed embodiment, the write request is first forwarded to the state machine through the leader consensus layer, and then the write request is sorted and numbered to generate a target sort, and the write request is executed based on the target sort by the state machine. Thus, the design of forwarding the write request to the state machine through the leader consensus layer, pre-numbering it to generate a target sort, and then executing the write request based on the sort not only ensures the consistency and reliability of the data, but also significantly improves the system performance and user experience.

[0060] In one possible implementation, after executing the write request based on the target sorting through the state machine, the execution result needs to be updated to the consensus data of the distributed cluster through the state machine.

[0061] In the above embodiment, the loading items are generated based on the latest data file, and the loading items can also be generated by Figure 3 Explaining further, the method includes:

[0062] S301, generating an operation identifier of the latest data file based on the target sorting.

[0063] In the disclosed embodiment, after the latest data file is acquired, an operation identifier of the file may be generated through target sorting. The operation identifier is unique in this writing, so that the files may be distinguished and sorted through the identifier.

[0064] S302, generating a loading item based on the operation identifier and the latest data file.

[0065] In the embodiment of the present disclosure, an operation identifier of the latest data file is generated based on the target sorting, and then a loading item is generated based on the operation identifier and the latest data file.

[0066] In the above embodiment, the loading event is asynchronously sent to the followers to load the latest data file, and the Figure 4 Explaining further, the method includes:

[0067] S401, creating an asynchronous callback processor.

[0068] It should be noted that creating an asynchronous callback handler is one of the key steps to achieve efficient and reliable data synchronization and status updates in a distributed system. Asynchronous callback handlers allow the system to process tasks in the background without blocking the main thread or affecting the execution of other operations.

[0069] S402, asynchronously sending the loading event to the follower through the asynchronous callback processor to load the latest data file.

[0070] In the disclosed embodiment, an asynchronous callback processor is first created, and then the loading event is asynchronously sent to the follower through the asynchronous callback processor to load the latest data file. Thus, the reliability and performance of the system are improved in this way, and a solid foundation is laid for future expansion and maintenance.

[0071] In a possible implementation of the present disclosure, the loading result is updated to the consensus data of the distributed cluster by asynchronous callback, and in response to the loading result being a successful loading, the event corresponding to the write request is removed from the buffer, and the consensus data is updated to release the resources corresponding to the write request. Alternatively, in response to the loading result being a failed loading, the loading event is placed in the reloading queue for reloading.

[0072] Figure 5 This is an architecture diagram of a distributed multi-copy high-performance and high-availability solution based on data files, such as Figure 5 As shown in the figure, two physical servers are used to deploy distributed cluster instances, forming a distributed cluster with two nodes and two replicas, marked as Node_1 and Node_2 respectively. The specific steps are as follows:

[0073] 1) Obtain the write request of the distributed cluster and execute the local write request in the consensus layer of Node_1, the leader of the distributed cluster;

[0074] In this embodiment, the received write requests are data points (100, "temperature", 23.5) and (101, "temperature", 24.0). Assume that the leader of the data partition corresponding to the two write requests is Node_1. The Node_1 consensus layer forwards the two requests to the corresponding state machine, pre-numbers them as 0 and 1, and writes them to the storage engine.

[0075] 2) After executing the local write request at the consensus layer, listen to the real-time write of the layer 0 data file generated by flushing the disk in the LSM storage engine;

[0076] In this embodiment, it is assumed that two write requests numbered 0 and 1 generate a 1MB layer 0 data file File_0, which contains the most recently written data points and is stored in the disk location / data / lsm / File 0;

[0077] 3) Extract the data file generated in real time, assign an operation sequence to it and package it into a data file synchronization event FileEvent_0;

[0078] In this embodiment, because the maximum pre-numbered write contained in File_0 is 1, data points are extracted from File_0 and an operation sequence commitId=1 is generated, and commitId and File_0 are packaged into FileEvent and stored in the asynchronous buffer queue.

[0079] 4) FileEvent_0 is transmitted to the sending thread through two layers of asynchronous buffer queues;

[0080] 5) The sending thread asynchronously sends FileEvent_0 to other nodes in the consensus group where the current node is located to form a replica group;

[0081] In this embodiment, the number of events to be retransmitted is 0. After checking that the buffer queue is empty, the sending thread pulls FileEvent from the transmission buffer sliding window, constructs FileEvent_0 as a synchronization request, and asynchronously sends the synchronization request to the replica node Node_2.

[0082] 6) After receiving FileEvent_0, the follower in the distributed cluster loads it according to the execution order of the sender;

[0083] In this embodiment, Node_2 should load FileEvent_0 according to the execution order, so it extracts the data file File_0 from FileEvent_0 and writes it into the local storage engine through the direct loading technology.

[0084] 7) Update the final system status through asynchronous callback to complete the consensus establishment of the distributed cluster;

[0085] In this embodiment, Node_1 receives a confirmation callback from Node_2, and after confirming that the synchronization is successful, removes the corresponding event and releases related resources.

[0086] 8) When the consensus task is restarted, all historical data files in the storage engine are extracted, and the above 3-7 processes are executed again for those that have not been synchronized;

[0087] In this embodiment, if the system restarts, it will first execute the disk flushing to flush all the memory data to the disk, and then load all the data files from the historical data directory / data / lsm / . Assuming that all the data files at this time include the synchronized file File_0 and the unsynchronized files File_1 and File_2, Node_1 will filter out the unsynchronized files File_1 and File_2, and send them to other replica nodes in sequence for synchronization.

[0088] Corresponding to the distributed multi-copy high-performance and high-availability solution based on data files provided by the above-mentioned embodiments, an embodiment of the present disclosure also provides a distributed multi-copy high-performance and high-availability device based on data files. Since the distributed multi-copy high-performance and high-availability device based on data files provided by the embodiment of the present disclosure corresponds to the distributed multi-copy high-performance and high-availability solution based on data files provided by the above-mentioned embodiments, the implementation method of the above-mentioned distributed multi-copy high-performance and high-availability solution based on data files is also applicable to the distributed multi-copy high-performance and high-availability device based on data files provided by the embodiment of the present disclosure, and will not be described in detail in the following embodiments.

[0089] Figure 6 is a schematic diagram of a distributed multi-copy high-performance and high-availability device based on data files according to an embodiment of the present disclosure, such as Figure 6 As shown, the distributed multi-copy high-performance and high-availability device 600 based on data files includes: an acquisition module 610, a monitoring module 620, a loading module 630 and an update module 640.

[0090] Among them, the acquisition module 610 is used to obtain the user's write request and execute the write request through the leader consensus layer in the distributed cluster.

[0091] The monitoring module 620 is used to monitor the latest data file generated in the storage engine after the leader consensus layer executes the write request.

[0092] The loading module 630 is used to generate loading items based on the latest data file, and asynchronously send the loading items to the followers to load the latest data file, while the leader continues to execute the next write request.

[0093] The update module 640 is used to update the loading result to the consensus data of the distributed cluster through asynchronous callback after the follower completes loading the latest data file through the direct loading technology.

[0094] In one embodiment of the present disclosure, the method further includes: in response to restarting a consensus task, obtaining historical data files in a storage engine; determining the latest data files that have not been loaded by the follower based on the historical data files; and resending the loading items corresponding to the latest data files that have not been loaded to the followers to load the latest data files.

[0095] In one embodiment of the present disclosure, a write request is executed through a leader consensus layer in a distributed cluster, including: forwarding the write request to a state machine through the leader consensus layer; sorting and numbering the write requests to generate a target sort, and executing the write request based on the target sort through the state machine.

[0096] In one embodiment of the present disclosure, after executing the write request based on the target sorting through the state machine, it includes: updating the execution result into the consensus data of the distributed cluster through the state machine.

[0097] In one embodiment of the present disclosure, generating a loading item based on the latest data file includes: generating an operation identifier of the latest data file based on a target sort; and generating a loading item based on the operation identifier and the latest data file.

[0098] In one embodiment of the present disclosure, asynchronously sending a loading event to a follower to load the latest data file includes: creating an asynchronous callback processor; and asynchronously sending the loading event to a follower to load the latest data file through the asynchronous callback processor.

[0099] In one embodiment of the present disclosure, the loading result is updated to the consensus data of the distributed cluster by means of asynchronous callback, including: in response to the loading result being a successful load, the event corresponding to the write request is removed from the buffer, and the consensus data is updated to release the resources corresponding to the write request; or, in response to the loading result being a failed load, the loading event is placed in a reloading queue for reloading.

[0100] Therefore, compared with the current mainstream consensus protocol, the present disclosure adopts an asynchronous replication mechanism to return the result of user write success without relying on replica synchronization, without the need for additional waiting; the present disclosure also uses high-compression data files for state synchronization, which effectively reduces the performance bottleneck of replica synchronization while ensuring the efficiency of system resource utilization, and has the advantage of linear performance expansion with the number of replicas.

[0101] In order to implement the above embodiment, the present disclosure further provides an electronic device 700, Figure 7 is a schematic diagram of an electronic device according to an embodiment of the present disclosure, such as Figure 7 As shown, the electronic device 700 includes: a processor 701 and a memory 702 that is communicatively connected to the processor, the memory 702 stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor 701 to implement the present disclosure. Figure 1-Figure 5 The embodiment provides a distributed multi-copy high-performance and high-availability solution based on data files.

[0102] In order to implement the above embodiments, the present disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to implement the above embodiments. Figure 1-Figure 5 The embodiment provides a distributed multi-copy high-performance and high-availability solution based on data files.

[0103] In order to implement the above embodiments, the present disclosure also provides a computer program product, including a computer program. When the computer program is executed by a processor, the computer program implements the above embodiments. Figure 1-Figure 5 The embodiment provides a distributed multi-copy high-performance and high-availability solution based on data files.

[0104] It should be noted that personal information from users should be collected for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. In addition, such collection / sharing should be carried out after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign the agreement / authorization including authorization of relevant user information before the user uses the function. In addition, any necessary steps should be taken to protect and safeguard access to such personal information data and ensure that others who have access to personal information data comply with its privacy policy and procedures.

[0105] The present application is expected to provide an implementation scheme for users to selectively block the use or access of personal information data. That is, the present disclosure is expected to provide hardware and / or software to prevent or block access to such personal information data. Once the personal information data is no longer needed, the risk can be minimized by limiting data collection and deleting the data. In addition, when applicable, such personal information is de-identified to protect the privacy of the user.

[0106] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0107] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0108] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0109] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that contains, stores, communicates, propagates or transmits a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0110] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0111] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0112] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0113] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A distributed multi-copy high-performance and high-availability solution based on data files, characterized in that: include: Obtain the user's write request and execute the write request through the leader consensus layer in the distributed cluster; After the leader consensus layer executes the write request, listen to the latest data file generated in the storage engine; Generate a loading event based on the latest data file, and asynchronously send the loading event to the follower to load the latest data file, while the leader continues to execute the next write request; After the follower has completed loading the latest data file through the direct loading technology, the loading result is updated to the consensus data of the distributed cluster through asynchronous callback.

2. The method according to claim 1, characterized in that The method further comprises: In response to the consensus task being restarted, obtaining the historical data file in the storage engine; Determine the latest data file that the follower has not loaded based on the historical data file; The loading event corresponding to the latest data file that has not been loaded is resent to the follower to load the latest data file.

3. The method according to claim 2, characterized in that The executing the write request through the leader consensus layer in the distributed cluster includes: Forwarding the write request to the state machine through the leader consensus layer; The write requests are sequenced and numbered to generate a target sequence, and the write requests are executed based on the target sequence by the state machine.

4. The method according to claim 3, characterized in that After executing the write request based on the target sorting by the state machine, the method further comprises: The execution result is updated to the consensus data of the distributed cluster through the state machine.

5. The method according to claim 3, characterized in that: The generating of loading items based on the latest data file includes: Generate an operation identifier of the latest data file based on the target sorting; The loading item is generated based on the operation identifier and the latest data file.

6. The method according to any one of claims 1 to 5, characterized in that The step of asynchronously sending the loading event to the follower to load the latest data file includes: Create an asynchronous callback handler; The loading event is asynchronously sent to the follower through the asynchronous callback processor to load the latest data file.

7. The method according to claim 3, characterized in that The method of updating the loading result to the consensus data of the distributed cluster by asynchronous callback includes: In response to the loading result being a successful loading, removing the event corresponding to the write request from the buffer, updating the consensus data, and releasing the resources corresponding to the write request; or, In response to the loading result being a loading failure, the loading item is placed in a reloading queue for reloading.

8. A distributed multi-copy high-performance and high-availability device based on data files, characterized in that: include: An acquisition module, used to acquire a user's write request and execute the write request through a leader consensus layer in a distributed cluster; A monitoring module, used for monitoring the latest data file generated in the storage engine after the leader consensus layer executes the write request; A loading module, used for generating a loading event based on the latest data file, and asynchronously sending the loading event to the follower to load the latest data file, while the leader continues to execute the next write request; The update module is used to update the loading result to the consensus data of the distributed cluster through asynchronous callback after the follower completes loading the latest data file through direct loading technology.

9. An electronic device, characterized in that: Including memory and processor; The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images